Multimedia program editing and presenting system with flexible layout capability by simplified input operations

[0002] 1. Field of the Invention

[0003] The invention generally relates to a multimedia program editing and presenting system and, more particularly, to a method and system for permitting a user to edit a multimedia program or scenario with a higher degree of freedom by simplified operations.

[0004] 2. Description of the Prior Art

[0005] A multimedia computer program or a scenario defines the details of a multimedia application. In editing a multimedia program, the user selects one out of the displayed media materials to assign to a layout window or frame either provided by a prepared template or specified by the user if the selected media material is of a visual type (i.e., any of the video, picture and text types) and to assign to a desired scenario time (or a time measured from the beginning of the scenario) to start replaying the selected material if the selected media material is of the audio type. This process is repeated till a desired scenario is obtained.

[0006] The present invention especially relates to a method and system for editing a scenario by using materials of a visual type and frames (or windows) where the materials are to be displayed.

[0007] Japanese unexamined patent publication No. Hei 7-287,646 discloses a scenario editing/presenting method and device therefor. A desired number of frames can be defined for a time period from the start time to the end time of each material. This provides full flexibility in defining frames. However, for each of the frames, the user has to specify a start time, the coordinates of the frame position and the size (width, height) at the start time, an end time, the coordinates of the frame position and the size (width, height) at the start time, and the start time.

[0008] In some multimedia presentation systems, a template of a frame layout is provided so that the user can create a multimedia application by putting materials in frames that have been provided in the template. In such a system, it can be assumed that the material Ma is put into a frame at a time T0, defined in a scenario (hereinafter referred to as “scenario time”), and the scenario is played and then terminated at a time T1. In this case, if the position of the frame of the material Ma is changed from a first position to a second position at a time T1, even if the scenario time is turned back to T0, the position of the frame remains at the second position. This is because the frames of the system are not controlled in association with time.

[0009] It is therefore an object of the invention to provide a multimedia program editing and/or presenting method and system that enables a layout with a higher degree of freedom by simplified input operations.

[0010] According to the invention, a multimedia program editing method that enables a layout with a higher degree of freedom by simplified input operations is obtained. The method comprising the steps of:

[0011] displaying available media materials;

[0012] displaying a plurality of predetermined frames

[0013] permitting a user to set a current scenario time to a desired time by clicking on a desired one of a plurality of time control icons;

[0014] permitting the user to associate a desired material with one of the predetermined frames at the desired time by mouse operations;

[0015] in response to the associating of the desired material, registering information on the association to a scenario to prepare for an instruction to present the desired material in the one of the predetermined frames;

[0016] permitting the user to change an attribute of the one of the predetermined frames;

[0017] permitting the user to terminate the association by specifying the desired material and clicking on a predetermined icon; and

[0018] in response to the terminating the association, restoring the attribute of the one of the predetermined frames to a state before the desired material was attached to the frame.

[0019] If the information is deleted from the scenario by specifying the desired material and clicking on a predetermined icon, then the attribute of the one of the predetermined frames is restored to a state before the desired material was attached to the frame.

[0020] If any attribute of a vacant (or not attached) frame is changed, then the attribute of the vacant frame is restored to an original state after playing an attached material.

[0021] Further objects and advantages of the present invention will be apparent from the following description of the preferred embodiments of the invention as illustrated in the accompanying drawing, in which:

[0022] FIG. 1 is a schematic block diagram showing an arrangement of a multimedia editing system 100 according to an illustrative embodiment of the invention;

[0023] FIG. 2 is a diagram showing exemplary contents of the mass storage device 112 of FIG. 1;

[0024] FIG. 3 is a diagram showing an exemplary structure of a media material table or file 20 which stores information on the media materials prepared for a certain multimedia title to be created;

[0025] FIG. 4 is a diagram showing an exemplary screen displayed when the multimedia editing system 100 is in an editing mode;

[0026] FIGS. 5A and 5B, when vertically combined, constitutes a diagram showing an exemplary editing process based on a user-defined frame layout in the multimedia program editing and presenting system 100;

[0027] FIGS. 6A and 6B, when vertically combined, constitutes a diagram showing an exemplary editing process based on a template in the multimedia program editing and presenting system 100;

[0028] FIG. 7 is a flowchart showing an attaching operation executed by the controller 110 under the condition of the multimedia editor program 2 according to the illustrative embodiment of the invention;

[0029] FIG. 8 is a flowchart showing a position change operation, for the attached frame, executed by the controller 110; and

[0030] FIG. 9 is a flowchart showing a detaching operation executed by the controller 110 under the condition of the multimedia editor program 2 according to the illustrative embodiment of the invention.

[0031] Throughout the drawing, the same elements when shown in more than one figure are designated by the same reference numerals.

[0032] FIG. 1 is a schematic block diagram showing an arrangement of a multimedia editing system 100 according to an illustrative embodiment of the invention. In FIG. 1, the video editing system 100 is a computer-based system which comprises a controller 100 including a central processing unit (not shown), a read only memory (not shown) and a random access memory (not shown) which are interconnected through a system bus 111; a mass storage device 112; a keyboard 116 and a mouse 118 connected through input interfaces (IFs) 114; a display device 122 and a loudspeaker 124 connected through a video/audio interface 120; and a FD (floppy disc) drive 128 connected through FD interface 126. The video editing system 100 further comprises an optical disc drive 134 which plays an optical disc such as a video-CD (video-compact disc), a PhotoCD, etc. The optical disc drive 134 is connected to the system bus 111 through a movie and still media resource interfaces 130. The system 100 further comprises a sound board 136 which provides an audio output function.

[0033] FIG. 2 is a diagram showing exemplary contents of the mass storage device 112 of FIG. 1. The mass storage device 112 stores an operating system 1. The operating system 1 may be any suitable operating system whether it is a tailored or standard one. The storage device 112 stores not only a multimedia editor program 2 according to the embodiment of the invention but also an audio/video decoder 4 such as an MPEG (Moving Picture Experts Group) decoder and an image decoder 6 for a still picture of a format such as JPEG, PhotoCD, etc. This enables the multimedia editing system 100 not only to create a multimedia application but also to play the created application. Also, The storage device 112 stores various data 8 including media materials for use in scenario editing.

[0034] FIG. 3 is a diagram showing an exemplary structure of a media material table or file 20 which stores information on the media materials prepared for a certain multimedia title to be created. Each of the records of the media material table comprises a material ID field, a media type field, media format field, a play time field, etc. The media type field contains a value indicative of any of the movie, still, text and audio types. The play time field contains the play time of the material if the material is either of the movie type or of the audio type, and the length in byte if the material is of the text type.

[0035] FIG. 4 is a diagram showing an exemplary screen displayed when the multimedia editing system 100 is in an editing mode. There are shown presentation frames 41 through 43 in which a moving picture (or movie) material, a still picture material and a text material are presented. The presentation frames 41 through 43 may be given as one of templates provided by the system 100 and/or preset by the user. Alternatively, the presentation frames 41 through 43 may be defined by the user. Defining the presentation frames can be achieved by simple mouse 118 operations in a like manner as practiced in figure drawing software. The screen 40 includes a play icon 47 to preview a scenario in course of editing or to play a completed scenario, a stop icon 48 to stop previewing or playing the scenario, a scroll bar 49 and scroll box 50 that roughly indicates the running time (RT) of the scenario and the current scenario time (CST) or the current temporal position in scenario by the scroll box 50 moving with the advance of scenario by playing. The user can roughly set the current scenario time (CST) by dragging the scroll box 50. The screen 40 further includes precise indications of RT 51 and CST 53 and respective pairs 52 and 54 of up and down icons for precise setting RT 51 and CST 53 to desired values. At the bottom of the screen 40, there is a media material display area 55 where available media materials are displayed.

[0036] The system 100 operates in either an edit mode or a presentation mode. The following description will be focused on the edit mode.

[0037] For the sake of better understanding the present invention, it is useful to describe basic concepts of the multimedia editor program 2 according to the principles of the invention.

[0038] The editing is performed either on the basis of a user-defined frame layout or on the basis of a selected template. In a template-based editing, the user is prohibited from changing any attributes (e.g., the position, the size, etc.) of a vacant presentation frame.

[0039] The presentation of any material is begins at the time of attaching of the material and ends at the time of detaching of the material. The attaching and the detaching are either programmed by the designer in case of a noninteractive or sequentially-presented multimedia application or caused by viewer's selection of a menu item, clicking on a control icon and so on in case of an interactive multimedia application.

[0040] A visual material, i.e., a material of the movie, still or text type is attached to a desired frame 41, 42 or 43 at a scenario time at which the user desires to start playing the visual material. In order to achieve this, the user first sets the CST indicator 53 to the desired scenario time either by stopping the previewing of the scenario at the desired time with the stop icon 48 or by dragging the scroll box 50 or clicking on the CST setting icon pair 54 till the CST indicator 53 reads the desired scenario time; select a desired visual material in the available material display area 55: select a desired presentation frame 41, 42 or 43; and clicks on the AT icon 57.

[0041] An audio material is attached to a scenario time at which the user desires to start playing the audio material. In order to achieve this, the user first sets the CST indicator 53 to the desired scenario time in the same manner as described-above; selects a desired audio material in the available material display area 55; and clicks on an AT icon 57.

[0042] The media materials of each type are separately attached in order of time to be presented.

[0043] If a material is detached at a certain scenario time (say, at Ti), the playing of the material is terminated just before the detached scenario time (at (Ti−1) in this example) so that another material of the same material type can be attached at the detached scenario time. The time of detaching defines the presentation end

[0044] FIGS. 5A and 5B, when vertically combined, constitutes a diagram showing an exemplary editing process based on a user-defined frame layout in the multimedia program editing and presenting system 100. In FIG. 5, T1 through T3 in the first column are scenario times; the second column shows user's operations, the third column shows displayed screens, the DEFINED FRAMES column shows the contents of a defined frame storage area in the controller 110, the SCENARIO column shows a schedule table of a material Mi which is registered in the scenario under editing. It is noted that the scenario time is advanced only by previewing or playing the scenario with the preview icon 47 or by controlling the scenario time controlling icon 53.

[0045] It is seen from the screen SC1 that the user has defined two frames F1 and F2. Defining a frame (e.g., Fi) is easily achieved by dragging the mouse 118 from one end (e.g., Pia) of a diagonal of the frame to the other end thereof (e.g., Pib). In response to the definition of the frames F1 and F2, the controller 110 stores the frame ID F1 and F2 and respective diagonal point pairs of coordinates, e.g., {(X1a, Y1a), (X1b, Y1b)} and {(X2a, Y2a), (X2b, Y2b)}. It should be noted that a frame Fi may be any polygon defined by N points {(Xi1, Yi1), (Xi2), Xi2), . . . ,(XiN, YiN)}. For the purpose of simplicity, these coordinate expressions of a frame Fi are hereinafter denoted en bloc as (Fi).

[0046] In an editing process based on a user-defined frame layout, the user is permitted to change the attribute (the position and/or size) of a vacant frame. If the frame attribute is changed, the old frame data is overwritten by the new frame data. For example, the position of the frame F1 has been changed in screen SC2. Responsively, the frame F1 has been changed to F3 in the defined frame storage area. Then a media material Mi is attached to the frame F3 at T1. FIG. 7 is a diagram showing an attaching operation executed by the controller 110 under the condition of the multimedia editor program 2 according to the illustrative embodiment of the invention. In FIG. 7, controller 110 adds a schedule table 50 giving a serial number to the table 50; enters the attached material ID (mi in this example) in step 200. In step 201, controller 110 copies the geometrical data of target frame (F3 in this case) from the defined frame storage area (no shown) to the schedule table 50 and deletes target frame data from the storage area. In step 202, controller 110 enters the current scenario time (CST) (T1 in this case) as the presentation start time in the table. In step 203, controller 110 makes a test to see if the material Mi is a time-series material or a material with a time axis. If so, controller 110 enters the CST plus the play time of the material as the presentation end time in the table 50 in step 204 and proceeds to step 205. Otherwise, controller 110 proceeds to step 205. If a material is attached to a frame in the editing mode, controller 110 displays the attached material (Mi) in the target frame in step 205 and returns to the program 2. The presentation start time, the presentation end time and the target (or attached) frame constitutes a record of the schedule table 50.

[0047] Though the vacant frame is usually displayed in the editing mode, the vacant frame is not displayed during the previewing period as shown in the screen SC4. When the previewing is stopped at a scenario time T2, the vacant frame is displayed again. The position of the frame f3 is changed from SC5 to SC6. FIG. 8 is a flowchart showing a position change operation, for the attached frame, executed by the controller 110. In step 210, controller 110 adds a new record by entering the time of the position change (as the presentation start time for the new record) and the geometrical data of the new frame, and by copying the presentation end time of the last record to the corresponding field of the new record. In step 211, controller 110 replaces the presentation end time of the last record with the time of the position change minus one second and returns to the program 2.

[0048] At time T3, the material Mi is detached from the frame F4. FIG. 9 is a flowchart showing a detaching operation executed by the controller 110. in step 220, controller 110 copies the geometrical data of the detached frame (F4 in this case) of the last record to the defined frame storage area. In this way, an attached frame keeps its position after being detached in the editing based on a user defined layout. In step 221, controller 110 stops displaying the detached frame F4 as shown in the screen SC9.

[0049] If the user clicks on the delete button 44 while selecting a material that has been attached, then the schedule table 50 is deleted from the scenario. In this case, the geometrical data of the attached frame of the first record (F3 in this case) is copied to the defined frame storage area. Thus, if the attached material is deleted from the scenario, then the frame having been attached restores the position just before the material has been attached.

[0050] FIGS. 6A and 6B, when vertically combined, constitutes a diagram showing an exemplary editing process based on a template in the multimedia program editing and presenting system 100. FIG. 6 is identical except that changing the vacant frame position is prohibited; a template is uses in stead of a user-defined layout; and accordingly the detaching and the deleting causes the detached or deleted material to restore the position defined by the template as shown in screens SC20 of FIG. 6A and SC21 of FIG. 6B, respectively.

[0051] As described above, the multimedia editing and presenting system enables flexible editing operation by simplified mouse operations.

[0052] The user is prohibited from changing any attribute of a vacant frame in the template-based editing. Alternatively, if the user has changed any attribute of a vacant frame in the template-based editing, the change is neglected after playing a material, i.e., the vacant frame restores its original position after a play of a material.

[0053] Many widely different embodiments of the present invention may be constructed without departing from the spirit and scope of the present invention. It should be understood that the present invention is not limited to the specific embodiments described in the specification, except as defined in the appended claims.