Specifying a point of origin of a sound for audio effects using displayed visual information from a motion picture

[0002] A motion picture generally has a soundtrack, and a soundtrack often includes special effects that provide the sensation to an audience that a sound is emanating from a location in a theatre. Such special effects are called herein “spatial audio effects” and include one-dimensional effects (stereo effects, often called “panning”), two-dimensional effects and three-dimensional effects (often called “spatialization,” or “surround sound”). Such effects may affect the amplitude, for example, of the sound in each speaker.

[0003] To create such spatial audio effects, the soundtrack is edited using a stereo or surround sound editing system or a digital audio workstation that has a graphical and/or mechanical user interface that allows an audio editor to specify parameters of the effect. For example, in the Avid Symphony editing system, a graphical “slider” is used to define the relative balance between left and right channels of stereo audio. For surround sound, an interface may be used to permit an editor to specify a point in three-dimensional space, from which the relative balance among four or five channels can be determined. Some systems allow the user simultaneously to hear the spatial audio effect and to see a representation of the effect parameters. Using such systems, the settings for various spatial audio effects are set subjectively by the audio editor based on the audio editor's understanding of how the point of emanation of the sound is related to images in the motion picture.

[0004] Displaying visual information from a motion picture in a visual field within a designated extent of a related aural field supports editing of a spatial audio effect for the motion picture. The extent of a related aural field also is displayed. Information specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field is received for each of a number of frames of a portion of the motion picture. This information may be received from a pointing device that indicates a point in the displayed extent of the aural field, or from a tracker that indicates a position of an object in the displayed visual information, or from a three-dimensional model of an object that indicates a position of an object in the displayed visual field. Using the specified point of origin and the relationship of the visual and aural fields, parameters of the spatial audio effect may be determined, from which a soundtrack may be generated. Information describing the specified point of origin may be stored. The frames for which points of origin are specified may be key frames that specify parameters of a function defining how the point of origin changes from frame to frame in the portion of the motion picture. The relationship between a visual field and an aural field may be different for each of the plurality of frames in the motion picture. This relationship may be specified by displaying the visual information from the motion picture and an indication of the extent of the aural field to a user, who in turn, through an input device, may indicate changes to the extent of the aural field with respect to the visual information.

[0005] FIGS. 1A-C illustrate example relationships between visual and aural fields;

[0006] FIG. 2 is a dataflow diagram of operation of a graphical user interface; and

[0007] FIG. 3 is a dataflow diagram of a system for generating a soundtrack.

[0008] Displaying visual information from a motion picture in a visual field within a designated extent of a related aural field supports editing of a spatial audio effect for the motion picture. The visual field represents the field of view of the visual stimuli of the motion picture from the perspective of the audience. The aural field represents the range of possible positions from which a sound may appear to emanate. The portions of the aural field that are also in the video field are “onscreen.” The portions of the aural field that are not in the video field are “offscreen.” The relationship between a visual field and an aural field may be different for each of the plurality of frames in the motion picture.

[0009] FIGS. 1A-C illustrate example relationships between visual and aural fields. In FIG. 1A, the visual field 100 is included within an aural field 102. Both the visual field and the aural field are two-dimensional and rectangular. Within the visual field 100, visual information from the motion picture may be displayed. An indication of the extent of the aural field 102 also may be displayed, for example, by a box 104. The aural field extends beyond the visual field on the left and right. The portion of the aural field extending beyond the visual field permits sound effect parameters, for example, left and right pan settings, to be defined for a sound that is offscreen.

[0010] In FIG. 1B, the visual field 110 is included within an aural field 112. Both the visual field and the aural field are two-dimensional. The visual field is rectangular whereas the aural field is an ellipse. Within the visual field 110, visual information from the motion picture may be displayed. An indication of the extent of the aural field 112 also may be displayed, for example, by an ellipse 114. The aural field extends beyond the visual field on the left, right, top and bottom.

[0011] In FIG. 1C, the visual field 120 is included within an aural field 122. The visual field is two-dimensional and rectangular and the aural field is three-dimensional, for example, a sphere. Within the visual field 120, visual information from the motion picture may be displayed. An indication of the extent of the aural field 122 also may be displayed, for example, by graphically depicting a sphere 124. The aural surrounds the visual field.

[0012] In FIGS. 1A-C, the visual field is shown within the aural field and has points at the edges of the aural field. The visual field may extend beyond portions of the aural field, or may not have any point on the edge of the aural field. The visual field also may be off center in the aural field. The visual field also may be three-dimensional, for example, where the visual information from the motion picture is generated using three-dimensional animation or where the visual information is intended to be displayed on a curved surface.

[0013] The aural field may be specified by default values indicating the size and shape of the aural field with respect to the visual field. The user may in turn, through an input device, indicate changes to the extent of the aural field with respect to the visual information. The default values and any changes specify a coordinate system within which a user may select a point, in a manner described below. For example, the range of available positions within an aural field may be specified as −100 to 100 in a single dimension (left to right or horizontally with respect to the visual field), with 0 set as the origin or center. The specified position may be in one, two or three dimensions. The specified position may vary over time, for example, frame-by-frame in the motion picture.

[0014] A data flow diagram of a graphical user interface for a system using such information about the visual and aural fields is described in connection with FIG. 2.

[0015] Information describing the visual field 200, the aural field 202 and the relationship 204 of the aural and visual fields is received by a display processing module 206 of the graphical user interface 208. The information describing the visual field may include, for example, its size, position, shape and orientation on the display screen, and a position in a motion picture that is currently being viewed. The information describing the aural field may include, for example, its size, position, shape and orientation. The information describing the relationship 204 of the aural and visual fields may include any information that indicates how the aural field should be displayed relative to the visual field. For example, one or more positions and/or one or more dimensions of the aural field may be correlated to one or more positions and/or one or more dimensions in the video field. The size of the visual field in one or more dimensions may be represented by a percentage of the aural field in one or more dimensions. Also, given an origin of the aural field and a radius, one or more edges of the visual field may be defined by an angle.

[0016] This information 200, 202 and 204 is transformed by into display data 210 for example, to illustrate the relative positions of these fields, which is then provided to a display (not shown) for viewing by the editor. The display processing module also may receive visual information from the motion picture for a specified frame in the motion picture to create the display, or the information regarding the aural field may be overlaid on an already existing display of the motion picture. The user generally also has previously selected a current point in the motion picture for which visual information is being displayed.

[0017] The editor then manipulates an input device (not shown) which provides input signals 212 to an input processing module 214 of the graphical user interface 208. For example, a user may select a point in the visual field corresponding to an object that represents the source of a sound, such as a person. The input processing module converts the input signals into information specifying a point of origin 216. This selected point may be represented by a value within the range of −100 to 100 in the aural field. This point of origin is associated with the position in the motion picture that is currently being viewed. This information may be stored as “metadata”, along with the information describing the aural field and its relationship to the visual field, for subsequent processing of the soundtrack, such as described below in connection with FIG. 3. During use of the system by the editor, the system may generate the sound effect to allow playback of the sound effect for the editor.

[0018] The information specifying the point of origin may be provided for each of a number of frames of a portion of the motion picture. Such frames may be designated as key frames that specify parameters of a function defining how the point of origin changes from frame to frame in the portion of the motion picture. The position of the sound for any intermediate frames may be obtained, for example, by interpolation using the positions at the key frames.

[0019] Information specifying a point of origin of a sound used in the spatial audio effect with respect to the visual field also may be received from a tracker that indicates a position of an object in the displayed visual information, or from a three-dimensional model of an object that indicates a position of an object in the displayed visual field.

[0020] Using the specified point of origin of a sound in the aural field, parameters of the spatial audio effect may be determined. In particular, the selected point in the aural field is mapped to a value for one or more parameters of a sound effect, given an appropriate formula defining the sound effect, for which many are available in the art. The sound effect may be played back during editing, or may be generated during the process of generating the final soundtrack.

[0021] A typical process flow for creating the final soundtrack of the motion picture is described in connection with FIG. 3. A motion picture with audio tracks 300 is processed by an editing system 302, such as a digital audio workstation or a digital nonlinear editing system, that has an interface such as described above. The editing system 302 outputs metadata describing the audio effects 304. A soundtrack generator 306 receives the metadata output from the editing system 302, and the audio data 308 for the audio track, and produces the soundtrack 310 by determining the parameters of any audio effects from the metadata, and applying the audio effects to the audio data.

[0022] The use of visual and aural fields as described above may be used, for example, in a nonlinear editing system. Such a system allows an editor to combine sequences of segments of video, audio and other data stored on a random access computer readable medium into a temporal presentation, such as a motion picture. During editing, a user specifies segments of video and segments of associated audio. Thus, a user may specify parameters for sound effects during editing of an audio-visual program.

[0023] Having now described an example embodiment, it should be apparent to those skilled in the art that the foregoing is merely illustrative and not limiting, having been presented by way of example only. Numerous modifications and other embodiments are within the scope of one of ordinary skill in the art and are contemplated as falling within the scope of the invention.