Method for the automatic computerized audio visual dubbing of movies

[0002] This application claims priority from U.S. Provisional Application No. 60/008874 filed Dec. 19, 1995.

[0003] Portions of this application are contained on compact disc(s) the contents of which are entirely incorporated herein by reference. The compact discs are labeled as Copy 1, and Copy 2, respectively. The compact discs are identical and each includes the following ASCII files:

[0004] The present invention relates to a method for automatic audio visual dubbing. More specifically the said invention relates to an efficient computerized automatic method for audio visual dubbing of movies by computerized image copying of the characteristic features of the lips movements of the dubber onto the mouth area of the original speaker. The present invention uses a method of vicinity-searching, three-dimensional head modeling of the original speaker, and texture mapping techniques in order to produce the new images which correspond to the dubbed sound track.

[0005] The invention overcomes the well known disadvantage of the correlation problems between lips movement in the original movie and the sound track of the dubbed movie.

[0006] First are provided some definitions of important key words employed in this specification.

[0007] Actor (the original actor)—an actor, speaker, singer, animated character, animal, an object in a movie, or a subject in a still photograph.

[0008] Audio visual Dubbing—Manipulating, in one or more frames, the mouth area of the actor so that its status will be similar as much as possible to that of the dubber in the reference frame.

[0009] Correlation Function—A function describing the similarity of two image regions. The higher the correlation, the better is the match.

[0010] Dubber—The person or persons, who speak/narrate/sing/interpret the target text. The dubber can be the same as the actor.

[0011] Dubbing—Replacing part or all of one or more of the original sound tracks of a movie, with its original text or sounds (including the case of the silent track of a still photograph), by another sound track containing the target text and/or sound.

[0012] Edge Detector—A known image processing technique used to extract boundaries between image regions which differ in intensity and/or color.

[0013] Face Parametrization—A method that numerically describes the structure, location, and expression of the face.

[0014] Head Model—A three-dimensional wire frame model of the face that is controlled by numerous parameters that describe the exact expression produced by the model (i.e. smile, mouth width, jaw opening, etc.).

[0015] Movie (the original movie)—Any motion picture (e.g. cinematic feature film, advertisement, video, animated cartoon, still video picture, etc.). A sequence of consecutive pictures (also called frames) photographed in succession by a camera or created by an animator. In the case of the movie being a still photograph, all of the consecutive pictures are identical to each other. When shown in rapid succession an illusion of natural motion is obtained, except for the case of still pictures. A sound-track is associated with most movies, which contains speech, music, and/or sounds, and which is synchronized with the pictures, and in particular where the speech is synchronized with the lip movements of the actors in the pictures. Movies are realized in several techniques. Common methods are: (a) recording on film, (b) recording in analog electronic form (“video”), (c) recording in digital electronic form, (d) recording on chips, magnetic tape, magnetic disks, or optical disks, and (e) read/write by magnetic and/or optical laser devices. Finally, in our context, an “original movie” is also an audio-visual movie altered by the present invention, which serves as a base for further alterations.

[0016] Original Text—A text spoken or sung by the actor when the movie is being made, and which is recorded on its sound track. The text may be narrated in the background without showing the speaker, or by showing a still photograph of the speaker.

[0017] Pixel—Picture element. A digital picture is composed of an array of points, called pixels. Each pixel encodes the numerical values of the intensity and of the color at the corresponding picture point.

[0018] Reference Similarity Frame—A picture (being a frame in the original movie, a frame in any other movie, or a still photograph) in which the original actor has the desired features of the mouth-shape and head posture suitable for the audio visually dubbed movie.

[0019] Target Text—A new vocal text, to replace the original vocal text of the actor. The target text may also be that which is to be assigned to an actor who was silent in the original movie. The new text can be in another language, to which one refers as DUBBING. However, this invention relates also to replacement of text without changing the language, with the original actor or with a dubber in that same language. The target text may have the same meaning as the original text, but may have also a modified, opposite, or completely different meaning. According to one of many applications of the present invention, the latter is employed for creation of new movies with the same actor, without his/her/its active participation. Also included is new vocal text used to replace the null vocal text attached to one or more still photographs.

[0020] Texture Mapping—A well known technique in computer graphics which maps texture onto a three-dimensional wire frame model.

[0021] Two-Dimensional Projection—The result of the rendering of the three-dimensional face model onto a two-dimensional device like a monitor, a screen, or photographic film.

[0022] Movies are often played to an audience that is not familiar with the original language, and thus cannot understand the sound track of such movies. Two well known common approaches exist to solve this problem. In one approach sub-titles in typed text of the desired language are added to the pictures, and the viewers are expected to hear the text in a foreign language and simultaneously to read its translation on the picture itself. Such reading distracts the viewers from the pictures and from the movie in general. Another approach is dubbing, where the original sound-track with the original text is being replaced by another sound-track with the desired language. In this case there is a disturbing mis-match between the sound-track and the movements of the mouth.

[0023] There have been some earlier attempts to overcome these disadvantages, none of which have been commercialized because of inherent principal difficulties which made the practical execution unrealistic. Thus, in U.S. Pat. No. 4,600,281 a method is described which performs the measurements of the shape of the mouth manually by a ruler or with a cursor, and corrects the mouth shape by moving pixels within each frame. As will be seen in the description of the invention, the method according to the present invention is inherently different and much superior in the following points: In the present invention the tracking of the shape of the mouth is done automatically and not manually. In the present invention changing the shape of the mouth is done by using a three-dimensional head model, for example like those described by P. Ekman and W. V. Friesen, (Manual for the Facial Action Unit system, consulting Psychologist Press, Palo Alto 1977). In the present invention the mouth area of the actor is replaced using the mouth area of a reference similarity frame. In the present invention mouth status parameters of the dubber are substituted for mouth status parameters of the actor.

[0024] The U.S. Pat. No. 4,260,229 relates to a method of graphically creating lip images. This U.S. patent is totally different from the present invention: In the U.S. patent, speech sounds are analyzed and digitally encoded. In the present invention no sound analysis is done; nor is any required at all.

[0025] To make for better viewing of the audio visually dubbed movie, the present invention provides a computerized method wherein, in addition to replacing the sound track to the target text, the mouth movements of the actor are being automatically changed to match the target text. The new mouth movements are linguistly accurate and visually natural looking according to all of the observable parameters of the actor's face.

[0026] The present invention provides a method for automated computerized audio visual dubbing of movies, comprising of the following steps (see FIG. 1):

[0027] (a) selecting from the movie a frame having a picture, preferably frontal, of the actor's head and, if available, a frame with its side profile;

[0028] (b) marking on the face several significant feature points and measuring their locations in the frame;

[0029] (c) fitting a generic three-dimensional head model to the actor's two-dimensional head picture by adapting the data of the significant feature points, as measured in stage (b), to their location in the model;

[0030] (d) tracking of the said fitted three-dimensional head model parameters throughout the movie, from one frame to its successor, iteratively in an automated computerized way and creating a library of reference similarity frames.

[0031] (e) taking a movie of a dubber wherein the dubber speaks the target text;

[0032] (f) repeating stages (a), (b), (c), and (d) with the dubber;

[0033] (g) normalizing the dubber's minimum and maximum values of each parameter to the actor's minimum and maximum values of the same parameters;

[0034] (h) mapping, on a frame to frame basis, the two-dimensional actors face onto its three-dimensional head model by using a texture mapping technique, making use of reference similarity frames;

[0035] (i) changing the texture mapped three-dimensional model obtained in stage (h) by replacing, on a frame to frame basis, the original mouth parameters with the mouth parameters as computed in stage (d) for the dubber and obtaining the parametric description for the new picture, with identical values to the original, except that the actor's mouth status resembles the mouth status of the dubber;

[0036] (j) texture mapping the lips area of the same actor from a frame in the movie, with identical or very similar mouth status to the desired new mouth status, onto the lips area of the actor's head model for the current frame and then projecting the lips area from the actor's head model onto the current new frame. (This stage is optional, depending on the application.)

[0037] By using the three dimensional head model one can control the audio visual dubbing process even if the actor is moving his head. In most applications about 15 significant feature points on the face are used in the tracking stage, such as eye corners, mouth corners, and the nostrills. Only those feature points which are visible to the viewer (using the information available to the model) are tracked.

[0038] In the present invention, audio visual dubbing is normally used in conjunction with the use of audio dubbing; but one may also use it in conjunction with an audio track where no equivalent track exists in the original movie.

[0039] The method according to the present invention is useful for the audio visual dubbing of motion pictures such as cinematic feature films, advertisements, video, and animated cartoon. Also the audio visual dubbing of still photographs, wherein all of the frames of the movie are the same, is made possible by the present invention. For instance, still photographs are used for this type of movie in T.V. news programs where the reporter's voice is heard while a still photograph of him/her is shown.

[0040] Thus, according to the present invention even speachless actors, infants, animals, and inanimate objects can be audio visually dubbed to speak in any language.

[0041] According to our invention, the animation process saves much of the labor associated with the animation of the mouth area.

[0042] The present invention further provides a computer program (see Appendix 1) for operating the computerized audio visual dubbing.

[0043] The present invention further relates to the library of reference similarity frames as created in Step d (above).

[0044] The present invention will be further described by FIGS. 1-4. These figures are solely intended to illustrate the preferred embodiment of the invention and are not intended to limit the scope of the invention in any manner. Likewise, the appended computer program represents an example of the implementation of the method disclosed in the present patent and is not intended to limit the scope of the method in any way.

[0045] FIG. 1 illustrates a block diagram showing the method stages.

[0046] FIGS. 2aand 2billustrate an example of significant points on a generic frontal picture of a head (FIG. 2a) and a generic side profile picture of a head (FIG. 2b).

[0047] FIG. 3 shows an example of a generic wire frame face model.

[0048] FIG. 4ais an example showing how a customer fitted wire frame model fits onto a frontal view of an actor's face.

[0049] FIG. 4bis an example showing how a custom fitted wire frame model fits onto a profile view of an actor's face.

[0050] Given an original movie where an actor speaks the original text, a movie of a dubber is made where the dubber speaks the target text in either another language or the same language. The movie of the dubber is taken while the dubber performs a routine dubbing adaptation of the original into the target text.

[0051] This invention provides a method for the changing of the actor's facial motions in the original movie to create a new movie having the sound-track in the target text from the movie of the dubber, while the pictures are of the original movie, whereas the motion of the mouth of the actor is modified to correspond to the new sound-track.

[0052] For brevity considerations, the description of this invention uses pictures in electronic digital form (composed of an array of pixels), but movies in any other form are treatable as well. In these cases, the movie is translated to a digital form by existing techniques, manipulated in the digital form, and returned back to any desired form by known techniques.

[0053] A facial expression can be described by an “action unit”, for example the Facial Action Coding System (FACS) by Ekman and Friesen (Ekman et. al.). Action units (AU) stands for a small change in the facial expression which depends on a conscious activation of muscles (H. Li, P. Roivainen, R. Forchheimer, 3-D Motion in Model-Based Facial Image Coding, IEEE Transactions in PAMI, 15 (2), 545-555 (1993)). The AU information is expressed in parameter form. Using the AU parameters, many facial expressions can be controlled. Parameters like face location and size, aspect-ratios of face regions, location of specific face features and many more.

[0054] As explained above, one of the stages of this invention is a three-dimensional parameterisation of the face. An example for one such model is the model of Parke (Fredric I. Parke, Parameterized Models for Fatial Animation, IEEE computer Graphics and Applications, 12 (11), 61-68, (1982)) which consists of about 25 parameters. Face parameters can be roughly divided into three main classes: structure parameters, location parameters, and expression parameters.

[0055] Structure parameters are fixed for every head and include distance ratios between the mouth and the eyes, the mouth and the chin, width of the model, jaw width, etc. The location parameters are, for example: three parameters for three-dimensional rotation in space and three parameters for three-dimensional translation (position in the real world). The expression parameters are, for instance: mouth width, smile (as an example, the parameter values here may be 0.0 for a very sad mouth and 1.0 for a very happy mouth), jaw opening, upper lip lifting, lower lip lowering, lip thickness, and so on.

[0056] Using a face model, the present invention is centered on a computer program (see Appendix 1), which automatically re-shapes the lip movements of the actor according to the lip movements of the dubber by searching for the nearest reference similarity frames. This computer program (software), or similar, is an integral and an important part of the present invention. The process, according to the present invention, is divided generally into the TRACKING phase and the NEW-MOVIE generation phase, as follows:

[0057] I. TRACKING phase

[0058] Step 1: The first step is personalizing the generic three-dimensional face model for both the actor and the dubber. In order to modify the generic face model to fit a specific face, some additional information is needed. The generic model has to be translated, scaled and stretched to fit the given actor's face, from its initial position and setting. This is done by manually pointing using a pointing device such as a mouth, a touch screen, etc., several facial feature points on the actor's face, e.g. eye corners, mouth corners, top and bottom of face. Typically a total of approximately 15 feature points are used, but this number may vary according to specifications. These feature points are marked on one (any) of the frames in the movie, in which the actor, preferably, faces the camera. The computer program then calculates automatically the exact model parameter's modifications needed for its two-dimensional projection to fit the actor face on the movie frame. In addition to using the facial feature points and in order to increase accuracy, the model is also adjusted to match the head edges, which are computed using an edge detector. If a side view of the actor is available it can be used to set several depth parameters, such as face depth and nose length. Otherwise, the face depth is scaled by some predetermined scale which is set experimentally.

[0059] Step 2: After the generic face model has been personalized to the desired actor, face features in several key frames in the movie are marked. The number of such frames can vary from a single first frame to about 5% of all frames, depending on the difficulty of the segment fitting the model to the actor using the marked facial features in those key frames achieves a stabilization of the automatic tracking (described later), and these key frames assures stable and continuous tracking. Next, the program calibrates according to several examples of mouth shapes, later to be used for mouth tracking. Finally, the range of the mouth parameters (mimimum and maximum values) for the specific actor are estimated using all the values of the model parameters fitted to all key frames.

[0060] Step 3: The next stage is the automatic tracking of the actor's face throughout the entire movie: This is performed from one frame to its successive frame, using the face model, in two steps: first, the two-dimensional face of the actor is mapped onto the three-dimensional face model using a texture-mapping technique. The model can now be altered by changing its parameters only, creating new, synthetic images, which are otherwise very similar to the original movie frames: everything remains unchanged except for different face location, its orientation, and its expressions. By using a minimization algorithm, either analytic or numerical (such as steepest descent algorithm), the program now computes those parameters which maximize the correlation function between the face area of the actor in the next frame and the synthesized projection of the texture-mapped face model. The steepest descent algorithm increases or decreases parameters in the direction that increases the correlation function. It can either work for each parameter separately (until it maximizes the correlation), or it can modify all the parameters at once.

[0061] Step 4: After the model is locked on the head of the actor in the next frame, the mouth has to be tracked. This is done by first, checking the parameters of all of the mouths in the key frames and in several previous frames already tracked. Then, the frame that gives the higher correlation is choosen as a first guess for the tracking. Next, the same minimization algorithm used to track the global head motion is used, until the correlation function has maximized. The parameters describing the face model in the tracked frame are written into a file for later use.

[0062] Step 5: Steps 3 and 4 are repeated until the entire movie is processed. For best results, instead of executing this process serially from the first frame to the last, the key frames can be used as initial points of tracking. Every two consecutive key frames are used to track from each of them to the frames between them. That way, stabilasation of the tracking is preserved.

[0063] Step 6: The tracking described above is applied to the dubber movie as well.

[0064] II. NEW MOVIE generation phase

[0065] This phase combines the tracking results of both the original and the dubber's movies, in order to synthesize the new audio visually dubbed movie. This audio visually dubbed movie, as explained above, is mostly formed out of the original movie, except for the face of the actor in this audio visually dubbed movie. This face is a texture-mapped face on the three-dimensional face model, synthesized, as described above, to fit the lip, mouth, and cheek shapes of the dubber at that particular time. Thus, the parameters of the face model computed as described in phase I, is used to produce the new audio visually dubbed movie, in which for every frame in the original movie, the mouth parameters are modified to those of the dubber. The exact process is as follows:

[0066] Step 7: For every frame in the original movie, the face of the actor is texture-mapped on the appropriate face model using the parameters that were calculated in step 3 for the original movie. The mouth parameters of the dubber as calculated in step 3 are used as follows for the new audio visually dubbed movie.

[0067] Step 8: Once the desired mouth-shape of the actor is known, the original movie is being searched in the neighborhood of the current frame (approximately 0.1-10 seconds forwards and backwards in time) for a mouth that is most similar in shape or parameters to the new desired mouth. This search for the reference similarity frame takes into account the mouth-shape already chosen for the previous frame in order to make the mouth motion smooth and continuous. From the several (5-10) best fit mouths, the mouth which is picked is from the frame that is closest in time to the previous picked mouth.

[0068] Step 9: The mouth chosen in step 8 is texture-mapped into the mouth model using its pre-computed parameters. The face model parameters are then changed to the desired mouth shape, producing a very realistic new frame, which replaces the old frame in the original movie. The user of the program can choose the desired mouth area to be texture-mapped in place—it can be either the inside of the mouth, the whole mouth including the lips or even a bigger area. This procedure creates a synthesized image, in which the face around the mouth, and in particular the lips, are re-shaped according to the sound track, while retaining the familiar face of the original actor. Step 8 can also be skipped, so that the inside of the mouth will be empty. This is useful for making a talking movie from still picture, where the inside information of the mouth is missing; because in the reference frame similarity dictionary of the actor there does not exist any near fit of lip shapes for the target. This black interior can also be filled with visual color/texture.

[0069] Step 10: Finally, the sound track from the dubbed movie (target text) replaces the original text and sound.

[0070] It is to be noted that an animator using the invented software tool, is free to modify, set, or fix any of the head or mouth parameters, in both the original or audio visually dubbed movie, and even pick a specific mouth to be textured-mapped in place, as described in step NM8, all of these at any of the above stages. The tracking program is highly interactive and user-friendly.

[0071] The involved software (see Appendix 1) of this invention is very versatile, and can be used in a very wide array of sound/text replacement applications many of which have been mentioned above. The following are examples of some applications of the present invention:

[0072] Advertising: For products sold world-wide, an original advertising commercial can be manipulated to produce the same commercial in any desired language. This saves the need to produce a new video for every country or language that the product is aimed for.

[0073] Another possibility is to edit a movie, by altering existing scenes without having to re-shoot them again. If, for example, after the movie production was over, the director/editor wishes to alter a specific scene, or change one sentence of a specific actor.

[0074] The present invention refers not only to narrated text but also to songs, operas, and music, opening the possibility to change the language of musical video clips.

[0075] The production of an animated cartoon is assisted by drawing a line segment for the actor's mouth, drawing a small actor's picture dictionary containing representive reference similarity frames with completely drawn mouths, and then allowing these lip-line segments to be replaced with the corresponding lip shapes of the dubber as are to be found in the actor's picture dictionary.

[0076] In general, applications of a method for automatic audio visual dubbing of movie include: cinematic movies, cartoons, documentaries, advertizing, news, educational programs, court documentations, speaches, lectures, historical documentation, hearing committees, home videos, sports events, entertainment events, operas, musicals, musical video-clips, simultaneous translation, and adding speach to sequences of either original or added still frames of the aforesaid.

[0077] Furthermore, using the library of the previously described reference similarity frames, the present invention allows one to create new movies altogether, and also to convert background narrative to audio visual speach, and to audio-visualize any written text.

[0078] The present invention will be further described by FIGS. 1-4.

[0079] FIG. 1 illustrates a block diagram showing the method stages, and its contents is detailed below.

[0080] FIGS. 2aand 2billustrate an example of significant points on a generic frontal picture of a head (FIG. 2a) and a generic side profile picture of a head (FIG. 2b).

[0081] FIG. 3 shows an example of a generic wire frame face model.

[0082] For illustrative purposes one can take the significant points shown in FIG. 2, measure them on pictures of a real actor, and apply them to a generic wire frame face model (FIG. 3). Fitting a three-dimensional head model to the actor's two-dimensional head picture by adapting the data of the significant points, as measured, results in an integration, as can be seen in FIGS. 4aand 4b.

[0083] FIG. 4ais an example showing how a custom fitted wire frame model fits onto a frontal view of an actor's face.

[0084] FIG. 4bis an example showing how a custom fitted wire frame model fits onto a profile view of an actor's face.

[0085] FIG. 1 illustrates a block diagram showing the method stages:

[0086] In the original movie (1) a frame is selected (2) having a nearly frontal picture of the original actor's head and, if available, a frame is also selected having a picture with his side profile.

[0087] A three-dimensional head model is fitted to the actor's two dimensional head picture(s). This model can be controlled by several parameters such as for the position of the head and the status of the mouth. This fitting stage (3) is done by adapting the data of the significant points, as measured in the selected frame, to the model.

[0088] The next step (4) is the automated computerized tracking of the fitted three-dimensional head model parameters throughout the movie, from one frame to the next. One partial or complete three-dimensional head model for each frame where the actor appears, is used. Any of the resulting frames can serve as a reference similarity frame for the lips replacement.

[0089] A movie of the dubber is taken (5). In this movie the dubber faces the camera in most of the frames. The dubber speaks the target text in this movie.

[0090] The same process as applied to the original actor's movie is applied to the dubber's movie: A frame form the dubber's movie is selected (6) having a frontal picture of the dubber's head, and if available, a frame with a picture of his side profile. A three dimensional head model is fitted to the dubber's two dimensional head picture (7) by adapting the data of the significant points, as measured in the selected frame, to the dubber's model. An automated computerized tracking (8) of the said dubber's fitted three dimensional head model parameters is taken throughout the movie, from one frame to the next.

[0091] The next stage in this method is to normalize (9) the dubber's minimum and maximum parameters to the actor's minimum and maximum parameters.

[0092] In a frame to frame fashion, the original actor's two dimensional face is mapped (10) onto his three dimensional head model. This mapping stage is done by using a texture mapping technique with the reference similarity frames. The result of this stage is one mapped three dimensional partial head model of the original actor for each frame of the original actor in the original movie; wherein the model, corresponding to a given frame may be complete in the case when the original frame contains a frontal view of the actor's face.

[0093] In the next stage the textured three dimensional model frames obtained for the original actor are changed (11) by replacing, on a frame to frame basis, the original mouth parameters with mouth parameters as computed for the dubber in the corresponding frames; correspondence being determined by the desired sound track substitution (i.e. the dubbing). Thus is obtained the parametric description for a new picture identical to the original, except that the actor's mouth status resembles the mouth status of the dubber; wherein the new picture corresponds to a frame in the new audio visually dubbed movie.

[0094] In order to overcome difficulties like for example those that arise when the dubber in (8) opens a closed mouth in (4), a frame or frames in the original movie are sought whose mouth status is similar to the desired new mouth status. These frames, termed reference similarity frames, are usually but not necessarily in a temporal proximity to the processed frame, and the lips from that frame are copied using texture mapping (12) into the lips area in the new frame. The search for a reference similarity frame is an essential component of the present invention. We therefore repeat its definition: a reference similarity frame is a picture (being a frame in the original movie, a frame in any other movie, or a still photograph) in which the original actor has the desired features of the mouth-shape and head posture suitable for the audio visually dubbed movie.

[0095] Alternatively, the reference similarity frame may be taken from a compiled picture library of the original actor or of other actors.

[0096] The process (12) is repeated all over again for each frame, until the entire movie is converted.