US 7,912,337 B2Grant
Spatial and temporal alignment of video sequences
Issue Date:2011-03-22
•29 Claims
•25 Drawing Sheets
Abstract
Some embodiments allow a video editor to spatially and temporally align two or more video sequences into a single video sequence. As used in this application, a video sequence is a set of images (e.g., a set of video frames or fields). A video sequence can be from any media, such as broadcast media or recording media (e.g., camera, film, DVD, etc.). Some embodiments are implemented in a video editing application that has a user selectable alignment operation, which when selected aligns two or more video sequences. In some embodiments, the alignment operation identifies a set of pixels in one image (i.e., a “first” image) of a first video sequence and another image (i.e., a “second” image) of a second video sequence. The alignment operation defines a motion function that describes the motion of the set of pixels between the first and second images. The operation then defines an objective function based on the motion function. The operation finds an optimal solution for the objective function. Based on the objective function, the operation identifies a transform, which it then applies to the first image in order to align the first image with the second image.
Metadata
Assignee
- Apple Inc.
Inventor
- Christophe Souchard
Application Information
Application Number:US 11/266,101
Filing Date:2005-11-02
Priority Date:2005-11-02
Art Unit:2621
Classifications
IPC:
H04N5/93H04N5/00
Patent Drawings (25 sheets)
Description
Background of the Invention
[0001] High quality video photography and digital video photography equipments are increasingly accessible to a broad range of businesses and individuals, from movie production studios to average consumers. Many of these equipments are not capable of recording wide angle video footages (i.e., panoramic video footages). Wide angle video footages are advantageous over normal angle video footages because they include more visual detail than normal video footages. However, those equipments that are capable of recording wide angle video footages are often very expensive. Thus, the recording of wide angle video footage is economically not practical for many users.
[0002] Therefore, there is a need in the art for a practical and economical method for recording and/or producing wide angle video footages. Ideally, such a method can be performed by a video editing application that can align two or more video footages to produce a wide angle video footage, even when the two or more recorded video sequences were recorded at different positions, angles and/or have different movements. Ideally, such a method would provide a method that blends the recorded video footage into a seamless panoramic video footage (i.e., the boundaries that overlap each recorded video footage are not seen).
[0003] Furthermore, many of the video and digital video equipments have limited dynamic range when recording a video footage. In other words, many of today's video and digital video equipments have limited range when recording the contrast of scenes (i.e., range between the lightest highlight and darkest shadow in the scene). Therefore, these equipments do not record as much detail as those equipments with higher dynamic range. However, high dynamic range equipments cost substantially more than limited dynamic range equipments. Thus, there is need in the art for a practical method for increasing the dynamic range of recorded video sequences.
Brief Summary of the Invention
[0004] Some embodiments allow a video editor to spatially and temporally align two or more video sequences into a single video sequence. As used in this application, a video sequence is a set of images (e.g., a set of video frames or fields). A video sequence can be from any media, such as broadcast media or recording media (e.g., camera, film, DVD, etc.).
[0005] Some embodiments are implemented in a video editing application that has a user selectable alignment operation, which when selected aligns two or more video sequences. In some embodiments, the alignment operation identifies a set of pixels in one image (i.e., a “first” image) of a first video sequence and another image (i.e., a “second” image) of a second video sequence. The alignment operation defines a motion function that describes the motion of the set of pixels between the first and second images. The operation then defines an objective function based on the motion function. The operation finds an optimal solution for the objective function. Based on the objective function, the operation identifies a transform, which it then applies to the first image in order to align the first image with the second image.
[0006] In some embodiments, the operation defines the motion function based on a motion model. Also, in some embodiments, the operation specifies a set of constraints and then finds an optimal solution for the objective function by optimizing the objective function based on the set of constraints. In some embodiments, the set of constraints is based on an optical flow constraint equation.
Brief Description of the Drawings
[0007] The novel features of the invention are set forth in the appended claims. However, for purpose of explanation, several embodiments of the invention are set forth in the following figures.
[0008] FIG. 1 illustrates a three stage process for creating a motion function to transform a frame.
[0009] FIG. 2 illustrates an error distribution with outliers.
[0010] FIG. 3 illustrates a method for aligning frames in video sequences.
[0011] FIG. 4 illustrates a frame in a first video sequence.
[0012] FIG. 5 illustrates a frame in a second video sequence.
[0013] FIG. 6 illustrates a frame in the second video sequence that is matched to a first frame in the first video sequence.
[0014] FIG. 7 illustrates a frame in the second video sequence that is matched to a second frame in the first video sequence.
[0015] FIG. 8 illustrates a frame produced from two aligned frames from different video sequences.
[0016] FIG. 9 illustrates a mask region used to exclude certain pixels of the aligned frame.
[0017] FIG. 10 illustrates a frame produced from a mask region and two aligned frames from different video sequences.
[0018] FIG. 11 illustrates a method for aligning two or more images to produce a panoramic image.
[0019] FIG. 12 illustrates a first image in a graphical user of a video editing application.
[0020] FIG. 13 illustrates a second image in the graphical user of the video editing application.
[0021] FIG. 14 illustrates a third image in the graphical user of the video editing application.
[0022] FIG. 15 illustrates a panoramic image based on the images of FIGS. 12-14 .
[0023] FIG. 16 illustrates a panoramic image based on the images of FIGS. 12-14 , where the boundary lines between the images are removed.
[0024] FIG. 17 illustrates a panoramic image based on the images of FIGS. 12-14 , where the boundary lines between the images are removed and the illumination of the images is matched.
[0025] FIG. 18 illustrates a panoramic image based on the images of FIGS. 12-14 , where the boundary lines between the images are removed and the images are locked to a first image.
[0026] FIG. 19 illustrates a panoramic image based on the images of FIGS. 12-14 , where the images are locked to a second image.
[0027] FIG. 20 illustrates an example of distances used to compute the weighted value of the pixels.
[0028] FIG. 21 illustrates a method for producing a high dynamic range image from a set of images.
[0029] FIG. 22 illustrates an image that is underexposed.
[0030] FIG. 23 illustrates an image that is overexposed.
[0031] FIG. 24 illustrates an aligned high dynamic range image.
[0032] FIG. 25 illustrates a computer system that can implement the methods described.
Detailed Description of the Invention
[0033] In the following description, numerous details are set forth for purpose of explanation. However, one of ordinary skill in the art will realize that the invention may be practiced without the use of these specific details. In other instances, well-known structures and devices are shown in block diagram form in order not to obscure the description of the invention with unnecessary detail.
I. Spatial and Temporal Alignment
[0034] A. Overview
[0035] Some embodiments allow a video editor to spatially and temporally align two or more video sequences into a single video sequence. As used in this application, a video sequence is a set of images (e.g., a set of video frames or fields). A video sequence can be from any media, such as broadcast media or recording media (e.g., camera, film, DVD, etc.).
[0036] Some embodiments are implemented in a video editing application that has a user selectable alignment operation, which when selected aligns two or more video sequences. In some embodiments, the alignment operation identifies a set of pixels in one image (i.e., a “first” image) of a first video sequence and another image (i.e., a “second” image) of a second video sequence. The alignment operation defines a motion function that describes the motion of the set of pixels between the first and second images. The operation then defines an objective function based on the motion function. The operation finds an optimal solution for the objective function. Based on the objective function, the operation identifies a transform, which it then applies to the first image in order to align the first image with the second image.
[0037] In some embodiments, the operation defines the motion function based on a motion model. Also, in some embodiments, the operation specifies a set of constraints and then finds an optimal solution for the objective function by optimizing the objective function based on the set of constraints. In some embodiments, the set of constraints is based on an optical flow constraint equation.
[0038] To align several images in a first video sequence with several other images in a second video sequence, some embodiments first compare at least one particular image in the first video sequence with several images in the second video sequence. Each comparison entails identifying a motion function that expresses the motion of a set of pixels between the particular image in the first video sequence and an image in the second video sequence. Some embodiments might examine different sets of pixels in the particular image when these embodiments define different motion functions between the particular image and different images in the second video sequence.
[0039] For each or some of the defined motion functions, some embodiments define an objective function, which they then optimize. Based on the optimal solution of each particular objective function, some embodiments then define a transform operation, which they apply to the particular image in order to align the particular image to an image in the second video sequence. These embodiments then select the image in the second video sequence to which they align the particular image by identifying the transform operation that resulted in the best alignment of the particular image to an image in the second video sequence. These embodiments then align the particular image in the first video sequence with the selected image in the second video sequence by applying the corresponding transform that these embodiments identified for this pair of images. To align several images in a first video sequence with several other images in a second video sequence, some embodiments compare each particular image in the first video sequence with several images in the second video sequence.
[0040] The alignment operation will now be further described by reference to FIGS. 1-10 . In these figures, the alignment operation is part of a video compositing application that allows a video editor to perform the alignment operation on video frames of two or more video sequences. FIG. 1 conceptually illustrates a three-stage process 100 that some embodiments perform to align two or more video sequences.
[0041] In the first stage, the process 100 performs (at 105) a classification operation that identifies a set of pixels to track between the video sequences. In some embodiments, the process might define (at 105) different sets of pixels to track for different pairs of frames in the different video sequences. In other embodiments, the process might identify (at 105) a set of pixels in each frame of one video sequence to track in any other frame of another video sequence. In some embodiments, the classification operation identifies only sets of pixels that have a spatial frequency above a particular value (e.g., high spatial frequency values).
[0042] In the second stage, the process 100 estimates (at 110) the motion between the sets of pixels in one video sequence and the sets of pixels in at least one other video sequence. To identify the motion, the process 100, as described above, identifies the motion and objective functions and optimizes the objective functions in view of constraints.
[0043] The motion between each pair of frames is expressed in terms of a transform. In the third stage, the process 100 aligns (at 115) the video sequences by applying the identified transforms to the frames of at least one video sequence to align the video sequence with at least one other video sequence.
[0044] B. Computing an Optimal Motion Function to Transform Frame
[0045] As described above, the alignment operation uses a motion function to transform a frame in a first video sequence and aligns the transformed frame to another frame in a second video sequence. In some embodiments, computing the motion function includes three stages: (1) Pixel Classification process, (2) Correspondence process, and (3) Motion Function process.
[0046] 1. Pixel Classification Process
[0047] As mentioned above, the alignment operation of some embodiments automatically selects a set of pixels to track. The pixels that are selected for tracking are pixels that might be of interest in the frames of the video sequence. Not all parts of each image contain useful and complete motion information. Thus, these embodiments select only those pixels in the image with high spatial frequency content. Pixels that have high spatial frequency content include pixels from comers or edges of objects in the image as opposed to pixels from a static monochrome, or white, background. Selecting only pixels with high spatial frequency content (i.e., useful for performing motion estimation), optimizes a pixel correspondence process that will be described next. Some embodiments can select a different set of pixels for the motion analysis of each pair of frames.
[0048] 2. Correspondence Process
[0049] During the correspondence process, an estimate of a motion flow between the set of pixels for each pair frames in the video sequences is computed. The estimate of the motion flow is computed by collecting constraints about points (e.g., pixels) around each pixel in the set of pixels. The correspondence process solves a mathematical expression using the collected constraints to compute an estimate of the motion flow between each set of pixels.
[0050] To define a set of constraints, some embodiments use the classical optical flow constraint equation:
framex*u+framey*v+framet=0 (equation 1)
framex*u+framey*v+framet=0 (equation 1)
[0051] where (u,v) are unknown components of the flow, and subscripts x, y, and t indicate differentiation.
[0052] By using the optical flow constraint equation to collect constraints of neighboring points and solve the resulting over-constrained set of linear equations, some embodiments exploit the information from a small neighborhood around the examined pixel to determine pixel correspondence between frames. The set of pixels applied to the constraint equations was selected for each pixel's optimum motion estimation properties by the pixel classification process above. Thus, the selected set of optimal pixels avoids the classical ill-condition drawback that typically arises when using local motion estimation techniques. The correspondence process generates a motion flow to represent the flow field between each pair of frames in the video sequences.
[0053] 3. Motion Function Process
[0054] For each pair of frames, some embodiments (1) define a motion function that expresses the motion between the frames in the video sequences, and (2) based on the motion function, define an objective function that expresses the difference between the two frames in the video sequences. For each objective function, these embodiments then try to find an optimal solution that will fit the flow-field constraints defined for that function.
[0055] In some embodiments, the motion function that expresses the motion between two frames X and Y, can be expressed as:
M(X)=Mo(X)*Pa (equation 2)
M(X)=Mo(X)*Pa (equation 2)
[0056] Here, M(X) is the function that expresses the motion between the frames X and Y, Mo(X) is the motion model used for expressing the motion between the two frames in the video sequences, and Pa represents the set of parameters for the motion model, which, when defined, define the motion function M(X). In other words, the motion model Mo(X) is a generic model that can be used to represent a variety of motions between two frames. Equation 2 is optimized in some embodiments to identify an optimal solution that provides the values of the parameter set Pa, which, when applied to the motion model, defines the motion function M(X).
[0057] In some embodiments, the motion model Mo(X) can be represented by an m-by-n matrix, where m is the number of dimensions and n is the number of coefficients for the polynomial. One instance of the matrix Mo(x) and the vector Pa are given below:
[0058]
[0059] In the example above, the motion model has two rows to indicate motion in the x-axis and motion in the y-axis.
[0060] As illustrated above, some embodiments may base the parametric motion model on a two dimensional polynomial equation represented by a two-by-twelve matrix and twelve corresponding vector coefficients. These embodiments provide the advantage of accurate motion model estimation within a reasonable computation time. The motion estimation model of other embodiments may be based on different (e.g., multi) dimensional polynomial equations. However, one of ordinary skill will recognize that polynomials having additional dimensions may require tradeoffs such as increased processing time.
[0061] Some embodiments use the motion function defined for each particular pair of frames to define an objective function for the particular pair of frames. The objective function is a sum of the difference in the location of the identified set of pixels between the two frames after one of them has been motion compensated based on the motion function. This objective function expresses an error between a motion-compensation frame (M(X)) in the pair and the other frame (Y) in the pair. By minimizing this residual-error objective function, some embodiments identify a set of parameters Pa that best expresses the motion between frames X and Y. Through the proper selection of the set of pixels that are analyzed and the reduction of the set of pixels that adversely affect the optimization of the objective function, some embodiments reduce the consideration of content motion between the pair of frames.
[0062] Equation 3 illustrates an example of the objective function R of some embodiments, which is a weighted sum of the difference between each pair of corresponding pixels (PY,i, PX,i) in a pair of successive frames after one pixel (PX,i) in the pair has been motion compensated by using its corresponding motion function.
[0063]
[0064] In this equation, i is a number that identifies a particular pixel, Num_P is the number of pixels in the set of pixels being examined, and Ci is a weighting factor used to value the importance of the particular pixel i in the motion analysis.
[0065] Some embodiments try to optimize the objective function of Equation (3) through two levels of iterations. In the first level of iterations, the alignment operation explores various different possible camera movements (i.e., various different sets of parameter values Pa) to try to identify a set of parameter values that minimize the objective function R, while meeting the set of defined optical flow constraints.
[0066] In the second level of iterations, the alignment operation changes one or more weighting factors Ci and then repeats the first level of iterations for the new weighting factions. The second level of iterations is performed to reduce the affect of outlier pixels that improperly interfere with the optimization of the objective function R. In other words, the first level of iterations is a first optimization loop that is embedded in a second optimization loop, which is the second level of iterations.
[0067] For its second level of iterations, the alignment operation uses a weighted least squares fit approach to adjust the weighted coefficients Ci. In some embodiments, the weights of all coefficients initially have a value of one, and are re-adjusted with each iteration pursuant to the adjustments illustrated in Equation 4 below.
[0068]
[0069] The motion model estimation process of these embodiments accepts or rejects each pixel based on its error (i.e., its parametric motion estimation) by adjusting the error coefficient weightings over the course of several iterations. The motion estimation process ends its iterative optimization process when the desired residual error R is reached or after a predetermined number of iterations. The iterative nature of the motion model estimation process and the accurate estimation of the error coefficients (Ci) allow the process to accurately estimate the motion in pixels between images even in the presence of outlier points that deviate significantly in the error distribution (e.g., even in the presence of object motion in the video sequence).
[0070] For instance, FIG. 2 illustrates the concept behind a weighted least square fit approach to eliminating outliers from a set of analysis point. In this distribution, the majority of the analysis pixels group approximately along the least square fit line 205, while some analysis pixels are far from this line, and these pixels are the outliers pixels (e.g., pixels associated with content motion between the two frames). FIG. 2 is only a conceptual illustration, as the least squares fit analysis is performed on more than one dimensions (e.g., on twelve dimensions associated with the twelve parameter values).
[0071] As further described below, the motion analysis of the first stage eliminates or reduces the influence of “outlier” pixels in the set of pixels that interfere with the motion analysis. Such outlier pixels have motion that if accounted for in the analysis would interfere with the motion analysis. In other words, the motion of these outlier pixels differs significantly from the motion of the other pixels in the selected pixel set. This differing motion might be due to the fact that the outlier pixels are part of objects that are moving in the scene(s) capture by the video sequence (i.e., of objects that have a desired movement in the sequence and that are not due to undesired camera movement). Outlier pixels might also be due to illumination changes. Previous video editing tools in the art assumed fixed lighting conditions. However, new cameras have automatic exposure and other features that affect the parameter values, such as illumination or lighting.
[0072] Thus, the first stage of some embodiments for motion estimation is robust, meaning that these embodiments distinguish between moving objects and illumination changes in the video sequence. Also, some embodiments allow the user to specify mask regions in order to exclude certain image areas from the transformation.
II. Alignment of Multiple Video Sequences
[0073] FIG. 3 illustrates a method 300 that some embodiments perform to spatially and temporally align two video sequences into a single video sequence. In some embodiments, the method 300 is used to combine two video sequences to produce a panoramic video sequence (i.e., wide angle video sequence).
[0074] As shown in this figure, the method 300 selects (at 305) a particular frame in the first video sequence. The method 300 selects (at 310) a frame in the second video sequence. The method 300 identifies (at 315) a set of pixels in the selected frames of the first and second video frames. Specifically, the method 300 identifies (at 315) a set of pixels in the frame in the second video sequence that best matches a set of pixels in the particular frame in the first video sequence. In some embodiments, the method 300 identifies (at 315) a set of pixels in the selected frames that has a spatial frequency content above a particular value (e.g., high spatial frequency content value).
[0075] The method 300 defines (at 320) a motion function based a motion model. To find an optimal motion function, the method 300 defines (at 325) an objective function based on the motion function. In some embodiments, the objective function is a weighted sum of the difference between each pair of corresponding pixels in a pair of frames after one pixel in the pair has been compensated by using its corresponding motion function (e.g., equation 2). Next, the method 300 finds (at 330) an optimal solution for the objective function on a set of constraints. In some embodiments, the optimal solution for the objective function is found by optimizing the objective function through several levels of iterations, such as the one described in Section. I.B.3. Once the optimal solution is found (at 330), the method 300 identifies (at 335) an optimal motion function between the particular pair of frames based on the optimal solution for the objective function.
[0076] Next, the method 300 determines (at 340) whether there is another frame in the second video sequence. If so, the method 300 proceeds to select (at 310) another frame in the second video sequence. In some embodiments, the steps 310-335 are iteratively performed until all the frames in the second video sequence have been selected. Therefore, the method 300 iteratively identifies a particular optimal motion function for each particular pair of frames in the video sequences.
[0077] If the method 300 determines (at 340) there is no other frame in the second video sequence, for the particular frame in the first video sequence, the method 300 selects (at 345) an optimal motion function that has the lowest objective function value from the set of identified motion function. In other words, the method 300 selects (at 345) the motion function that produces the lowest error between each pair of corresponding pixels in the frames of the first and second video sequences.
[0078] The method 300 defines (at 350) a transform based on the selected motion function for the particular frame in the first video sequence. Next, the method 300 applies (at 355) the defined transform to the particular frame in the first video sequence to align the particular frame to a corresponding frame in the second video sequence. The corresponding frame in the second video sequence is based on the pair of frames that defined the selected (at 345) motion function. For example, if the selected motion function was computed from the second frame in the first video sequence and the fifth frame in the second video sequence, then the transformed second frame in the first video sequence would be aligned to the fifth frame in the second video sequence.
[0079] After aligning the pair of frames (by applying the transform), the method 300 determines (at 360) whether there is another frame in the first video sequence. If so, the method 300 proceeds to 305 to select another frame in the first video sequence. If not, the method 300 ends. In some embodiments, several iterations of the above method 300 are performed until all the frames in the first and second video sequences are selected.
[0080] In some embodiments, some or all of the steps of selecting an optimal motion function, defining a transform, and applying the transform to align the frames are performed after the motion functions have been computed for all possible pairs of frames in the first and second video sequences. In other words, some or all the steps at 345, 350 and/or 355 are performed after determining (at 360) there are no other frames in the first video sequence. In such instances, a list of possible pairs of frames is generated, along with its corresponding motion function and objective function value. From this list, the method 300 (1) selects a corresponding motion function for each frame in the first video sequence based on the objective function value, (2) defines a transform for each frame in the first video sequence based on the corresponding motion function, and (3) applies the transform to the particular frame to align the particular frame to its corresponding frame in the second video sequence.
[0081] In some embodiments a particular transform is defined for each pair of corresponding frames, while other embodiments define one universal transform for all pairs of associated frames. Moreover, some embodiments compute motion functions for several pairs of frames between the first and second video sequences to determine which particular pair of frames should be the first pair of frames. Some embodiments then sequentially associate subsequent frames in the first and second sequences based on the determined first pair of frames. Thus for example, if the fourth frame of the first video sequence is associated to the ninth frame of the second video sequence, the fifth frame of the first video sequence is associated to the tenth frame of the second video sequence, and so on and so forth. In such instances, a particular transform is used for each pair of frames, where the transform is based on the motion function for that particular pair of frames. In some instances, one universal transform may be used for all pairs of frames, where the universal transform is based on the motion function of the first pair of frames.
[0082] Having described a method for spatially and temporally aligning video sequences, an implementation of the method in a video editing application will now be described. FIGS. 4-10 illustrates a graphical user interface of a video editing application that can perform the method 300. FIG. 4 illustrates a first frame 405 in a first video sequence. FIG. 5 illustrates a first frame 505 in a second video sequence. The frames 405 and 505 show a similar scene, except that some of the objects (e.g., person, background) in the frames are not in the same position.
[0083] FIG. 6 illustrates a frame 605 in the second video sequences that best matches the first frame 405 of the video sequence. In some embodiments, this particular frame 605 from a set of frames in the second video sequences best matches the first frame 405 because the particular frame 605 has the lowest objective function value. FIG. 7 illustrates another frame 705 from the set of frames in the second video sequence that best matches a second frame in the first video sequence. FIG. 8 illustrates a frame 805 based on frames 810 and 815 in the first and second video sequences. As shown in this figure, the frame 810 from the first video sequence is aligned to the frame 815 from the second video sequence to produce a composited frame 805.
[0084] As shown in FIG. 8 , the composited frame 805 has an uneven boundary. The uneven boundary occurs because the frames do not completely overlap each other. Some embodiments specify a mask region to remove the uneven boundary, such as the mask region 905 shown in FIG. 9 . In some embodiments, the mask regions exclude certain frame areas from the motion function transformation. Thus, certain pixels in the frames are not transformed. FIG. 10 illustrates the composited frame 805 after the mask region 905 is specified.
[0085] The above alignment operation describes spatially and temporally aligning two video sequences. However, the alignment operation can also be used to align more than two video sequences. Furthermore, the above alignment operation can be used to align images that are not part of a video sequence.
III. Alignment of Images to Produce a Panoramic Image
[0086] FIG. 11 illustrates a method 1100 for aligning several images to produce a panoramic image (i.e., wide angle image). Specifically, the method 1100 will be described by reference to FIGS. 12 to 19 , which illustrate a graphical interface of a video application editor that can perform the method 1100 for producing a panoramic image from three different images.
[0087] As shown in FIG. 11 , the method 1100 classifies (at 1105) a set of pixels in a first image. As mentioned above, during this pixel classification process (at 1105), the method 1100 selects a set of pixels in each image of the set of images for tracking. In some embodiments, the identified set of pixels only includes pixels with high spatial values.
[0088] After the pixel classification process is performed (at 1105), the method 1100 identifies (at 1110) constraints for the identified set of pixels. In some embodiments, the constraints are identified by using a classical optical flow constraints equation.
[0089] Next, the method 1100 identifies (at 1115) a motion function for each pair of images in the set of images by optimizing an objective function for each pair of images. Each motion function is based on a motion model, such as one described in Section I.B.3. The motion function for a particular pair of images expresses a motion difference between the particular pair of images. Thus, when the set of images includes three images 1205-1215, some embodiments identify a motion function for the set of pixels in the first and second images 1205 and 1210, the first and third images 1205 and 1215, and the second and third images 1210 and 1215.
[0090] To identify the motion function for a particular pair of images, the method 1100 optimizes (at 1115) an objective function based on the set of constraints that was previously identified (at 1110). In some embodiments, identifying the motion function includes identifying the parameters of the motion function.
[0091] Next, the method 1100 defines (at 1120) a transform for a particular image based on the parameters of the motion functions that were identified (at 1115) for a particular pair of images that includes the particular image. During some embodiments, the method 1100 (at 1115) first determines which image from the set of images is the locked image. In some embodiments, the locked image is the reference image that other images will be aligned to during the alignment process, which will be described next. Once the locked image is determined, the method 1100 defines a transform for each particular image in the set of images based on the identified motion function corresponding to the particular image and the locked image.
[0092] After the transform has been defined (at 1120), the method 1100 applies (at 1125) the defined transform to align a particular image to another image and ends. In some embodiments, the method 1100 applies (at 1125) a particular transform to each particular image to align each particular image to the locked image.
[0093] FIG. 15 illustrates a panoramic image that was produced based on the three images of FIGS. 12-14 . As shown in this figure, the second image 1210 is the locked image. As such, the first image 1205 and the third image 1215 are defined transforms based on their respective motion function with the second image 1210. The defined transforms are then applied on their respective images 1205 and 125 to align the first and third images 1205 and 1215 with the second image 1210. As further shown in this figure, the boundary lines where the first, second and third images meet are shown.
[0094] FIG. 16 illustrates the same panoramic image as in FIG. 15 , except that the boundary lines in the panoramic image has been removed through the use of a blend operation. In some embodiments, the blend operation is an operation that determines regions in the panoramic image that contain overlapping images. Some embodiments blend these overlapping regions by taking a weighted value of the pixels relative to each pixel's distance from the boundary lines. Some embodiments use the following equation to take the weighted value of the pixels:
[0095]
[0096] In this equation, P1 and P2 are the pixel values at a particular location in the first and second images, D1 and D2 are each pixel's respective distances from the boundary lines. FIG. 20 illustrates an example of distances used to compute the weighted value of the pixels. As shown in this figure, the first and second pixels of frames 2005 and 2210 are both located at location 2015. The first distance d1 is measured from the location 2015 to the left boundary of the first frame 2005. The second distance d2 is measured from the location 2015 to the right boundary of the second frame 2010. However, different embodiments may use different distances. For instance, some embodiments may use the distance to the upper or lower boundaries of the frames or combinations of distances to the upper, lower, left, and/or right boundaries.
[0097] Some embodiments also perform an illumination matching operation on the panoramic image in lieu of, conjunction or addition to blending images to remove boundary lines in a panoramic image, thus creating a seamless panoramic image. FIG. 17 illustrates a panoramic image after blending and illumination matching operations have been performed. As shown in this figure, the boundary lines have been removed, and the illumination of the pixels around the overlapping regions have been matched.
[0098] As mentioned above, some embodiments align a set of images to a locked image. The locked image can be any image in a set of images. FIG. 18 illustrates a panoramic image where the first image 1205 is locked, and the second and third images 1210-1215 are transformed and aligned to the first locked image 1205. FIG. 19 illustrates a panoramic image where the second image 1210 is locked, and the first and third images 1205 and 1215 are transformed and aligned to the second locked image 1210.
[0099] While the method of aligning a set of images to produce a panoramic image has been described with reference to numerous steps that are performed in a particular order, one of ordinary skill in the art will recognize that some of the above mentioned steps can be performed in a different order. For example, in some embodiments, the method 1100 determines which image in the set of images is the locked image before identifying (at 1110) a motion function for each pair of images. In such instances, the method only identifies a motion function for each pair of images that includes the locked image, and not all pairs of images in the set of images.
IV. High Dynamic Range Images
[0100] In addition to producing panoramic images from a set of images, some embodiments provide a method for producing high dynamic range images. A dynamic range is the range between the lightest highlight and darkest shadow in the image. In some embodiments, a dynamic range of an image is the contrast between the lightest and darkest region in the image. In some embodiments, a dynamic range is the range of luminance of an image.
[0101] FIG. 21 illustrates a method 2100 for producing such a high dynamic range image. The method 2100 will be described in reference to FIGS. 22-24 , which illustrates the graphical user interface of a video editing application that is capable of producing high dynamic range images.
[0102] As shown in FIG. 21 , the method 2100 aligns (at 2105) first and second images. In some embodiments, the method 2100 aligns (at 2105) the first and second images by transforming one of the images by applying a motion function, such as the one described above.
[0103] The method 2100 selects (at 2110) the image that is underexposed. FIG. 22 illustrates an image 2205 that is underexposed. As shown in this figure, some of the regions and objects (e.g., trees, rocks, stream) in the image 2205 are too dark, while other regions (e.g., sky, clouds) have the right amount of exposure. In contrast, FIG. 23 illustrates an image 2210 that is overexposed (i.e., too much illumination). As shown in this figure, the sky has little or no detail, whereas the trees, rocks and stream are shown with detail.
[0104] After selecting (at 2110) the underexposed image, the method 2100 computes (at 2115) a monochrome, low pass version of the underexposed image. In other words, the method 2100 computes (at 2115) a blurred single color version (e.g., black, white) of the underexposed image. The image is blurred because the low pass filter blocks out the high spatial frequency (e.g., detailed) components of the underexposed image. In some embodiments, the low pass filter is a Gaussian filter.
[0105] The method 2100 blends (at 2120) the first and second image by using the computed monochrome, low pass version of the underexposed image as a mask. FIG. 24 illustrates a high dynamic range image 2215 of the first and second images 2205 and 2210. As shown in this figure, certain regions in image 2210 that had no details, such as the clouds, are now shown with detail. In some embodiments, the exposure level of the high dynamic range image can be adjusted by changing a blend factor that weights the pixel values of the first and second images.
V. Computer System
[0106] FIG. 25 conceptually illustrates a computer system with which one embodiment of the invention is implemented. Computer system 2500 includes a bus 2505, a processor 2510, a system memory 2515, a read-only memory 2520, a permanent storage device 2525, input devices 2530, and output devices 2535. The bus 2505 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the computer system 2500. For instance, the bus 2505 communicatively connects the processor 2510 with the read-only memory 2520, the system memory 2515, and the permanent storage device 2525.
[0107] From these various memory units, the processor 2510 retrieves instructions to execute and data to process in order to execute the processes of the invention. The read-only-memory (ROM) 2520 stores static data and instructions that are needed by the processor 2510 and other modules of the computer system.
[0108] The permanent storage device 2525, on the other hand, is read-and-write memory device. This device is a non-volatile memory unit that stores instruction and data even when the computer system 2500 is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 2525.
[0109] Other embodiments use a removable storage device (such as a floppy disk or Zip® disk, and its corresponding disk drive) as the permanent storage device. Like the permanent storage device 2525, the system memory 2515 is a read-and-write memory device. However, unlike storage device 2525, the system memory is a volatile read-and-write memory, such as a random access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention's processes are stored in the system memory 2515, the permanent storage device 2525, and/or the read-only memory 2520.
[0110] The bus 2505 also connects to the input and output devices 2530 and 2535. The input devices enable the user to communicate information and select commands to the computer system. The input devices 2530 include alphanumeric keyboards and cursor-controllers. The output devices 2535 display images generated by the computer system. For instance, these devices display the GUI of a video editing application that incorporates the invention. The output devices include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD).
[0111] Finally, as shown in FIG. 25 , bus 2505 also couples computer 2500 to a network 2565 through a network adapter (not shown). In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet) or a network of networks (such as the Internet). Any or all of the components of computer system 2500 may be used in conjunction with the invention. However, one of ordinary skill in the art would appreciate that any other system configuration may also be used in conjunction with the present invention.
[0112] While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. For instance, some embodiments are implemented in one or more separate modules, while other embodiments are implemented as part of a video editing application (e.g., Shake® provided by Apple Computer, Inc.). Furthermore, the mask region is described during the alignment of video sequences. However, the mask regions can also be used during the creation off panoramic images and/or high dynamic range images. Thus, one of ordinary skill in the art will understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.
Claims
Certificate of Correction present
The claims shown below may not include correction changes. Use the corrected PDF for authoritative text.
What is claimed is:
1. A non-transitory computer readable medium comprising a video editing program for editing a set of video images comprising first and second sequences of video images, the video editing program for execution by a processor, the program comprising sets of instructions for:
comparing a first video image in the first sequence of video images with each of a plurality of video images in the second sequence of video images to identify a second video image in the second sequence of video images that matches the first video image;
using the first and second video images to associate each of a plurality of video images in the first video sequence to a different video image in the second video sequence;
defining a set of transform operations based on a difference between video images in one or more pairs of associated video images; and
aligning the first and second sequences of video images by applying the set of transform operations to one of the images in each pair of video images to align each pair of associated images to produce a panoramic sequence of video images.
2. The non-transitory computer readable medium of claim 1, wherein the set of instructions for defining the set of transform operations comprises sets of instructions for:
defining a set of motion functions, each motion function for expressing a motion between the video images in each pair of associated video images;
defining a set of objective functions based on the set of motion functions;
optimizing the set of objective functions to compute a set of optimal motion functions; and
defining the set of transform operations based on the set of optimal motion functions.
3. The non-transitory computer readable medium of claim 2, wherein each objective function in the set of objective functions expresses a difference between two frames in a pair of associated video images.
4. The non-transitory computer readable medium of claim 2, wherein the set of instructions for optimizing the set of objective functions comprises a set of instructions for minimizing the set of objective functions based on a set of constraints.
5. The non-transitory computer readable medium of claim 1, wherein each video image comprises a plurality of pixels, wherein the set of instructions for comparing the first video image with each of the plurality of video images in the second sequence comprises sets of instructions for:
selecting a set of pixels in the first video image and a set of pixels in each of the plurality of video images in the second sequence; and
comparing the selected set of pixels in the first video image with the selected set of pixels in each of the plurality of video images in the second sequence.
6. The non-transitory computer readable medium of claim 5, wherein the set of instructions for selecting the set of pixels comprises a set of instructions for selecting the set of pixels that include a spatial frequency above a particular value.
7. The non-transitory computer readable medium of claim 1,
wherein each video image in the first sequence of video images has a position in the first sequence, the position in the first sequence expressing a sequential order of the video image in the first sequence of video images,
wherein each video image in the second sequence of video images has a position in the second sequence, the position in the second sequence expressing a sequential order of the video image in the second sequence of video images,
wherein the position of the first video image in the first sequence is different than the position of the second video image in the second sequence.
8. The non-transitory computer readable medium of claim 7, wherein the position of the second image in the first second sequence is before the position of the first image in the first sequence.
9. The non-transitory computer readable medium of claim 7, wherein the position of the second image in the second sequence is after the position of the first image in the first sequence.
10. The non-transitory computer readable medium of claim 1, wherein the program further comprises a set of instructions for specifying a mask before applying the set of transform operations.
11. The non-transitory computer readable medium of claim 10, wherein each image comprises a set of pixels, wherein the mask excludes a plurality of pixels from being transformed when applying the set of transform operations to one of the images in each pair of associated video images.
12. The non-transitory computer readable medium of claim 1, wherein the set of instructions for comparing comprises a set of instructions for comparing each of a plurality of video images in the first sequence with each of the plurality of video images in the second sequence, wherein the first and second video images are images that match as good as any video image in the plurality of images in the first sequence and the plurality of images in the second sequence.
13. The non-transitory computer readable medium of claim 1, wherein the set of instructions for comparing comprises a set of instructions for comparing a portion of the first video image with a portion of each of the plurality of video images in the second sequence.
14. The non-transitory computer readable medium of claim 1, wherein the set of instructions for aligning comprises a set of instructions for applying a transform operation to each image in the first sequence of video images.
15. The non-transitory computer readable medium of claim 1, wherein the set of instructions for aligning comprises a set of instructions for applying a transform operation to each image in the second sequence of video images.
16. The non-transitory computer readable medium of claim 1, wherein the set of instructions for using the first and second video images comprises a set of instructions for sequentially selecting each image in the first and second sequences after the first and second images, each pair of video images comprising one sequentially selected image from each of the first and second sequences.
17. The non-transitory computer readable medium of claim 1, wherein the set of transform operations comprises only one transform operation that is based on a difference between the first video image and the second video image, wherein the set of instructions for aligning comprises a set of instructions for applying said one transform operation to one of the images in each pair of associated video images.
18. The non-transitory computer readable medium of claim 1, wherein the set of transform operations comprises a plurality of transform operations, wherein each transform operation is based on a difference between the video images in a corresponding pair of video images, wherein the set of instructions for aligning comprises a set of instructions for applying each transform operation to one of the images in the corresponding pair of associated video images.
19. The non-transitory computer readable medium of claim 1, wherein the set of instructions for comparing the first video image with each of the plurality of video images to identify the second video image comprises sets of instructions for:
defining a plurality of motion functions, each motion function for expressing a motion between the first video image and one of the images in the plurality of video images in the second sequence;
defining a plurality of objective functions based on the plurality of motion functions;
optimizing the plurality of objective functions to compute a plurality of optimal motion functions;
defining a transform operation between the first video image and each of the images in the plurality of video images in the second sequence based on the plurality of optimal motion functions;
aligning the first video image with each of the plurality of video images in the second sequence; and
identifying the second video image as a video image that aligns to the first video image better than any other image in the plurality of video images.
20. The non-transitory computer readable medium of claim 19, wherein each objective function in the plurality of objective functions expresses a difference between the first video image and a video image in the plurality of video images in the second sequence.
21. The non-transitory computer readable medium of claim 1, wherein the second image is an image that matches the first video image as good as any other image in the plurality of images.
22. A non-transitory computer readable medium comprising a computer program for creating a panoramic video sequence from first and second video sequences, the computer program comprising sets of instructions for:
comparing a first video frame in the first video sequence with each of a plurality of video frames in the second video sequence to identify a second video frame in the second video sequence that matches the first video frame;
using the first and second video frames to associate each of a plurality of video frames in the first video sequence to a different video frame in the second video sequence;
producing a plurality of composite video frames by aligning each pair of associated video frames in the first and second video sequences; and
defining the panoramic video sequence to include the plurality of composite video frames.
23. The non-transitory computer readable medium of claim 22, wherein the set of instructions for producing the plurality of composite video frames comprises sets of instructions for:
defining a set of transform operations based on a difference between video frames in one or more pairs of associated video frames; and
applying the set of transform operations to one of the frames in each pair of associated video frames to align each pair of associated video frames.
24. The non-transitory computer readable medium of claim 23, wherein the set of instructions for defining the set of transform operations comprises sets of instructions for:
defining a set of motion functions, each motion function for expressing a motion between the video frames in each pair of associated video frames;
defining a set of objective functions based on the set of motion functions;
optimizing the set of objective functions to compute a set of optimal motion functions; and
defining the set of transform operations based on the set of optimal motion functions.
25. The non-transitory computer readable medium of claim 24, wherein each objective function in the set of objective functions expresses a difference between two frames in a pair of video frames.
26. A method for creating a panoramic video sequence from first and second video sequences, the method comprising:
comparing a first video frame in the first video sequence with each of a plurality of video frames in the second video sequence to identify a second video frame in the second video sequence that matches the first video frame;
using the first and second video frames to associate each of a plurality of video frames in the first video sequence to a different video frame in the second video sequence;
by a computer, producing a plurality of composite video frames by aligning each pair of associated video frames in the first and second video sequences; and
defining the panoramic video sequence to include the plurality of composite video frames.
27. The method of claim 26, wherein each video frame comprises a plurality of pixels, wherein comparing the first video frame with each of the plurality of video frames in the second sequence comprises:
selecting a set of pixels in the first video frame and a set of pixels in each of the plurality of video frames in the second sequence; and
comparing the selected set of pixels in the first video frame with the selected set of pixels in each of the plurality of video frames in the second sequence.
28. The method of claim 27, wherein producing the plurality of composite video frames comprises:
defining a set of transform operations based on a difference between video frames in one or more pairs of associated video frames; and
applying the set of transform operations to one of the frames in each pair of associated video frames to align each pair of associated video frames.
29. The method of claim 27, wherein each video frame comprises a set of pixels, wherein producing the plurality of composite video frames further comprises specifying a mask before applying the set of transform operations, wherein the mask excludes a plurality of pixels from being transformed when applying the set of transform operations to one of the frames in each pair of video frames.
Patent Citations (13)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US5627905(A) | 1997-05-01 | Sebok et al. | Applicant |
| US6459822(B1) | 2002-10-01 | Hathaway et al. | Applicant |
| US6535650(B1) | 2003-03-01 | Poulo et al. | Applicant |
| US7023913(B1) | 2006-04-01 | Monroe | Examiner |
| US7280753(B2) | 2007-10-01 | Oya et al. | Applicant |
| US7602401(B2) | 2009-10-01 | Nishida et al. | Examiner |
| US2003/0090593(A1) | 2003-05-01 | Xiong | Applicant |
| US2004/0085340(A1) | 2004-05-01 | Dimitrova et al. | Applicant |
| US2004/0114799(A1) | 2004-06-01 | Xu | Applicant |
| US2005/0041883(A1) | 2005-02-01 | Maurer et al. | Applicant |
| US2006/0177150(A1) | 2006-08-01 | Uyttendaele et al. | Applicant |
| US2009/0202176(A1) | 2009-08-01 | Hwang et al. | Applicant |
| US2010/0157078(A1) | 2010-06-01 | Atanassov et al. | Applicant |
Non-Patent Literature (8)
- U.S. Appl. No. 11/107,327, filed Apr. 16, 2005, Souchard, Christophe, Commonly owned U.S. Appl. No. 11/107,327.Applicant
- Non-Final Office Action of U.S. Appl. No. 11/107,327, filed Mar. 2, 2009, Souchard, Christophe, Non-Final Office Action of commonly owned U.S. Appl. No. 11/107,327, listed as item #4 above.Applicant
- Kimia, B.B., Siddiqi, K., “Geometric Heat Equation and Nonlinear Diffusion of Shapes and Images”, Computer Vision and Pattern Recognition 1994, Proceedings CVPR '94., 1994 IEEE Computer Society Conference on Jun. 21-23, 1994, pp. 113-120.Applicant
- Portions of prosecution history of U.S. Appl. No. 11/107,327, Feb. 16, 2010, Souchard, Christophe, Portions of prosecution history of commonly owned U.S. Appl. No. 11/107,327 including actions dated Feb. 16, 2010 and Nov. 13, 2009; and responses/amendments dated Nov. 30, 2009, Nov. 14, 2009 and Aug. 3, 2009 (97 pages).Applicant
- Liang, Yu-Ming, et al., “Stabilizing Image Sequences Taken by the Camcorder Mounted on a Moving Vehicle”, Proceedings of IEEE 6th International Conference on Intelligent Transportation Systems, Oct. 2003, pp. 90-95, vol. 1, Shanghai, China.Applicant
- Updated portions of prosecution history of U.S. Appl. No. 11/107,327, Aug. 2, 2010, Souchard, Christophe, Updated portions of prosecution hisory to commonly owned U.S. Appl. No. 11/107,327, listed as item #4 in the 1449 filed Aug. 20, 2009, including action(s) dated Oct. 28, 2009; and response(s)/amendment(s) dated Oct. 20, 2009, Jul. 16, 2010 and Aug. 2, 2010 (30 pages).Applicant
- Debevec, Paul E., et al., “Recovering High Dynamic Range Radiance Maps from Photographs”, 1997, Month N/A, pp. 369-378, ACM Press/Addison-Wesley Publishing Co., New York, NY.Applicant
- Ward, Greg, “Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Handheld Exposures”, Journal of Graphics Tools 8.2, 2003, Month N/A, pp. 17-30, A K Peters, LTD, Natick, MA.Applicant