2-D extended image generation from 3-D data extracted from a video sequence

[0002] Not applicable

[0003] Not applicable

[0004] The present invention relates to the processing of video sequences, and more particularly to 2-D extended image generation from 3-D data extracted from a video sequence of a natural three-dimensional scene as observed through a monocular moving camera.

[0005] Presently video sequences provide two-dimensional images of natural three-dimensional scenes as observed through a monocular moving camera. Normally in order to produce three-dimensional objects or images a graphics generator is required for building such objects or images, which objects or images are then projected onto a two-dimensional plane for viewing as part of the video sequence. Three-dimensional manipulation of the three-dimensional objects or images is performed by the graphics generator, but the results are seen as a two-dimensional object or image in the video sequence.

[0006] What is desired is a method of generating 2-D extended images from 3-D data extracted from a two-dimensional video sequence.

[0007] Accordingly the present invention provides a method of generating 2-D extended images from 3-D data extracted from a video sequence representing a natural scene. In an image pre-processing stage image feature points are determined and subsequently tracked from frame to frame of the video sequence. In a structure-from-motion stage the image feature points are used to estimate three-dimensional object velocity and depth. Following these stages depth and motion information are post-processed to generate a dense three-dimensional depth map. World surfaces, corresponding to extended surfaces, are composed by integrating the three-dimensional depth map information.

[0008] The objects, advantages and other novel features of the present invention are apparent from the following detailed description when read in conjunction with the appended claims and attached drawing.

[0009] FIG. 1 is a flow diagram view for the generation of 2-D extended images for an object from a video sequence representing a natural 3-D scene according to the present invention.

[0010] FIG. 2 is a flow diagram view of a camera motion parameter estimator according to the present invention.

[0011] FIGS. 3A, 3B and 3C respectively are illustrations of an image in a scene shot from the video image sequence, a mask for the image defining the object, and a 2-D extended image for the defined object according to the present invention.

[0012] FIG. 4 is a flow diagram view of a hierarchical matching scheme according to the present invention.

[0013] FIG. 5 is a general flow diagram view of a structure-from-motion (SFM) algorithm according to the present invention.

[0014] FIG. 6 is a flow diagram view of a pre-processing stage for the SFM algorithm of FIG. 5 according to the present invention.

[0015] FIGS. 7A, 7B and 7C respectively are illustrations of an image in a scene from a video sequence, a depth map for the image, and a segmentation mask for a foreground object in the image according to the present invention.

[0016] FIG. 8 is a flow diagram view of a 2-D extended image generator according to the present invention.

[0017] FIG. 9 is an illustration of the pasting of points of an image in a scene from a video sequence into a 2-D extended image according to the present invention.

[0018] FIG. 10 is a flow diagram view of a filter for producing diadic pyramids according to the present invention.

[0019] FIG. 11 is an illustrative view of the mapping of points from an image from the video sequence into the 2-D extended image according to the present invention.

[0020] The present invention describes the generation of a 2-D extended image, referred to as a world image, sprite or wall paper, from a video of a natural 3-D scene as observed through a monocular moving camera. This 2-D extended image represents in some way, usually with some loss, the information contained in the video representing a natural 3-D scene. Also the original video, with some loss in quality, may be synthesized from this 2-D extended image. The method described below is divided into two parts: 3-D camera parameter estimation and 2-D extended image generation.

[0021] Referring now to FIG. 1 a temporal sequence of raw video images representing a natural 3-D scene projected onto the moving camera is input to the system. A scene cut detector 12 identifies the temporal boundaries within the scene that make up an input video sequence, and the sequence of images that lies within such temporal boundaries constitute one scene shot. The images for the detected scene shot are input to a pre-segmentor 14 where objects are identified based upon various characteristics, such as color, shape and velocity, by using manual, semi-automatic or automatic processes. The pre-segmentation process also selects and tracks features on the objects from image to image within the scene shot. For example in a video clip of a tennis match, a tennis player may be identified as an object and the player's shoes or tennis racquet may be selected as features. The information from the pre-segmentor 14 is input to a 3-D camera parameter estimator 16 to estimate camera parameters, namely (i) the 3-D velocity of the objects in the scene and of the camera relative to some origin in 3-D space, and (ii) the depth of the objects from the camera. This depth and motion information is processed to generate a dense 3-D depth map for each object, as is described below. A decision block 18 determines the “flatness” of the object, and also provides feedback to the pre-segmentor 14 to assist in separating objects on the basis of depth layers. The depth map information and the scene shot images are input to a 2-D extended image generator 20 to produce a 2-D extended image for each object in the scene shot.

[0022] The 3-D camera parameter estimator 16 uses a structure-from-motion (SFM) algorithm to estimate camera parameters from the pre-segmented video, as described in greater detail below. Currently SFM is typically used to estimate the 3-D velocities and depths of points of rigid opaque objects. In this application SFM is used for two purposes: first to estimate the 3-D parameters of independently moving objects, called foreground objects, such as a car moving along a street in a video clip; and second to estimate the 3-D parameters of the scene shot's background, called background object, such as the audience in the tennis match video clip. The background object represents that part of the scene shot that is static in the world, i.e., has a zero velocity with respect to some point in 3-D space in a world coordinate system, while the camera and foreground objects may be in motion with respect to the same point. However in the video clip the background object may appear to be in motion because of the camera movement. The video clip therefore records the velocity of the background object with respect to the set of coordinate axes located at the focal plane of the camera. This apparent 3-D velocity of the background object is opposite in sign but identical in magnitude to the velocity of the camera with respect to the fixed point in 3-D space. Thus by estimating the apparent background velocity from the input video, the 3-D trajectory and orientation of the camera are determined with respect to the world coordinate system.

[0023] SFM currently requires that foreground objects be pre-segmented from the background object, and that feature points on the different objects be chosen manually. In this application feature points on the objects are automatically selected and tracked while these objects are still assumed to be pre-segmented. SFM, when applied to the background object, provides the 3-D camera velocity parameters and the depth from the camera of feature points in the background object. Using the camera velocity parameters, the depth is estimated at the feature points and the resulting values are interpolated/extrapolated to generate a depth value for each image pixel in the scene shot to provide a dense depth map.

[0024] The 2-D extended image generator 20 combines the results of the depth map compositing, using a planar perspective projection (PPP) model to generate the image. The PPP model has nine parameters, which enables the estimation or prediction of the 2-D extended image given any single image, i.e., current frame or snapshot of the scene shot. The PPP model parameters are also used to estimate or predict a current frame from the extended image. The depth map, which is generated for pairs of successive images, is combined with the PPP model parameters associated with the first image in the pair to generate approximate PPP model parameters associated with the second image. These approximate PPP model parameters are refined using a dyadic pyramid of the current image and the second image in the pair. The current image represents the composition of the images in the scene shot up to and potentially including the first image in the pair. The final PPP parameters along with the second image are used to update the current extended image to include the second image.

[0025] As indicated above, structure-from-motion is a method by which a 3-D rigid object velocity (motion) and depth (structure) may be estimated using different 3-D views of the same scene. The present SFM implementation provides a robust estimation of the 3-D camera motion parameters and the depth at selected points in the 3-D scene shot from a mono-scopic video sequence, i.e., video recorded by a monocular camera. Independently moving objects, i.e., foreground objects, in the scene shot are pre-segmented as described above.

[0026] The 3-D camera velocity is estimated from a set of image feature points belonging to static parts, i.e., background objects, that are extracted from and tracked through the scene shot being analyzed. For a static 3-D scene as recorded by a moving camera, the effective motion of the feature points projected onto the camera's focal plane is related to the camera's 3-D motion. In fact a camera moving along a 3-D trajectory induces an “equal and opposite” motion on its focal plane for any feature point. In the absence of camera motion the velocity of the feature points is zero. Non-zero feature point velocity is used in SFM to estimate the camera's 3-D motion parameters, also known as “egomotion”, and trajectory. The camera velocity is described by the parameters of rotation and translation in 3-D space. Once these parameters are known, the scene depth at the feature points may be estimated, since scene depth is merely the distance between the feature point and the camera.

[0027] The 3-D camera parameter estimator 16 has the following submodules, as shown in FIG. 2: (i) feature point extractor 24, (ii) feature point tracker 26, (iii) structure-from-motion 28, and (iv) depth map generator 30. The inputs to the 3-D camera parameter estimator 16 are raw video images, denoted by I_k, and the corresponding “alpha” images, denoted by A_k. The alpha image is a binary mask that determines the “valid” regions inside each image, i.e., the regions of interest or objects, as shown in FIG. 3 where FIG. 3A represents an image I_kfrom a tennis match and FIG. 3B represents the alpha image A_kfor the background object with the tennis player blanked out. The alpha images are obtained from the pre-segmentor 14, either through a rough pre-segmentation or through user interaction.

[0028] Feature points are the salient features of an image, such as corners, edges, lines or the like, where there is a high amount of image intensity contrast. The feature points are selected because they are easily identifiable and may be tracked robustly. The feature point extractor 24 may use a Kitchen-Rosenfeld comer detection operator C, as is well known in the art. This operator is used to evaluate the degree of “cornerness” of the image at a given pixel location. “Corners” are generally image features characterized by the intersection of two directions of image intensity gradient maxima, for example at a 90° angle. To extract feature points the Kitchen-Rosenfeld operator is applied at each valid pixel position of I_k. The higher the value of the operator C at a particular pixel, the higher its degree of “cornerness”, and the pixel position (x,y) in I_kis a feature point if C at (x,y) is greater than at other pixel positions in a neighborhood around (x,y). The neighborhood may be a 5×5 matrix centered on the pixel position (x,y). To assure robustness the selected feature points have a degree of cornerness greater than a threshold, such as T_c=10. The output from the feature point extractor 24 is a set of feature points { F_k} in image I_kwhere each F_kcorresponds to a “feature” pixel position in I_k.

[0029] Given a set of feature points F_kin image I_kthe feature point tracker 26 tracks the feature points into the next image I_k+1of the scene shot by finding their closest match. “Closest match” is defined by the pixel position in I_k+1that maximizes a cross-correlation measure CC defined as:

[0030] CC={W²(Σ_m,nI_k(x+m,y+n)I_k+1(x′+m,y′+n))−(Σ_m,nI_k(x+m,y+n))(Σ_m′,n′I_k+1(x′+m′,y′+n′))}/{SQRT(W²(Σ_m,nI²_k(x+m,y+n))

[0031] −(Σ_m′,n′I_k(x+m′,y+n′))²}*1/{SQRT(W²(Σ_m,nI²_k+1(x′+m,y′+n))−(Σ_m′,n′I_k+1(x′+m′,y′+n′))²}

[0032] evaluated by overlapping a D_v×D_hintensity neighborhood around the feature point at (x,y) in image I_kover an equal sized neighborhood around a candidate target pixel (x′,y′) in I_k+1for W=(2D_v+1)(2D_h+1). CC has a value of one if the two neighborhoods being compared are identical, and minus one if they have exactly opposite intensity profiles. The feature point tracker 26 gives a “closest match” point in I_k+1corresponding to each feature point in I_k. Each pair formed by a feature point in I_kand its closest match in I_k+1is referred to as a “feature point correspondence” pair.

[0033] In order to improve robustness and increase processing speed in the feature point tracker 26, four additional processes may be used:

[0034] (i) Outlier removal due to low cross-correlation—every feature point correspondence pair must have an associated cross-correlation that is greater than a threshold T_cc. Feature points whose closest matches do not satisfy this threshold are eliminated. Further feature point correspondence pairs are sorted in descending order according to their cross-correlation values, the higher cross-correlations being the most reliable ones, and vice versa. A T_ppercentage of feature point correspondence pairs with the lowest cross-correlations are eliminated from the correspondence list. Representative values for T_ccand T_pare 0.9 and 40 respectively.

[0035] (ii) Outlier removal due to lack of strength—a feature correspondence pair (x,y)<-->(x′,y′) in I_kand I_k+1respectively is strong if a bidirectional matching criteria is satisfied by the feature points involved, i.e., the closest match of a point (x,y) in I_kmust be the point (x′,y′) in I_k+1, and conversely the closest match of the point (x′,y′) in I_k+1must be the point (x,y) in I_k. Otherwise the correspondence is weak and is eliminated from the correspondence list.

[0036] (iii) Hierarchical matching—in order to increase speed of establishing feature point correspondences, a hierarchical approach is used in which the closest matches are searched for using a Gaussian pyramid of the images involved, as explained further below.

[0037] (iv) Subpixel resolution match refinement—in order to increase accuracy of matching after the closest matches have been determined at pixel resolution, a local refinement of the matches is done at subpixel resolution, such as by adapting the method described in U.S. Pat. No. 5,623,313.

[0038] The process of finding closest matches using hierarchical matching is shown in FIG. 4. Depending upon the expected maximum image velocity, the number of pyramid levels required L_maxis estimated:

[0039] L_max={L|Σ_{i=1−>(L−1)}Sⁱ<V_max<Σ_i=1−>LSⁱ}

[0040] where V_maxis the maximum expected image velocity, which may be provided by the user in lieu of a default, and S is a specified search range per pyramid level in pixels per frame, such as +/−2 range about the target pixel. The inputs to the hierarchical matching are the feature point set F_kin image I_kand the L_max-level Gaussian pyramid of images I_kand I_k+1. For the initial conditions F_k+1=F_kand L=L_max. As shown in FIG. 4 the steps include:

[0041] 1. Decimate point in F_kto level L, i.e., input feature point Fⁱ_k=(x_i,y_i) becomes round(x_i/2^L,y_i/2^L)—if L=L_max, decimate points in F_k+1to level L.

[0042] 2. Due to this decimation/rounding of point positions, F_kpoints that are close together at the bottom of the pyramid may merge into the same point higher up in the pyramid. To save time and without loss of accuracy these “redundant” points are temporarily removed in the higher levels of the pyramid. An inheritance list, describing the mapping of all points F_kⁱinto unique (non-redundant) points, is stored.

[0043] 3. One-way match—closest matches are found for the unique decimated feature points in F_k. The corresponding points Fⁱ_k+1=(x′_i,y′_i) are used as “initial guesses” for the position of the closest matches, i.e., the cross-correlation operator CC is applied between each point F_kⁱand each point (x′,y′) in a forward search neighborhood around the point Fⁱ_k+1. The position (x′,y′) that maximizes CC is stored as Fⁱ_k+1. In this step the images used are those of the L^th-level of the Gaussian pyramid.

[0044] 4. Using the inheritance list redundant points are restored, assigning to them their corresponding closest matches.

[0045] 5. Bring the points in F_k+1one level down the pyramid so that feature point Fⁱ_k+1=(x′_i,y′_i) becomes (2x′_i,2y′_i).

[0046] 6. Set L=L−1.

[0047] 7. Repeat the above steps until and including when L=0—when L=0 the data set F_k+1represents the closest matches of F_kin I_k. Step 5 is not executed, and the outlier removal due to low cross-correlation is enforced.

[0048] After the above “forward matching” is completed, the weak matches are removed by “backward matching”, i.e., for each point Fⁱ_k+1the cross-correlation operator is applied in all points (x,y) in a backward search neighborhood around F_kⁱ. If the point that maximizes CC is different from F_kⁱ, then this point is a weak match and is rejected. A less strict constraint is to accept such (x,y) if it falls within a radius r_strongaround F_kⁱ. If the backward search neighborhood is large, then the backward matching is done hierarchically to save time. The resulting output from the feature point tracker 26 is a set of strong, robust feature point correspondence pairs contained in point sets F_kand F_k+1.

[0049] The SFM submodule 28, as shown in FIG. 5, provides estimates for a camera rotation matrix R and a camera translation vector T, which are done following an “essential matrix” approach to SFM. Since this approach is sensitive to input noise, two different approaches may be used that result in accurate 3-D camera parameters. Both of these approaches require a “preprocessing stage” in which different sets of camera parameters are estimated. Then the best possible answer is chosen from the approach followed.

[0050] The preprocessing stage 29 has as inputs the feature point correspondence pairs contained in F_kand F_k+1, the images I_{k and Ik+1}, and measured feature point velocity vectors, Vin_kⁱ=(vⁱ_k,x,vⁱ_k,y)=Fⁱ_k+1−Fⁱ_k=(x′_i,y′_i)−(x_i,y_i). The initial conditions are: divide I_kinto approximately idential image blocks B^j_k, i.e., rectangular image regions, for j={1, . . . ,N_points} so that each block “contains” approximately the same number of feature points Fⁱ_kassociated with it, and each point Fⁱ_kbelongs to one and only one block. Each block exclusively represents one region of I_k. This enforces that the feature points used in the estimation of the camera parameters span the whole extension of the input image, resulting in a more general, robust set of parameters. The preprocessing stage 29 is shown in greater detail in FIG. 6, as described below.

[0051] Step 1: From each block randomly draw one feature point correspondence pair according to a uniformly-distributed random number generator so that all feature point correspondence pairs within each block are likely to be chosen—the result is N_pointsfeature correspondence pairs that span over the whole of I_k.

[0052] Step 2: Calibrate the feature point coordinates using a coordinate calibration transformation so that (i) the spatial center of mass of the feature points is approximately at the (0,0) coordinate origin and (ii) the feature points fall approximately with a radius of SQRT(2) of the center of mass—this results in a more stable estimation of an essential (E) matrix. The fastest, easiest transformation is: $\left|\begin{array}{c}{x}_i^n\\ y_i^n\end{array}\right|=\left|\begin{array}{ccc}2/\left(N_{\mathrm{cols}}-1\right)& 0& -1\\ 0& 2/\left(N_{\mathrm{rows}}-1\right)& -1\end{array}\right|\left|\begin{array}{c}{x}_i\\ y_i\\ 1\end{array}\right|$

[0053] where (x_iⁿ,y_iⁿ) becomes the “normalized” (x_i,y_i) image position, and N_rowsand N_colsare the number of rows and columns of I_krespectively.

[0054] Step 3: Compute the essential matrix E using the above normalized N_pointspoint correspondence pairs.

[0055] Step 4: Extract the camera rotation matrix R and the camera translation vector T from the computed essential matrix E.

[0056] Step 5: Given R and T estimate the depth Z_iat every feature point Fⁱ_k.

[0057] The above steps should be repeated enough times to exhaust all possible combinations of feature points. However such an approach may be intractable from a computational point of view—the total number of operations may be too large to be feasible for realtime implementations. Instead a robust suboptimum statistical approach determines the choice of the “best” set of camera motion parameters from a subset of the possible combinations. Therefore the above steps are repeated N_combtimes, resulting in approximately N_combdifferent estimated sets of 3-D camera motion parameters.

[0058] The robustness and speed of the preprocessing may be enhanced by the following two steps:

[0059] (i) After Step 3 and before Step 4 above, compute depth values Z for each feature point in the set of N_points. If at least one out of the N_pointsdepth values has a different sign, i.e., all N_pointsdepths but at least one have a positive sign, there exists at least one outlier in the set of randomly chosen N_pointspoints and, therefore, it is very likely that the resulting E matrix will be erroneous, hence making the remaining Steps fruitless. In such a case, especially when the level of outliers in the input data sets is high, throwing away this combination of points both reduces overall processing time, for time is spent only on meaningful combinations, and increases the confidence in the final results.

[0060] (ii) If Vin_kⁱis zero for all N_pointsrandomly chosen points, then according to this combination of points the camera is static, i.e., to save time instead of computing the E matrix and extracting camera parameters the result of this combination is automatically set to a motionless camera—R=diag(1,1,1) and T=[0,0,0]^T.

[0061] After the preprocessing stage is completed, the N_combsets of R, T and Z_irepresent candidate solutions for camera parameters. Then the best set is determined by one of the following two methods 31A or 31B.

[0062] Method 1

[0063] Given N_combsets of 3-D parameters, compute the resulting output image velocity vectors Vout_kⁱat every feature point Fⁱ_k. Then compute the “overall velocity estimation error”:

[0064] E^k_vel=Σ_i||Vout_kⁱ−Vin_kⁱ||²

[0065] The combination that minimizes the overall velocity estimation error is chosen as the best set of camera parameters. This process provides accurate results even in the presence of outliers in the feature point correspondence pairs that may have survived the outlier removal procedures. An “early exit” condition may be used to detect when a satisfactory estimate of the motion parameters is found and stop searching. The condition is that for every combination the percentage of points F_kⁱis computed that satisfy the inequality ||Vout_kⁱ−Vin_kⁱ||≦T_vwhere T_vis a specified maximum accepted velocity error in pixels per frame. If such percentage is higher than a minimum required performance percentage T_perf, then no further search is required as this combination satisfies the minimum requirement. The camera motion parameters produced by this combination of feature points is accurate enough as long as T_perfis strict enough, such as T_v=0.25 and T_perf=90.

[0066] Method 2

[0067] Given N_combsets of 3-D parameters the best camera parameters are found in a statistical sense. The best translation vector is obtained as $T=1/N_{\mathrm{comb}}\left|\begin{array}{c}{\Sigma }_{m=1->\mathrm{Ncomb}}{t}_x^m\\ {\Sigma }_{m=1->\mathrm{Ncomb}}{t}_y^m\\ {\Sigma }_{m=1->\mathrm{Ncomb}}{t}_z^m\end{array}\right|$

[0068] where t_x^m, t_y^mand t_z^mare the components of the T vector resulting from the m^thrandom combination of feature points. The resulting T vector is then normalized so that ||T||=1 (if t_x=t_y=t_z=0 then T is left as a zero vector). This normalization of T does not affect the results because T is only estimated up to a scaling factor.

[0069] To estimate the best rotation matrix R the statistics of the angles of rotation, namely [αβγθ]. α, β and γ describe the axis of rotation and are subject to the constraint that

[0070] cos²(α)+cos²(β)+cos²(γ)=1

[0071] and θ specifies the angle of rotation about such axis.

[0072] To compute [αβγ]:

[0073] (i) Choose α₁, the angle α, β or γ with the smallest variance—the best value of such angle is its own statistical mean. Then

[0074] α₁=(1/N_comb)Σ_m=1−>Ncombα₁^m.

[0075] (ii) Choose α₂, the angle α, β or γ with the second smallest variance, the best value of such angle being its own mean constrained

[0076] α₂=(1/card(Y))Σ_mεYα₂^m

[0077] where Y={m|mε{1, . . . ,N_comb} and cos²(α₁)+cos²(α₂^m)≦1}, card(Y) being the number of elements in Y.

[0078] (iii) Choose α₃, the angle with the greatest variance, the best value being α₃=arc cos SQRT(1−cos²α₁−cos²α₂).

[0079] Finally the angle of rotation is determined by its own mean, namely θ=(1/N_comb)Σ_m=1−>Ncombθ^m.

[0080] After all four best angles are computed the rotation matrix R is formed which corresponds to such angles. $R=\left|\begin{array}{ccc}{n}_1^2+\left(1-n_1^2\right)\cos \theta & n_1{n}_2\left(1-\cos \theta \right)-n_3\sin \theta & n_1{n}_3\left(1-\cos \theta \right)+n_2\sin \theta \\ n_1{n}_2\left(1-\cos \theta \right)+n_3\sin \theta & n_2^2+\left(1-n_2^2\right)\cos \theta & n_2{n}_3\left(1-\cos \theta \right)-n_1\sin \theta \\ n_1{n}_3\left(1-\cos \theta \right)-n_2\sin \theta & n_2{n}_3\left(1-\cos \theta \right)+n_1\sin \theta & n_3^2+\left(1-n_3^2\right)\cos \theta \end{array}\right|$

[0081] where n₁=cos α, n₂=cos β and n₃=cos γ.

[0082] Either of the above methods produce a best estimate for the 3-D camera motion parameters. It is possible that a small subset of feature points have corresponding negative depths. Since the above estimated camera parameters give an accurate result for the large majority of feature points, the small subset of points with negative depths is tagged as outliers and discarded. The output of the SFM submodule 28 is the best set of 3-D camera parameters and the depth at the location of the image feature points in F_k.

[0083] Continuous depth for all points in an image are obtained by interpolation from the depths at the feature points. From the sets of feature points present in the image pair and an estimate of the depth at each feature point, and assuming that the feature points are chosen so that they lie relatively close to each other and span the whole image, the depth map generator 30 creates a 2-D mesh structure by interconnecting such feature points in which the feature points lie at the vertices of formed polygons. The closer the feature points are to each other, the denser the resulting 2-D mesh structure. Since the depth at each vertex of the 2-D structure is known, the depths at the points within each polygon may be estimated. In this way the depth at all image pixel positions may be estimated. This may be done by planar interpolation, as described below.

[0084] A robust and fast method of generating the 2-D mesh structure is Delaunay triangulation. The feature points are connected to form a set of triangles whose vertices lie at feature point positions. Using the depth associated with each feature point and its corresponding vertex, a “depth plane” may be fitted to each individual triangle from which the depths of every point within the triangle may be determined.

[0085] As a rule feature points at or very close to the boundaries of the image are not chosen to avoid having problems with artifacts in image boundaries and to avoid wasting time tracking points which may go out of the frame in subsequent images of the sequence. Therefore depths at or very close to the image boundaries cannot be obtained using the above procedure. Depth at or near image boundaries may be obtained by extrapolating from the known depths of the closest points near the target boundary point. A measure of the weighted average of the depths at neighboring points is used. In this measure the weights are inversely proportional to the distance from the target boundary point and the neighboring points with known depths. In this way the points closest to the target image boundary point have greater effect than more distant points. From the depth map generator 30 an image Z_kis obtained that contains the estimated depths for every pixel position in image I_k. A typical result is shown in FIG. 7 where FIG. 7arepresents the image I_kand FIG. 7brespresents the image Z_k.

[0086] The density of the map is the user's choice. If the desired output is a dense depth map, then the user may so specify and a high number of feature points is selected. Otherwise default values are applied and the process runs faster with fewer feature points. This is the end of the SFM algorithm 16. From the input images I_kand I_k+1the final products are the sets of feature points F_kand F_k+1, the rotation matrix R, the translation vector T and the depth map Z_k.

[0087] The decision block 18 has the role of determining, given the segmentation, 3-D camera parameters and scene depth information, for which objects the system generates 2-D extended images. The decision to generate 2-D extended images is based on the fact that the depth of each foreground or background object exhibits a small spread relative to its average value. If Z^α is the average depth value for the αth object, and if ΔZ^α is the largest difference in depth between pairs of object points, then the condition for a small depth spread is |ΔZ^α|/Z^α <<1. This means that the αth object is considered as “flat” relative to a given collection of camera view points for which the depth values were computed. In general the flatness condition occurs when the objects are far from the camera, such that Z^α is large, or when the objects are flat themselves. In addition to the flatness condition for each individual object, a further decision is made on how far each object is relative to the other objects and how these objects are distributed in 3-D space according to their depth. The relative object depth is determined by the ratios of their average depth values, and their 3-D distribution is based on the relative ordering of the average depth values, i.e., in ascending order.

[0088] If the objects are considered to be “flat” and they are ordered in a sequence of superimposed layers, then the decision block 18 determines that 2-D extended images should be generated for each layer.

[0089] The decision block 18 also generates feedback information to the pre-segmentor 14, as shown in FIG. 1. This occurs when depth information about a given object is used to further segment it, such as separating a foreground tree from an image as shown in FIG. 7c.

[0090] The input video may be thought to represent one or more physically meaningful objects. A 2-D extended image may be generated for each object, and more commonly for the background object. The location and shape of the object of interest is available as an alpha image map, as shown in FIG. 3b.The alpha map is an image having the same dimensions as the input images I_k, and may be generated through depth segmentation performed as part of the decision process or by other means as described above. For certain kinds of input video materials, where (a) object segmentation is performed by other means and (b) motion between consecutive frames is not significant, the outputs from the SFM algorithm, R, T and Z, are optional. For this simple subset of video a refine prediction box 34, as shown in FIG. 8, uses the refined parameters generated using a previous picture as an initial approximation (coarse or rough estimates). The amount of motion that the 2-D extended image generator 20 can handle without inputs from the SFM algorithm depends on (a) the number of levels of the pyramid used in refining parameters and (b) the presence or absence of local minima in the cost function used in refining parameters. I_k(x,y) represents the intensity (luminance) at location (x,y), and U_k(x,y) and V_k(x,y) represent the chrominance values. A value of A_k(x,y)≦T_α means the pixel at position (x,y) belongs to the object of interest, otherwise it doesn't. The rotation, translation and depth maps are estimated by the SFM algorithm, as described above, and the median of the estimated depth map at valid pixel locations for the object is determined.

[0091] An integrator 32 is used to generate the 2-D extended image W_kat time instances 1 through k by combining SFM information with PPP information. The intensity component of the 2-D extended image is WI_k, the chrominance components are WU_kand WV_k, and the alpha component is WA_k. One goal of 2-D extended image generation is to be able to predict input images at time instances 1 through k from W_k. A nine parameter plane perspective projection (PPP) is used to perform the prediction:

[0092] I_k(x,y)=ρ_k*WI_k(f_k(x,y),g_k(x,y))

[0093] U_k(x,y)=ρ_k*WU_k(f_k(x,y),g_k(x,y))

[0094] V_k(x,y)=ρ_k*WV_k(f_k(x,y),g_k(x,y))

[0095] A_k(x,y)=ρ_k*WA_k(f_k(x,y),g_k(x,y))

[0096] where f_kand g_kare defined through:

[0097] f_k(x,y)={P_k(0,0)*x+P_k(0,1)*y+P_k(0,2)}/{P_k(2,0)*x+P_k(2,1)*y+P_k(2,2)}

[0098] g_k(x,y)={P_k(1,0)*x+P_k(1,1)*y+P_k(1,2)}/{P_k(2,0)*x+P_k(2,1)*y+P_k(2,2)}

[0099] P_k(2,2) is forced to be 1.

[0100] The goal of 2-D extended image generation is to compute the planar image W_kand associated P_kand ρ_kfor each k. At termination where k=K the final W_Kand the history of P_Kand ρ_kmay be used to synthesize images I_k, U_k, V_k, 1≦k≦N. For each k the nine parameters, i.e., eight unknown components of the 3×3 matrix P_kand ρ_k, are estimated such that the synthesized images are:

[0101] I′_k≈I_k

[0102] U′_k≈U_k

[0103] V′_k≈V_k

[0104] A′_k≈A_k

[0105] The cost function used to achieve this approximation is:

[0106] MSE(P_k,ρ_k)Δ(1/N_v){Σ_(x,y)||I_k(x,y)−I′_k(x,y)||²}

[0107] where (x,y) in the above summation is such that A′_k(x,y)≧T_α and A_k(x,y)≧T_α and N_vis the number of valid pixel locations (x,y).

[0108] The first 2-D extended image generated is equal to the first image in the scene shot.

[0109] k=1

[0110] WI₁=I₁

[0111] WU₁=U₁

[0112] WV₁=V₁

[0113] WA₁=A₁

[0114] and the estimated parameters are simply: $P_1=\left|\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right|$ ${\rho }_1=1$

[0115] such that

[0116] f₁(x,y)=x

[0117] g₁(x,y)=y

[0118] Initialize an image “HITS” of dimensions equal to those of I_k.

[0119] HITS(x,y)=1 if A₁(x,y)≧T_α

[0120] For every new picture I_k, k>1, from the input video sequence SFM is optionally performed between the previous image I_k−1and the present image I_kto obtain T_k−1, R_k−1and Z_k−1for the previous image. From the previous image depth map a representative depth for interested objects in the previous picture is obtained as

[0121] Z′=median{all Z_k−1(x,y), where A_k−1(x,y)≧T_α}

[0122] Then a 3×3 matrix Q_k−1is computed:

[0123] Q_k−1=R⁻¹_k−1−(1/Z′)R⁻¹_k−1T[001].

[0124] Q_k−1is normalized so that the element in the last-row, last-column of Q_k−1, i.e., Q_k−1(2,2), equals 1.0:

[0125] Q_k−1←Q_k−1/Q_k−1(2,2)

[0126] Given a point (x_n,y_n) in normalized coordinates of image I_k, Q_k−1is used to find an approximate location (x′_n,y′_n) of the corresponding point in image I_k−1as:

[0127] x′_n={Q_k−1(0,0)*x_n+Q_k−1(0,1)*y_n+Q_k−1(0,2)}/{Q_k−1(2,0)*x_n+Q_k−1(2,1)*y_n+Q_k−1(2,2)}

[0128] y′_n={Q_k−1(1,0)*x_n+Q_k−1(1,1)*y_n+Q_k−1(1,2)}/{Q_k−1(2,0)*x_n+Q_k−1(2,1)*y_n+Q_k−1(2,2)}

[0129] The normalized coordinates (x′_n,y′_n) may be converted to pixel coordinates (x′,y′) through

[0130] x′=(x′_n+1)(N_cols−1)/2

[0131] y′=(y′_n+1)(N_rows−1)/2

[0132] where N_colsand N_rowsrefer to the input image dimensions. The corresponding point (x″,y″) in W_k−1is:

[0133] x″=f_k−1(x′,y′)

[0134] y″=g_k−1(x′,y′)

[0135] Thus for a given point in I_ka corresponding point may be found in W_k−1. This correspondence is used to obtain approximate P_kas follows:

[0136] 1. For the four corner points in I_kcorresponding points are found in W_k−1, i.e., for (x,y)=(0,0), (width-1,0), (0,height-1) and (width-1,height-1) the (x″,y″)s are found using the above equations.

[0137] 2. Using the four sets of (x,y) and corresponding (x″,y″) the linear equations for the eight unknown element of matrix P_kdefined by:

[0138] x″=f_k(x,y)={P_k(0,0)*x+P_k(0,1)*y+P_k(0,2)}/{P_k(2,0)*x+P_k(2,1)*y+P_k(2,2)}

[0139] y″=g_k(x,y)={P_k(1,0)*x+P_k(1,1)*y+P_k(1,2)}/{P_k(2,0)*x+P_k(2,1)*y+P_k(2,2)}

[0140] are solved with the understanding that P_k(2,2)=1.

[0141] The matrix P_kobtained serves as an initial approximation, which is refined using the following procedure. If SFM is not performed, Q_k−1is not available and P_k−1serves as the approximate P_kto be refined.

[0142] The approximate P_kfrom above is refined using the intensity and alpha components of the input image and the presently existing 2-D planar image. The refinement attempts to minimize a mean-squared-error over all possible P_kand ρ_k. The refinement process is shown in FIG. 9. The full resolution input image is considered to be the level 0 representation in a diadic pyramid. Separable filters are used to generate the diadic pyramid. Other types of filtering schemes could equally well be used for this purpose. Given an image at level L, the image at level L−1 is obtained using the system shown in FIG. 10. Each lowpass filter in this figure computes outputs only at half the input sample positions. For example the lowpass filter may have a length of 5 and filter coefficients of (13/256, 64/256, 102/256, 64/256, 13/256). This structure is repeatedly used to generate an L_max+1 level pyramid consisting of images at levels 0,1, . . . ,L_max. Note that this L_maxmay be different from that used in the hierarchical matching described above. For example in one implementation L_max=2. For the hierarchical refinement of P_kand ρ_kpyramids of I_k, A_k, WI_k−1and WA_k−1are generated.

[0143] Given P_kfor images at level L=L₁, P_kmay be scaled so that it may be applied on images at level L=L₂through:

[0144] P_k(0,2)=2^L1−L2*P_k(0,2)

[0145] P_k(1,2)=2^L1−L2*P_k(1,2)

[0146] P_k(2,0)=2^L2−L1*P_k(2,0)

[0147] P_k(2,1)=2^L2−L1*P_k(2,1)

[0148] Other elements of P_kand ρ_kremain unchanged in this scaling process.

[0149] To refine P_kat a given level of the pyramid:

[0150] 1. Using the current values of P_kand ρ_k, compute I′_kand A′_kas above, and then the mean squared error, MSE(P_k,ρ_k).

[0151] 2. Estimate a value ρ′_k:

[0152] ρ′_k=Σ_(x,y)I_k(x,y)/Σ_(x,y)I′_k(x,y)

[0153] where (x,y) a valid pixel location.

[0154] 3. Using the current P_kand ρ′_kcompute I′_kand A′_kand the resulting mean squared error, MSE(P_k,ρ′_k)

[0155] 4. Decide between ρ_kand ρ′_kas follows. If MSE(P_k,ρ′_k)<MSE(P_k,ρ_k), then ρ_k=ρ′_kand bestMSE=MSE(P_k,ρ′_k); otherwise retain the value of ρ_kand bestMSE=MSE(P_k,ρ_k).

[0156] 5. Set a counter ctr=0, bestP_k=P_k, TmpP_k=P_k.

[0157] 6. Initialize all elements of an 8×8 matrix A and an 8×1 vector B to zero, set a counter overlapCnt=0 and set an accumulator error=0.0.

[0158] 7. For each valid position (x,y) in image I_kdo:

[0159] Compute (x′,y′) using current values of TmpP_kand ρ_k, x′=f_k(x,y) and y′=g_k(x,y).

[0160] In general (x′,y′) are not integers but real numbers.

[0161] If (x′y′) falls within the boundaries of WI_kand if WA_k−1at the four pixels (x′,y′) supporting the nearest pixel (x′,y′) indicate validity, do:

[0162] overlapCnt=overlapCnt+1

[0163] Through bilinear interpolation compute

[0164] I′_k(x,y)=ρ_k*WI_k−1(x′,y′).

[0165] Set error=error+{I_k(x,y)−I′_k(x,y)}²

[0166] Compute partial differentials

[0167] dedp(0)Δδerror/δTmpP_k(0,0)=(x*ρ_k/Den)(δWI_k−1(x′,y′)/δx′

[0168] dedp(1)Δδerror/δTmpP_k(0,1)=(y*ρ_k/Den)(δWI_k−1(x′,y′)/δx′

[0169] dedp(2)Δδerror/δTmpP_k(0,2)=(ρ_k/Den)(δWI_k−1(x′,y′)/δx′

[0170] dedp(3)Δδerror/δTmpP_k(1,0)=(x*ρ_k/Den)(δWI_k−1(x′,y′)/δy′

[0171] dedp(4)Δδerror/δTmpP_k(1,1)=(y*ρ_k/Den)(δWI_k−1(x′,y′)/δy′

[0172] dedp(5)Δδerror/δTmpP_k(1,2)=(ρ_k/Den)(δWI_k−1(x′,y′)/δy′

[0173] dedp(6)Δδerror/δTmpP_k(2,0)=(−x*ρ_k/Den){x′*(δWI_k−1(x′,y′)/δx′)+y′*(δWI_k−1(x′,y′)/δy′)}

[0174] dedp(7)Δδerror/δTmpP_k(2,1)=(−y*ρ_k/Den){x′*(δWI_k−1(x′,y′)/δx′)+y′*(δWI_k−1(x′,y′)/δy′)}

[0175] where the denominator “Den” is given by:

[0176] Den=TmpP_k(2,0)*x+TmpP_k(2,1)*y+TmpP_k(2,2)

[0177] and the partial differentials are computed taking into account the non-integer values of x′ and y′. Next perform the two loops:

[0178] For k=0 to 7, for l=0 to 7, do:

[0179] A(k,l)=A(k,l)+dedp(k)*dedp(l)

[0180] done.

[0181] For k=0 to 7, do:

[0182] B(k)=B(k)−error*dedp(k)

[0183] done.

[0184] done.

[0185] done.

[0186] 8. If (error/overlapCnt)<bestMSE,

[0187] bestMSE=error/overlapCnt

[0188] bestP_k=TmpP_k

[0189] Otherwise do not change bestMSE and bestP_k.

[0190] 9. Update TmpP_k:

[0191] TmpP_k=TmpP_k+T_s*(A^TA)⁻¹A^TB

[0192] where T_sis a selectable stepsize for updating TmpP_k, such as 0.5.

[0193] 10. If the changes to TmpP_kare significant, i.e., max{(A^TA)⁻¹A^TB}>T_dand if cnt<T_c, continue the iterations by going to step 6. Otherwise the iterations may be terminated and go to the next step. Representative values for T_dand T_care 1.0⁻¹⁰and 10.

[0194] 11. If the fraction number of pixels predicted, (overlapCnt)/(Number of samples in I_kwith A_k≧T_a)≧T_fset P_k=bestP_k. Otherwise P_kcould not be changed based on the iterations just performed. Here T_fis a threshold which equalled 0.5 in one implementation.

[0195] The quantity “error” above is not guaranteed to be monotoniclly decreasing with iterations. For this reason matrices TmpP_k, bestP_kand quantity bestMSE are used. These quantities enable the capture of the value of P_kthat results in minimum mean-squared-error while performing the iterative procedure above.

[0196] In the pasting block 38 the images I_k, U_k, V_kand A_kare combined with corresponding components of W_k−1using P_kand ρ_kto generate W_k:

[0197] 1. Compute the corresponding positions of the four corner pixels (x_i,y_i) of I_kin W_k−1(x′_i,y′_i), as illustrated in FIG. 11, using P_kand

[0198] x′=f_k(x,y)

[0199] y′=g_k(x,y)

[0200] (x₀,y₀)=(0,0)

[0201] (x₁,y₁)=(width-1,0)

[0202] (x₂,y₂)=(width-1,height-1)

[0203] (x₃,y₃)=(0,height-1)

[0204] where width and height refer to the width and height of I_k. Values of (x′_i,y′_i) are forwarded to the refresh decider block 40.

[0205] 2. Compute max_i{x′_i}, max_i{y′_i}, min_i{x′_i} and min_i{y′_i}.

[0206] 3. If min_i{x′_i}<0, leftgrow=min_i{x′_i}, otherwise leftgrow=0.

[0207] 4. If min_i{y′_i}<0, topgrow=min_i{y′_i}, otherwise topgrow=0.

[0208] 5. If max_i{x′_i}≧WidthofW_k−1, rightgrow=max_i{x′_i}+1−WidthofW_k−1, otherwise rightgrow=0.

[0209] 6. If max_i{y′_i}≧HeightofW_k−1, bottomgrow=max_i{y′_i}+1−HeightofW_k−1, otherwise bottomgrow=0.

[0210] 7. Pad W_k−1and HITS in the left with leftgrow columns of pixels, in the right with rightgrow columns of pixels, in the top with topgrow rows of pixels, and in the bottom with bottomgrow rows of pixels. This involves padding all the four components of W_k−1and HITS with the above amounts. As padding material black pixels are used, i.e., WI_k−1, WA_k−1and HITS are padded with 0 while WU_k−1and WV_k−1are padded with 128 if the source material is 8-bits per component.

[0211] 8. Update P_kdue to change in dimensions of W_k−1:

[0212] P_k(0,0)=P_k(0,1)+leftgrow

[0213] P_k(0,1)=P_k(0,1)+leftgrow

[0214] P_k(0,2)=P_k(0,2)+leftgrow

[0215] P_k(1,0)=P_k(1,0)+topgrow

[0216] P_k(1,1)=P_k(1,1)+topgrow

[0217] P_k(1,2)=P_k(1,2)+topgrow

[0218] 9. Project {I_k,U_k,V_k,A_k} to make them look like W_k−1by computing the inverse P_k,invto P_k:

[0219] P_k,inv(0,0)=(P_k(1,1)*P_k(2,2)−P_k(2,1)*P_k(1,2))/(P_k(0,0)*P_k(1,1)−P_k(1,0)*P_k(0,1))

[0220] P_k,inv(0,1)=(P_k(0,2)*P_k(2,1)−P_k(2,2)*P_k(0,1))/(P_k(0,0)*P_k(1,1)−P_k(1,0)*P_k(0,1))

[0221] P_k,inv(0,2)=(P_k(1,2)*P_k(0,1)−P_k(0,2)*P_k(1,1))/(P_k(0,0)*P_k(1,1)−P_k(1,0)*P_k(0,1))

[0222] P_k,inv(1,0)=(P_k(1,2)*P_k(2,0)−P_k(1,0)*P_k(2,2))/(P_k(0,0)*P_k(1,1)−P_k(1,0)*P_k(0,1))

[0223] P_k,inv(1,1)=(P_k(2,2)*P_k(0,0)−P_k(0,2)*P_k(2,0))/(P_k(0,0)*P_k(1,1)−P_k(1,0)*P_k(0,1))

[0224] P_k,inv(1,2)=(P_k(0,2)*P_k(1,0)−P_k(1,2)*P_k(0,0))/(P_k(0,0)*P_k(1,1)−P_k(1,0)*P_k(0,1))

[0225] P_k,inv(2,0)=(P_k(2,1)*P_k(1,0)−P_k(1,1)*P_k(2,0))/(P_k(0,0)*P_k(1,1)−P_k(1,0)*P_k(0,1))

[0226] P_k,inv(2,1)=(P_k(0,1)*P_k(2,0)−P_k(0,0)*P_k(2,1))/(P_k(0,0)*P_k(1,1)−P_k(1,0)*P_k(0,1))

[0227] P_k,inv(2,2)=1

[0228] Next I_k,U_k,V_k,A_kare projected to form WI′_k, WU′_k, WV′_k, WA′_kthat appear like WI_k−1, WU_k−1, WV_k−1, WA_k−1respectively:

[0229] WI′_k(x,y)−(1.0/ρ_k)I_k(f_k,inv(x,y),g_k,inv(x,y))

[0230] WU′_k(x,y)=(1.0/ρ_k)U_k(f_k,inv(x,y),g_k,inv(x,y))

[0231] WV′_k(x,y)=(1.0/ρ_k)V_k(f_k,inv(x,y),g_k,inv(x,y))

[0232] WA′_k(x,y)=(1.0/ρ_k)A_k(f_k,inv(x,y),g_k,inv(x,y))

[0233] where f_k,invand g_k,invare defined through:

[0234] f_k,inv(x,y)=(P_k,inv(0,0)*x+P_k,inv(0,1)*y+P_k,inv(0,2))/P_k,inv(2,0)*x+P_k,inv(2,1)*y+P_k,inv(2,2))

[0235] g_k,inv(x,y)=(P_k,inv(1,0)*x+P_k,inv(1,1)*y+P_k,inv(1,2))/(P_k,inv(2,0)*x+P_k,inv(2,1)*y+P_k,inv(2,2))

[0236] Bilinear interpolation is used to obtain values of I_k,U_k,V_k,A_kat non-integer values of f_k,inv(x,y) and g_k,inv(x,y).

[0237] 10. Combine WI′_k, WU′_k, WV′_k, WA′_kwith WI_k−1, WU_k−1, WV_k−1, WA_k−1through weighted summation. Specifically if WA′_k(x,y)≧T_aand WA_k−1≧T_a,

[0238] WI_k(x,y)=(1.0−γ_i)*WI_k−1(x,y)+γ_i*WI′_k−1(x,y)

[0239] WU_k(x,y)=(1.0−γ_u)*WU_k−1(x,y)+γ_u*WU′_k−1(x,y)

[0240] WV_k(x,y)=(1.0−γ_v)*WV_k−1(x,y)+γ_v*WV′_k−1(x,y)

[0241] WA_k(x,y)=(1.0−γ_a)*WA_k−1(x,y)+γ_a*WA′_k−1(x,y)

[0242] HITS(x,y)=HITS(x,y)+1

[0243] where γ_i, γ_u, γ_vand γ_aare selectable scalars in the range 0 to 1. Otherwise if WA′_k(x,y)≧T_a

[0244] WI_k(x,y)=WI′_k(x,y)

[0245] WU_k(x,y)=WU′_k(x,y)

[0246] WV_k(x,y)=WV′_k(x,y)

[0247] WA_k(x,y)=WA′_k(x,y)

[0248] otherwise

[0249] WI_k(x,y)=WI_k−1(x,y)

[0250] WU_k(x,y)=WU_k−1(x,y)

[0251] WV_k(x,y)=WV_k−1(x,y)

[0252] WA_k(x,y)=WA_k−1(x,y)

[0253] Some special cases of the scalars are:

[0254] (a) γ=0. This means copying or propagating old pixel values available in W_k−1to W_kas much as possible, implying that older values in the planar image are given importance and are retained.

[0255] (b) γ=0.5. This gives equal weights to the accumulated information from old input images and the information in the new input image.

[0256] (c) γ=1.0/2. This means the contribution coming from any particular image in the planar image falls off exponentially as the process goes through the sequence of input images. This particular value provides the largest value with convergence property to the temporal integration, i.e., summation, happening at each position of the planar image.

[0257] (d) γ=1. This means destroying old pixel values in W_k−1as much as possible and copying the new information available in I_kinto the planar image.

[0258] (e) γ=1/HITS(x,y). This value give equal weight to each image that contributes to location (x,y) in the planar image.

[0259] The decision to terminate the evolution of the current 2-D planar image and start a new planar image is made by the refresh decide block 40, and is based on several criteria:

[0260] 1. End of Sequence. The process is terminated if all the input images in the input video have been processed.

[0261] 2. Scene Change. If a scene change is identified by the scene cut detector 12, a new planar image is created for the subsequent scene shot.

[0262] 3. Memory Constraints. A decision is made to refresh the planar image if any of the following are true:

[0263] (a) WidthofW_k−1≧T_wid,

[0264] (b) HeightofW_k−1≧T_hei,

[0265] (c) AreaofW_k−1≧T_area

[0266] where the thresholds are selectable, for example, may be 2048, 2048 and 2²¹respectively.

[0267] 4. Excessive camera zoom-in or zoom-out. If there is a large zoom-out happening in the video with respect to the first image, the planar image size grows big, even after preprocessing only a few input images, making the planar image inefficient for compression purposes. Also the resulting planar image is blurred. On the other hand if there is a large zoom-in happening with respect to the first image, the updates to the planar image are small from picture to picture. The process of pasting onto the planar image is quite lossy for the details (higher spatial frequency information) present in the input image. Therefore when a large zoom-in or zoom-out is detected with respect to the first input image, a decision is made to refresh the planar image. Based on the values of (x′_i,y′_i) computed above in the refinement process, the detection of the presence of large zoom-in or zoom-out is made as follows:

[0268] (i) denoting by width and height the widths and heights of I_k, a large zoom-in is determined if any of the following are true:

[0269] (a) min{SQRT((x′₀−x′₃)²+(y′₀−y′₃)²),SQRT((x′₁−x′₂)²+(y′₁−y′₂)²)}≦(T_minh*height)

[0270] (b) min{SQRT((x′₀−x′₁)²+(y′₀−y′₁)²),SQRT((x′₃−x′₂)²+(y′₃−y′₂)²)}≦(T_minw*width)

[0271] (c) area of the polygon {(x′₀,y′₀),(x′₁,y′₁),(x′₂,y′₂),(x′₃,y′₃)}≦(T_mina*area)

[0272] where the thresholds are selectable, i.e., all equal to 0.5 in one implementation. The height, width and area (height*width) refer to I_k.

[0273] Similary a large zoom-out is identified for any of the following:

[0274] (a) max{SQRT((x′₀−x′₃)²+(y′₀−y′₃)²),SQRT((x′₁−x′₂)²+(y′₁−y′₂)²)}≧T_maxh*height)

[0275] (b) max{SQRT((x′₀−x′₁)²+(y′₀−y′₁)²),SQRT((x′₃−x′₂)²+(y′₃−y′₂)²)}≧(T_maxw*width)

[0276] (c) area of the polygon {(x′₀,y′₀),(x′₁,y′¹),(x′₂,y′₂),(x′₃,y′₃)}≧(T_maxa*area)

[0277] where the thresholds are selectable, such as all being 2. Finally excessive zoom-in or zoom-out may also be detected by accumulating the 3-D translation vector t_zcomponents which indicate 3-D zoom-in or zoom-out, depending on their signs. This is only available when the SFM algorithm is being used.

[0278] 5. Excessive camera rotation. If there is excessive rotation in 3-D by the camera with respect to the first image, there is excessive slew in the polygon area of the polygon {(x′₀,y′₀),(x′₁,y′₁),(x′₂,y′₂),(x′₃,y′₃)}. This implies that the planar image is missing some details present in the input images. The excessive rotation of the camera may be detected either based on cumulated R_k−1's or based on the angles of the polygon. The rotation parameters are only available when the SFM algorithm is being used. If the minimum of four angles at vertices of the polygon is smaller than a given threshold, and excessive skew is detected, and a decision is made to refresh the planar image. This threshold may be 30°.

[0279] Thus the present invention provides for 2-D generation of extended images from a video sequence by processing pre-segmented objects from images in the video sequence using a surface-from-motion algorithm and, from resulting depth maps and camera motion parameters for each image, generating the 2-D extended image for flat or background objects.