Method for generated resolution enhanced still images from compressed video data

[0002] The present invention relates to computer processing of image data, and more particularly, to a method for generating still images from compressed video data.

[0003] The storage and data processing capabilities of personal computers have increased to the point that is now possible to provide video processing on such computers. Computer systems that can capture video from cameras, recorders, or television tuners are currently available. Video signals usually have very low quality. The video appears to be of better quality, because the errors from frame to frame are “averaged” by the viewer's eyes.

[0004] There is a growing interest in creating high quality still images from video sequences. Since each video frame has relatively poor resolution, the information in a number of frames is combined to generate a still image having a resolution that is much better than the individual frames. The prior art methods for combining the frames operate on uncompressed data in the “pixel” domain.

[0005] While storage costs have decreased considerably, the amount of data inherent in a video sequence is still too large to allow more than short video clips to be stored without some form of compression. Accordingly, many video capture systems output compressed video data. The compression algorithms reduce the high frequency information, which is needed to create high-quality still images, and hence, further complicate the still image construction problem. In addition, the compressed video contains quantization information that is not easily utilized in the pixel domain.

[0006] Broadly, it is the object of the present invention to provide an improved still image reconstruction system.

[0007] It is a further object of the present invention to provide a still image reconstruction system that operates on compressed video data.

[0008] It is a still further object of the present invention to provide a still image reconstruction system that takes into account the quantization information in the compressed video sequence.

[0009] These and other objects of the present invention will become apparent to those skilled in the art from the following detailed description of the invention and the accompanying drawings.

[0010] The present invention is a method for generating a high-resolution image from a plurality of low-resolution images generated from quantized coefficients in the spatial frequency domain and the uncertainties in the quantized coefficients. The method generates a set of linear inequalities relating the quantized coefficients, the uncertainties therein, and the pixel values for the high-resolution image. Each linear inequality relates one of the quantized coefficients to a linear sum of the high-resolution pixel values. A high-resolution image whose pixels satisfy this set of inequalities is then generated.

[0011] FIG. 1 is a block diagram of a typical MPEG motion picture compression algorithm utilizing motion compensation.

[0012] In the following discussion, it will be assumed that a continuous motion picture sequence F(x₁,x₂,t) has been captured in the form of a video stream. The video stream consists of a sequence of discrete frames having pixels that are determined by a discrete sampling lattice, D. The individual frames of the video sequence will be denoted by g(m₁,m₂,k), where m₁, m₂, and k are integers. This sequence is compressed by the MPEG compression algorithm to generate a compressed video stream, which will be discussed in detail below.

[0013] The goal of the present invention is to reconstruct a high-resolution version of a frame within this sequence. Denote the high-resolution frame by f. In general, this frame will correspond in time to one of the frames in the video sequence discussed above. The actual output of the present invention is a discretized frame f(n₁, n₂) where (n₁, n₂) are specified by a second sampling lattice which has a higher resolution than D.

[0014] To obtain the high-resolution frame, the data in a number of frames adjacent to f are combined. It is assumed that the scene from which f is to be extracted is changing slowly enough that the pixels of f are also present in the frames on each side of f; although the locations of the pixels may be different. For example, if the camera recording the scene pans and f is located in the middle of the pan sequence, the pixels in f will also be present in the frames on each side of f at locations displaced from the locations in f. It should be noted that not all of the pixels of f are available in all of the frames on each side.

[0015] The relationship of the pixels in the frames surrounding the frame corresponding to f may be represented by a continuous, linear, shift-variant blur function. The actual motion in the video will determine the location and possibly the form of the blur h. The blur function will be discussed in more detail below.

[0016] The manner in which the present invention achieves its advantages can be more easily understood with reference to FIG. 1, which illustrates the manner in which a video stream is processed to generate an MPEG compressed video stream utilizing motion compensation. In the figure, F(x₁,x₂,t) denotes the original continuous motion picture scene. This scene is “captured” by the camera and sampled on a lattice, D, to generate a discrete sequence of frames g(m₁,m₂,k), where (m₁,m₂) are determined by the lattice and k denotes the frame number in the sequence of frames.

[0017] The camera output, g, is compressed in two steps. In the first step, each frame is approximated using a motion compensation algorithm, which attempts to represent the frame in terms of blocks from a reference frame. Each frame is divided into blocks. One of the frames in the sequence is defined to be a reference scene. Each block in the frame being compressed is compared to blocks of the same size in the reference frame to find the block that most closely approximates that block. The block in the reference frame is defined by providing a motion vector, which is the displacement of the block in the current frame relative to the block in the reference frame. The difference of the two blocks is then taken. If the block in the reference frame is a good approximation of the current block, the difference “image” will have much less intensity that the original block. If the match is poor, the difference image will be no more intense than the original block, hence, little is lost by such an approximation even when the match is poor.

[0018] The difference image is then transformed using a discrete cosine transform (DCT). In the MPEG standard, 8×8 pixel blocks from the motion compensated image are transformed by the DCT. The resulting 8×8 blocks of DCT coefficients, d, are quantized using a quantization algorithm, Q, in which each coefficient is converted to an integer by dividing the coefficient by a step size that depends on the coefficient's position in the 8×8 block. The quantization introduces errors into the reconstructed image, since two DCT coefficient values that differed by less than the step size are now set to the same value.

[0019] As noted above, the frames on each side of f in the sequence can be represented in terms of f and a blur function h. From the above discussion, the discretized output of the camera can be written as:

[0020] g(m₁,m₂,k)=∫h(x₁,x₂;m₁,m₂,k)f(x₁,x₂)dx₁,dx₂+v(m₁,m₂,k)  (1)

[0021] where v denotes additive noise, and the blur h, implicitly applies to some reference time for f. It should be noted that discrete indices m₁, m₂, and k are applied to the continuous spatio-temporal coordinates x₁and x₂through the sampling lattice D. Note that the true scene motion is taken into account via h. Since the actual data is all discrete, it is useful to convert Eq. (1) to a discrete form, consider the discretization method so that the continuous spatial coordinates x₁and x₂can be avoided.

[0022] Define an estimate, {circumflex over (f)}, by $\begin{array}{cc}\hat{f}\left(x_1,x_2\right)=\left(f\left(x_1,x_2\right)\sum _{n_1,n_2}\delta \left(x_1-n_1,x_2-n_2\right)\right)*h_r\left(x_1,x_2\right),& \left(2\right)\end{array}$

[0023] where {circumflex over (f)} is an estimate of f after the Dirac-sampled version is reconstructed using h_r, and where the ordered pair (n₁,n₂)specifies a location determined by the lattice used to specify locations in (x₁,x₂). Note that this lattice is most likely rectangular, however, other lattices may be utilized. By using this representation, it can be shown that: $\begin{array}{cc}{h}_d\left(n_1,n_2;m_1,m_2,k\right)={\left[h\left(x_1,x_2;m_1,m_2,k\right)*h_r\left(x_1,x_2\right)\right]}_{\left(x_1,x_2\right)\Rightarrow \left(n_1,n_2\right)},g\left(m_1,m_2,k\right)=\sum _{n_1,n_2}f\left(n_1,n_2\right)h_d\left(n_1,n_2;m_1,m_2,k\right)+v\left(m_1,m_2,k\right),& \left(3\right)\end{array}$

[0024] where h_ddenotes the discretized blur function, and where the convolution occurs on (x₁,x₂) and is evaluated on the high-resolution sampling lattice indexed by (n₁,n₂).

[0025] As will be explained in more detail below, the present invention makes use of the quantization information contained in the MPEG representation of the video sequence. In particular, the present invention provides a relatively simple method for taking the quantization errors into account while determining the best high-resolution image that could have given rise to the MPEG encoded data.

[0026] The manner in which the MPEG data is utilized can be more easily understood from a more detailed description of the MPEG coding algorithm. As noted above, each block in the current frame is matched to a block in a reference frame, which is subtracted from the image frame. Denote the motion compensated prediction by ĝ ;(m₁, m₂, k). Then:

[0027] g_m(m₁,m₂k)=g(m₁,m₂k)−ĝ ;(m₁,m₂k)+v(m₁,m₂,k).  (4)

[0028] The MPEG algorithm then takes the 8×8 block-based DCT of the motion compensated block to arrive at the DCT coefficients, d(m₁,m₂,k) that are to be quantized. Taking the DCT transform of Eq. (4) yields: $\begin{array}{cc}d\left(m_1,m_2,k\right)=G\left(m_1,m_2,k\right)-\hat{G}\left(m_1,m_2,k\right)+V\left(m_1,m_2,k\right),G\left(m_1,m_2,k\right)=\sum _{n_1,n_2}f\left(n_1,n_2\right)\{\sum _{l_1=L\left(m_1\right)}^{L\left(m_1\right)+7}\sum _{l_2=L\left(m_2\right)}^{L\left(m_2\right)+7}{C}_{8\mathrm{x8}}\left(m_1\%8,m_2\%8;l_1,l_2\right)& \left(5\right)\end{array}$

[0029] where G, Ĝ ;, and V denote the 8×8 DCT of g, g_mand v, respectively, C_8×8is the 8×8 DCT coefficient generating function, and where the limit function is defined by:

[0030] L(m)=(mdiv8)*8.  (6)

[0031] If one defines the blur operator by $\begin{array}{cc}{h}_{\mathrm{DCT}}\left(n_1,n_2;m_1,m_2,k\right)=\sum _{l_1=L\left(m_1\right)}^{L\left(m_1\right)+7}\sum _{l_2=L\left(m_2\right)}^{L\left(m_2\right)+7}{C}_{8\mathrm{x8}}\left(m_1\%8,m_2\%8;l_1,l_2\right)h_d(n_1,n_2;l_1,l_2,k),& \left(7\right)\end{array}$

[0032] Then $\begin{array}{cc}d\left(m_1,m_2,k\right)=\sum _{\left(n_1,n_2\right)}f\left(n_1,n_2\right)h_{\mathrm{DCT}}\left(n_1,n_2;m_1,m_2,k\right)+\mathrm{DCT}\left\{\hat{g}\right\}+\mathrm{DCT}\left\{v\right\}.& \left(8\right)\end{array}$

[0033] Here, “m %8” denotes m modulo 8.

[0034] It should be noted that the MPEG data consists of the quantized versions of d(m₁,m₂,k), d_q(m₁,m₂,k). As noted above, these values may be in error by as much as one quantization step. That is,

[0035] d_q(m₁,m₂,k)−S(m₁,m₂,k)/2≦d(m₁,m₂,k)≦d_q(m₁,m₂,k)+S(m₁,m₂,k)/2  (9)

[0036] where S(m₁,m₂,k) is the step size used in quantizing d(m₁,m₂,k). Hence, given h_d, Eq.(9) defines a set of inequalities that can be used to compute f(n₁,n₂). The present invention is based on the observation that this set of inequalities is particularly simple in the DCT domain.

[0037] In prior art methods for generating high-resolution images from a plurality of low-resolution images, a set of equations that are analogous to Eqs. (8 )-(9) are constructed and an attempt made at a solution thereof in the pixel domain. In the pixel domain, the equivalent analysis leads to a maximum likelihood fit that depends on a non-Guassian probability distribution. The present invention avoids this complexity by operating in the DCT domain.

[0038] The function, h_d, is only valid where the motion can be accurately estimated. Additionally, motion boundaries where occlusion and uncovered background exist are particularly difficult to model. As a result, Eq. (8) will not be valid for every observation location (m₁,m₂,k). In the preferred embodiment of the present invention, the set of inequalities is only generated for the points in the reference frame at which the motion modeling is known to be valid. The manner in which h_dis computed from a motion sequence is known to those skilled in the art, and hence, will not be discussed in detail. The reader is directed to Andrew Patti, et al., “Superresolution Video Reconstruction with Arbitrary Sampling Lattices and Nonzero Aperture Time”, IEEE TRANSACTIONS ON IMAGE PROCESSING, 6, pp. 1064-1075, 1997 for a more detailed discussion. This article is reproduced in Appendix A.

[0039] In the preferred embodiment of the present invention, the motion of a pixel from one frame to another is computed using an optical flow model. In such a model the position of a the pixel located at (x₁,y₁) in frame 1 is determined in frame 2 by assuming that the intensity of the pixel in frame 2 is the same as that in frame 1 and by assuming some model for the motion of the pixel, i.e.,

[0040] x₂=x₁+u(x₁, y₁)

[0041] and

[0042] y₂=y₁+v(x₁, y₁)  (10)

[0043] where u(x,y) and v(x,y) are the x and y motion vectors. The motion estimation operates by assuming a model for u and v and then testing how well that model actually performs in estimating the motion of the pixel in question. If the model accurately predicts the motion, then an inequality for the point in question is included in the set of inequalities discussed above.

[0044] In the preferred embodiment of the present invention, it is assumed that a block of pixels around the pixel of interest moves in the same manner as the pixel in question. Under this assumption, the displacement of the pixel in frame 2 is determined by defining a block of pixels around the pixel of interest at (x₁,y₁) in frame 1 and then searching for a block of the same size in frame 2 that matches this block. If the model is valid, the measured differences between the two blocks should be less than some predetermined value. If this condition is met, than a value for (u,v) between the first and second frames for the pixel in question can be deduced. This procedure may be repeated for each frame to determine the motion of a particular pixel in the reference frame. This set of displacements defines the motion trajectory utilized in computing the blur function h discussed with reference to Eq. (1) above.

[0045] While the preferred embodiment of the present invention assumes that all of the pixels in a given block move with the same speed between frames, other models may be utilized. For example, u(x,y) and v(x,y) can be expanded as Taylor series in (x,y) and the parameters of the series determined by matching pixels in the successive frames to determine the best fit for the parameters. Other motion estimation models will also be apparent to those skilled in the art.

[0046] Any method for solving the set of inequalities defined by Eq. (8) to obtain an estimate of f(n₁,n₂) may be used. The preferred method for solving the inequalities represented by Eqs. (8)-(9) is the method of projection on convex sets (POCS). Since this method is known to those skilled in the art, it will not be discussed in detail here. The reader is directed to Sezan, “an overview of convex projections theory and its applications to image recovery problems”, Ultramicroscopy, no 40, pp. 55-67, 1992 for a more detailed discussion. The method is an iterative method that starts from an estimation of f(n₁,n₂). This estimation is provided by interpolating the reference frame to provide values on the f(n₁,n₂) grid. It should be noted that the set of inequalities may not include inequalities that define f(n₁,n₂), if the motion estimation failed for pixels in the vicinity of (n₁,n₂) in the reference frame. In this case, the value of f(n₁,n₂) will not be updated by POCS algorithm. Hence, the quality of the image in the region in question will not improve. It is important to note, however, that the quality of the image in these regions will not be degraded either.

[0047] The above-described embodiments of the present invention have utilized optic flow based motion estimation. However, it will be obvious to those skilled in the art from the preceding discussion that any form of motion estimation may be utilized.

[0048] Various modifications to the present invention will become apparent to those skilled in the art from the foregoing description and accompanying drawings. Accordingly, the present invention is to be limited solely by the scope of the following claims.