Method and Apparatus for Compression of Low Bit Rate Video Signals

This invention relates generally to the processing of video signals. More particularly, this invention relates to the processing of low bit rate video signals of the type that are used in personal video communication devices, such as video telephones and wireless communication systems.

Recent advances in computer networks and signal compression technology have made point to point digital video communication possible. Applications for this new technology can be classified by the amount of bandwidth available for transmission. A broadcast quality video-on-demand system requires between 2 and 8 megabits per second (Mbits/s) for typical television sourced material. Commercially available vidoeconferencing systems typically use a few hundred kilobits per second (kbits/s) to represent several people moving and speaking independently. At the low end of the bandwidth scale is personal video communication. An example application of this technology is the "video telephone", which allows people to communicate using both moving video and sound. The actual medium for data transfer might be ordinary telephone lines, a wireless system, or a computer network. The bandwidth available for such systems is very restrictive, typically 20 kbits/s and below. The present invention is directed toward the processing of video signals at these low bit rates.

There are ongoing efforts to standardize digital video compression techniques. The majority of these efforts focus on high end applications. The MPEG 1 and 2 committees, for example, specified systems for transmitting data at rates of 1.4 Mbits/s and higher. Some attempts have been made to standardize lower end applications. For example the CCITT developed recommendation H.261 to function at multiples of 64 Kbits/s, and recommendation H.263 to function at still lower rates. All of these attempts at standardization use a technique known as motion compensated DCT. This technique uses a previous video frame to construct a current video frame, with only the difference in signal content between the two video frames being sent over the transmission channel. The difference in signal content is coded using the Discrete Cosine Transform (DCT). Unfortunately, such motion compensated DCT systems do not perform well at the very low bit rates needed for personal video communication. At these rates, the transmitted signal is represented with fewer significant DCT coefficients, each with a very coarse quantization. This results in noticeable distortion and visible block artifacts. Thus, it would be highly desirable to have a signal compression technique for low bit rate video signals that does not rely upon DCT coding and can otherwise avoid the distortion and visible block artifacts associated with DCT coding.

The invention is an improved apparatus and method to compress a video signal, particularly a low bit rate video signal. The technology is analogous to motion compensated DCT (Discrete Cosine Transform) techniques. However, instead of relying upon a DCT coder to process a motion residual signal, the motion residual signal is processed by a pattern matcher. The pattern matcher identifies regions in the motion residual signal that have not been accurately reproduced on account of image motion. Each of these regions is compared to a set of library patterns in a pattern library. The pattern matcher links each region with a selected library pattern, which accurately characterizes the region. The pattern matcher forms an atom parameter signal characterizing each selected library pattern. The atom parameter signal is transmitted as a compressed version of the video signal. After transmission, the video signal is reconstructed from the atom parameter signal. In particular, a pattern fetcher is used to interpret the atom parameter signal to identify the selected library patterns used to characterize the motion residual signal. The selected library patterns are combined with other information to reconstruct the video signal.

The method of the invention includes the steps of deriving a frame of a motion residual signal from a frame of a video signal. The highest energy pattern within the frame of the motion residual signal is then identified. The highest energy pattern is then compared to library patterns of a pattern library to identify a matched pattern. A set of variables are then assigned to characterize the matched pattern; the set of variables form an atom parameter signal segment. The foregoing steps are repeated to identify the highest energy patterns within the frame of the motion residual signal. Thereafter, the atom parameter signal segments are combined to form an atom parameter signal. The atom parameter signal is transmitted as a compressed video signal corresponding to the video signal.

Since the disclosed technology is not DCT based, it does not suffer the shortcomings associated with DCT systems operating at low bit rates. For example, with the present invention, the highest energy regions of a motion residual signal are processed first. Thus, only the most significant coefficients are used to characterize the image. This factor is especially significant at low bit rates. An overcomplete pattern library also improves the quality of the transmitted information at low bit rates since the overcomplete library can provide a more accurate representation of the selected pattern than is available by processing the original set of coefficients characterizing the selected pattern, as done in a DCT system.

In addition, by allowing patterns of the pattern library to overlap arbitrarily, the artificial block segmentation used in DCT-based systems is avoided. Further, since patterns can be selected to decay smoothly at the edges, basis edge artifacts are greatly reduced. As a result, the reconstructed image is more natural than comparable images generated by DCT-based systems.

For a better understanding of the nature and objects of the invention, reference should be made to the following detailed description taken in conjunction with the accompanying drawings, in which:

FIG. 1 illustrates the processing associated with one embodiment of the matching pattern video signal compression apparatus of the invention.

FIG. 2 graphically illustrates the process of obtaining a motion prediction signal and a motion vector signal.

FIG. 3 is an example of a motion residual signal frame processed in accordance with the invention.

FIG. 4 illustrates one embodiment of the processing performed by an pattern matcher in accordance with the invention.

FIG. 5 illustrates the division of a motion residual signal frame for the purpose of finding high energy "patterns" within the signal.

FIG. 6 illustrates the location of a selected pattern in a motion residual signal frame.

FIG. 7 illustrates a matching operation for the selected pattern of FIG. 6.

FIG. 8 illustrates parameters that may be used to construct an atom parameter signal.

FIG. 9 illustrates the apparatus of FIG. 1 implemented in software.

FIG. 10 illustrates the horizontal component of a sample library pattern.

FIG. 11 illustrates the vertical component of a sample library pattern.

FIG. 12 illustrates a contour plot of a sample library pattern.

FIG. 13 illustrates a 3-dimensional visualization of a library pattern element corresponding to FIGS. 10-12.

FIG. 14 illustrates a search algorithm that may be used in conjunction with the invention.

Like reference numerals refer to corresponding parts throughout the several views of the drawings.

FIG. 1 illustrates the processing executed by a matching pattern video signal compression apparatus 20 constructed in accordance with one embodiment of the invention. The apparatus 20 includes a matching pattern coder 22 that receives an input video signal and generates a compressed video output signal. The compressed video output signal, specifically comprising a coded motion vector signal and a coded atom signal, is applied to a transmission channel 24, which may be a telephone line, a wireless communications frequency band, a computer network, or similar medium. The transmission channel 24 delivers the compressed video output signal to a matching pattern decoder 26. The matching pattern decoder 26 processes the compressed video output signal to generate a reconstructed video signal, which is subsequently processed in a conventional manner.

FIG. 1 illustrates that the video signal is initially processed by a motion compensator 30 of the matching pattern coder 22. A reference frame signal is also processed by the motion compensator 30. The reference frame signal 42 is the previous frame of the video signal. Instead of transmitting the entire current video frame signal over the transmission channel 24, the motion compensator 30 identifies those regions of the reference frame signal that can be used to construct the current video frame signal. That is, since the reference frame video signal has already been transmitted over the transmission channel 24, an attempt is made to use the transmitted information in constructing the current video frame signal. If this can be done, the only signal that needs to be transmitted over the transmission channel 24 is a signal identifying the region of the reference frame signal that can be used to construct the current frame signal and the difference between the reference frame signal and the current frame signal in that region.

This operation is more fully appreciated with reference to FIG. 2. The current frame signal 40 is divided into blocks. For each block 48 in the current frame signal 40, a search window 46 is created around the corresponding location in the reference frame signal 42. A search block 44 is moved within the search window 46 and the search block 44 location which most closely matches the block 48 in the current frame signal 40 is selected. This location is called a match block. The location of the match block relative to the original block is represented by a motion vector, represented in FIG. 2 by the arrow connecting match block 44 with the current frame block 48. The motion vectors for all the blocks in the current frame signal 40 are compiled into a motion vector signal. The motion vector signal is then transmitted over the transmission channel 24. At the decoder 26, this signal is used to move the search block 44 of the reference frame signal 42 to a position wherein it serves to accurately generate a portion of the current frame signal 40. This process is repeated for each region of the reference frame signal.

The motion compensator 30 also generates a motion prediction signal. The motion prediction signal characterizes the difference between the signal content of the search block 44 and the match block 48. The motion compensator 30 effectively subtracts the pixel values of the search block 44 from corresponding pixel values of the match block 48. If the resultant motion prediction signal has low values it is said to contain low "energy", indicating that there is a close match between the two signals. Conversely, if the resultant motion prediction signal has high values it is said to contain high "energy". A high energy signal is associated with motion between video frames.

Returning now to FIG. 1, illustrated therein is the motion vector signal and motion prediction signal leaving the motion compensator 30. The motion prediction signal is subtracted from the video signal at subtractor 50 to produce a motion residual signal. Thus, the motion residual signal is a signal representing those portions of the reference frame signal that are distinct from the current frame signal. In other words, the motion residual signal identifies high energy regions of the reference frame signal; the high energy regions indicate motion between frames that could not be compensated for by the motion compensator 30.

FIG. 3 is an exemplary motion residual signal 52, more particularly, a frame of a motion residual signal. Note that the signal 52 has several high energy regions 54. The high energy regions 54 correspond to those regions of the video signal that are experiencing motion. It is seen in FIG. 3 that the eyes, mouth, hair, and shoulder regions of an individual have moved from one frame to the next.

In the prior art, a motion residual signal is processed by a DCT coder. Specifically, the motion residual signal is segmented into blocks of data, or pixel blocks, which can be represented by an N.times.N matrix. The matrix is then transformed to the DCT domain through a well-known mathematical operation. The mathematical operation results in an N.times.N matrix of coefficients. Since the input matrix is the same size as the output matrix, the DCT operation is said to operate on a "complete basis".

The present invention does not use a DCT coder. Instead, the motion residual signal is processed by a pattern matcher 60. The pattern marcher 60 accesses a pattern library 80, which includes a large number of signal patterns 82. The large number of signal patterns forms an "overcomplete basis" library, meaning that the input matrix of information applied to the pattern library can result in a larger output matrix of information. This approach stands in contrast to the "complete basis" used in the DCT domain. The benefit of the overcomplete basis library is that the coefficients used to characterize and code the selected input pattern are chosen from a large and diverse collection of functions. Thus, each coefficient is more effective in representing the true patterns in the residual signal. Since each coefficient is more effective, the residual signal can be coded with fewer significant coefficients, which is highly advantageous at low bit rates.

For the sake of simplicity, only four signal patterns 82A, 82B, 82C, and 82D are shown in FIG. 1. The pattern matcher 60 identifies high energy regions of the motion residual signal, called "input patterns". Each input pattern is then compared to the library patterns 82 of the pattern library 80. The closest library pattern 82 in the pattern library 80 is selected, and a weighting coefficient for this pattern is computed by a method described below. The specific pattern, the weighting coefficient, and the exact position of the pattern within the image describe an atom.sunburst.. This atom is used to represent a specific energy structure in the coded version of the motion residual signal. The same processing steps are then repeated for other high energy regions of the motion residual signal. The pattern matcher 60 outputs a coded residual signal which approximates the selected input patterns from the motion residual signal by using the selected atoms. This signal is effectively the coded version of the motion residual signal. The pattern marcher 60 also outputs a coded atom signal, which provides information on the selected atoms.

The operation of the pattern matcher 60 is more fully described in reference to FIG. 4. FIG. 4 illustrates the processing steps executed by the pattern marcher 60. The first processing step is to select an input pattern (block 62). This processing step requires the division of the motion residual signal into seek blocks 64, as shown in FIG. 5. The sum of the squares of all pixel intensities is computed for each seek block 64. The seek block with the largest value is adopted as a selected input pattern 66, as shown in FIG. 6. As shown in FIG. 7, an S.times.S window is then formed around the region of the selected input pattern.

This pre-processing of the motion residual signal frame signal is highly advantageous since it reduces the number of comparisons that must be made with the pattern library and it insures that the most problematic areas of the image are processed. Thus, the most significant coefficients associated with the image are transmitted to the destination site.

The next processing step associated with the pattern matcher 60 is to match a pattern (block 68). That is, the content of the S.times.S window is compared to the different patterns (G) of the pattern library 80 to find the closest match. In the simple example of FIG. 1, it can be seen that pattern 82C is the closest to the pattern of the S.times.S block of FIG. 7. Note that the matching procedure compares each pattern in the pattern library 80 to the content of the S.times.S search window at all possible locations within the window, and thus the exact position of the match must be recorded in addition to the specific pattern matched and the weighting coefficient.

Once the closest match is identified, an atom parameter signal segment is constructed (block 90). The atom parameter signal segment specifies the best match structure element from the pattern library 80. This element is identified with the values (.alpha., .beta.) . The atom parameter signal segment also specifies the location of the best match in the reference frame with (x, y) coordinates. Finally, the atom parameter signal segment specifies the projection of image data at (x, y) onto the selected match element, referred to as "p". That is, the best match element from the pattern library is multiplied by a value "p", as will be further described below.

These parameters are shown in FIG. 8. Their arbitrary arrangement into an atom parameter signal segment 92 is also illustrated in FIG. 8.

As shown in FIG. 4, if bits are still available (decision block 94), processing of another input pattern commences at block 62. That is, the processing of several high energy input patterns is executed to improve the quality of the reconstructed image at the high energy input pattern locations. When this processing is completed, the constructed atom parameter signal is processed (block 96).

The constructed atom parameter signal is processed by the pattern marcher 60 to form a coded residual signal. As indicated above, the coded residual signal corresponds to the motion residual signal, but has atoms substituted for the processed high energy input patterns. These atoms represent the energy structures in the motion residual signal and are used by the decoder to supplement the motion prediction signal when the reconstructed image is generated.

The atom parameter signal is also processed by passing it to an atom coder 100. Typically, the atom coder 100 will receive an atom parameter signal that includes information on a set of sequentially identified atoms.

The atom coder 100 performs known quantization and variable length coding operations. The quantization operation transforms the coefficients of the atom parameter signal into bits. The number of bits is a function of the rate and quality requirement of the video system. Variable length coding is then used to assign short bit patterns to signals with a high probability of occurrence and longer bit patterns to signals with a lower probability of occurrence. Thus, the throughput of the transmission channel 24 can be increased. Quantization and variable length coding operations are also performed on the motion vector signal by the motion vector coder 102.

The adder 104 of the matching pattern coder 22 combines the coded residual signal with the motion prediction signal to reconstruct the video signal. The video signal is then passed to a frame buffer 106 where it is stored until the next signal frame is available. When the next signal frame is available at the motion compensator 30, the reference frame signal from the frame buffer 106 is passed to the motion compensator 30 and processing is repeated.

The matching pattern decoder 26 performs operations which are the inverse of those performed by the matching pattern coder 22. In particular, the atom decoder 110 performs inverse variable length coding and quantization operations to recover the atom parameter signal. The atom parameter signal is then passed to the pattern fetcher 112. The pattern fetcher 112 uses the information in the atom parameter signal to identify a selected pattern in the pattern library 80. The selected pattern is then used to form a coded residual signal.

The motion vector decoder 114 also performs inverse variable length coding and quantization operations, in this case to recover the motion vector signal. The motion vector signal is applied to a motion constructor 116, which also receives a reference frame signal from a frame buffer 120. Thus, the motion constructor 116 moves regions of the reference frame signal in accordance with the instructions provided by the motion vector signal. The result of this processing is a motion prediction signal. The motion prediction signal is combined with the coded residual signal at adder 118 to generate a reconstructed video signal. The motion prediction signal provides the content for the portions of the video signal which are well represented by translating blocks from the reference frame, while the coded residual signal provides information about the video signal in regions where block translation fails.

FIG. 9 illustrates the apparatus of FIG. 1 implemented in software. In particular, the figure illustrates a video signal being applied to a matching pattern coder 22, which includes interface devices 130. The interface devices 130 include physical connections and signal conversion circuitry required for communication with a central processing unit (CPU) 132. The CPU 132 is connected to a memory 134, which includes a set of stored programs and data. For example, one program is the motion compensator 30. This program is a set of executable instructions used to achieve the previously described functions. Similar executable programs are provided for the subtractor 50, the pattern matcher 60, the atom coder 100, the motion vector coder 102, and the adder 104. The pattern library 80 is stored in memory 134. In addition, there is space in the memory 134 for the frame buffer.

The configuration of the matching pattern coder 22 of FIG. 9 is simplified for illustrative purposes. A preferable embodiment has the pattern library 80 and frame buffer 106 implemented addition, the executable program In addition, the executable programs are preferably embedded in silicon. Thus, for example, the motion compensator 30 and pattern matcher 60 are implemented as separate integrated circuits.

After processing by the CPU 132, the video signal is applied to the interface devices 130, which are used to pass the coded atom signal and coded motion vector signal to the transmission channel 24. The matching pattern decoder 26 of FIG. 9 has a similar construction and operates under similar principals as those used by the matching pattern coder 22.

The invention has now been fully described. Attention presently turns to implementation details associated with the disclosed technology. As indicated above, the pattern matcher 60 matches selected input patterns with patterns in a pattern library. Preferably, the pattern matcher 60 executes a "matching pursuits algorithm". The "matching pursuits algorithm" is described by S. Mallat and Z. Zhang, in "Matching Pursuits with Time-Frequency Dictionaries", IEEE Transactions in Signal Processing, Vol. 41, No. 12, Dec. 1993. The matching pursuits algorithm expands a signal using an overcomplete basis of Gabor functions. The functions are scaled, shifted, and modulated versions of a Gaussian window function.

A prototype window g(t) is defined as: ##EQU1## This is a Gaussian function of unit norm. The triple .gamma.=(s,u,.xi.).epsilon.R.sup.+ .times.R.times.R is defined where R is the set of all real numbers and R.sup.+ is the set of positive reals. Here s represents a positive scale value, u is a time shift, and .xi. is a modulation frequency. The Gabor basis set consists of all g.sub..gamma. (t) defined by: ##EQU2## An expansion is commenced by matching the signal f(t) to the most appropriate basis function g.sub..gamma. (t) in the dictionary. In other words, a selected input pattern 66 (f(t)) is matched with the most appropriate pattern 82 (g.sub..gamma. (t)) in the pattern library 80. This is accomplished by choosing .gamma. to maximize the absolute value of the following inner product:

As known in the art, an inner product can be alternately defined as: ##EQU3## Since discrete pixels are being processed, the inner product is more accurately defined as: ##EQU4## The term p is then used as an expansion coefficient for the signal onto the dictionary function g.sub..gamma. (t). A residual signal is computed as:

Equation (4) indicates that the matched pattern from the pattern library is multiplied by the coefficient p. The resultant product is subtracted from the motion residual signal f(t) so that processing of the remaining high energy patterns can proceed.

This procedure continues iteratively until either a set number of expansion coefficients are generated or some energy threshold for the residual is reached. Each of the M stages yields a dictionary structure specified by .gamma..sub.n, an expansion coefficient p.sub.n, and a residual R.sub.n which is passed on to the next stage. This signal can then be approximated by a linear function of the dictionary elements: ##EQU5## Equation five indicates that the atoms identified in the processing of the high energy input patterns are combined to reconstruct the motion residual signal. Note that the M dictionary structures g.sub..gamma.n (t) are not necessarily orthogonal. The approximation in Equation 5 generally improves as M is increased.

The pattern library 80 consists of discrete 2-D basis functions (library patterns 82). Since the matching pursuit technique requires exhaustive inner-product searches in the library 80, the library 80 must be limited to a reasonable number of library patterns 82, each with a finite spatial extent. To reduce the search time, each basis function 82 should be separable. To meet these requirements, a separable product of real, discrete Gabor functions with N.times.N support is defined. These are based on 1-D sequences of length N: ##EQU6## where g(t) is the prototype Gaussian window from Equation 1, and .alpha.=(s, .xi., .phi.) is a triple consisting respectively of a positive scale, a modulation frequency, and a phase shift. B is defined to be the set of all such triples. The constant K.sub..alpha. is chosen such that the resulting sequence is of unit norm. Also, the index i is shifted by (N/2-1) to effectively center the Gaussian envelope and allow the resulting sequence to decay smoothly at both edges. Sequences as defined by Equation 6 may be used to separably construct 2-D discrete functions with N.times.N support. A dictionary set may be composed of these 2-D structures: ##EQU7## Examples of the structures defined in Equations 6 and 7 are shown in FIGS. 10-13. FIG. 10 shows the 1-D horizontal component g.sub..alpha. (i) with .alpha.=(5,0,0). FIG. 11 shows the 1-D vertical component g.sub..beta. (j) with .beta.=(3,0,0). A contour plot of the resulting 2-D basis structure G.sub..alpha..beta. (i,j) is shown in FIG. 12, and a 3-D visualization of the structure is shown in FIG. 13. This structure is completely specified by .alpha.and .beta..

The matching pursuit technique is preferably used to match the pattern library of such structures to energy patterns in the motion residual signal. During the matching operation, the structure is centered at any integer-pixel location in the image. The result is a large, diverse basis set. Note that unlike the DCT technique, this technique is not block based.

To keep the search time reasonable, the dictionary size is limited. Preferably, the dictionary set is defined to include all G.sub..alpha..beta. (i,j) with .alpha., .beta..epsilon.B', where B' is a finite subset of B.

To keep the search time reasonable, the dictionary size is limited. Preferably, the dictionary set is defined to include all G.sub..alpha..beta. (i,j) with .alpha., .beta..epsilon.B', where B' is a finite subset of B.

The reduced dictionary set B' is constructed in the following manner. A large subset of B containing evenly spaced instances of the three parameters (s, .xi., .phi.) is used to define a 2-D dictionary with N=16. A set of motion residual images from a training sequence is then decomposed on this dictionary using matching pursuits. The fifteen parameter triples which are most often selected by the matching algorithm are retained in a reduced set, denoted B'. This set is shown in Table 1.

The dictionary defined by the triples in Table 1 is much larger than the standard block-DCT basis. To see this, consider the decomposition of a 176.times.144 pixel image. An 8.times.8 pixel block-DCT basis must represent the image using a set of 64 2-D basis functions, each of which can be placed at 396 possible block locations in the image. This defines a DCT dictionary containing 25344 basis structures. The matching pursuit dictionary defined above uses 225 2-D functions, each with 176.times.144 possible image locations. This set contains 5.7 million basis structures. More importantly, the structures in the matching pursuit basis do not conform to artificial block shapes or grid locations. The various scale parameters s.sub.i used in Table 1 produce structures in a variety of sizes and shapes. These are then matched to similar structures at all possible locations in the motion residual image.

A dictionary of 2-D structures which will be used to decompose motion residual images using matching pursuits has been defined. A direct extension of the matching pursuit algorithm requires an exhaustive computation of the inner product of each structure in the dictionary with all possible N.times.N regions in the residual image. In order to reduce this search to a manageable level, assumptions are made about the residual image to be coded. Specifically, it is assumed that the image is sparse, containing pockets of energy at locations where the motion prediction model was inadequate, for example, as shown in FIG. 3. As indicated previously, this sparse image is processed, or "pre-scanned", to identify selected patterns representing high-energy pockets.

Once a pattern is identified, each N.times.N dictionary structure is centered at each location in the S.times.S search window, and the inner product between the structure and the corresponding N.times.N region of image data is computed. Computation time is reduced by the fact that dictionary structures are separable. The largest inner product, along with the corresponding dictionary structure and image location, form the five parameters of FIG. 8:

As previously indicated, these five parameters define an atom, a coded structure within the motion residual signal image. When an atom is found, it is ##EQU8## subtracted from the image to form the next stage residual. This corresponds to Equation 4 for the 1-D case. This residual is passed to the next coding stage, and the search process for the next high energy input pattern is repeated. The original motion error image is thus approximated as a summation of individual atoms.

One of the main benefits of the pattern marcher 60 of the invention is that, unlike a DCT coder, it is not block based. Still, some block artifacts from the motion model can be found in the reconstructed sequence. In an attempt to eliminate these effects, a smoother motion model using overlapping blocks is preferably used. For instance, a 16.times.16 sine-squared window may be used. ##EQU9##

This window decays smoothly at the edges, providing a more continuous motion estimate than a rectangular motion window. A rectangular window may be used to compute the motion vectors in a standard manner, but the smoother 16.times.16 window is employed to predict the current frame from these vectors and a previous frame reconstruction. A 16.times.16 block of data from the previous reconstruction is weighted by the sine-squared window and added to other such blocks on an overlapping 8.times.8 pixel grid. This produces a much smoother motion prediction without adding significant computation to the coding procedure. It is possible to improve the prediction even further by using the overlapping window to compute the motion vectors. The foregoing description assumes an 8.times.8 pixel based motion grid.

The present invention was implemented by replacing the DCT residual coder in an SIM3 system with the pattern matcher 60. As known in the art, the SIM3 low bit rate coding model was developed by the European COST211ter project. It is a hybrid DCT coding system designed to run at very low bit rates. The system uses the QCIF format in which each frame consists of a 176.times.144 pixel luminance image and two 88.times.72 pixel chrominance images. Frames are partitioned into macroblocks, each consisting of tour 8.times.8 luminance blocks and two 8.times.8 chrominance blocks. A macroblock thus represents a 16.times.16 pixel block of luminance, and a QCIF image contains a total of 99 macroblocks.

SIM3 uses a block translational motion model to predict the current frame from a previous reconstruction. A motion search at integer resolution is performed as a first estimate using 16.times.16 pixel motion blocks. The estimate is refined by a .+-.half-pixel search about the integer position, and a decision function is employed to determine whether the 16.times.16 motion block should be divided into four 8.times.8 blocks. Motion vectors are coded differentially using variable length codewords for each component.

The model includes a simple buffering system which drops frames adaptively to prevent buffer overflow. Each time a frame is coded, the number of bits used is added to the buffer. This generally puts the buffer above a threshold T and causes the system to enter a waiting state. At the onset of each input frame, a number of bits equal to the target bit rate in bits/s divided by the input frame rate in frames/s is subtracted from the buffer and the resulting buffer content is compared to the threshold T. If the buffer contains more than T bits, the current frame is not coded. If fewer than T bits are in the buffer, the frame is coded and bits are again added to the buffer. Between the quantizer adjustment and the buffering system, the target bit rate and frame rate are approximately met.

As previously indicated, the image format used in SIM3 is QCIF, consisting of 176.times.144 pixel luminance image and two 88.times.72 pixel chrominance images. Color atoms were coded by the disclosed search method. The previously described pre-screening procedure was performed on each of the three image components, and the energy totals for blocks from the chroma subimages were weighted by an adjustable constant C. The largest block energy total from the three subimages was then used as an estimate for the S.times.S exhaustive search. The atoms associated with each of the three subimages were grouped together and coded, as described above, and further described below.

The initial 16.times.16 pixel search at integer resolution was performed using chrominance pixels as well as luminance. Each subsampled chrominance pixel was weighted equally with each luminance pixel in the sum-of-absolute-difference metric used in the motion search. The modification tends to reduce color artifacts by using more of the available information to compute the motion vectors.

The smooth motion model was paired with the matching pursuit coding system. The controller computes a target bit budget F for each coded frame by dividing the target bit rate by the target frame rate. When frame number k is coded, the motion field is computed and coded first, requiring m.sub.k bits. This leaves approximately r=F-m.sub.k bits with which to code the motion residual image. The residual is coded using a total of M.sub.k atoms where M.sub.k, is computed as: ##EQU10## and a.sub.k-1 is the average number of bits used to code an atom in the previous frame. The actual residual budget, r.sub.k is the number of bits used to code the M.sub.k atoms. This value cannot be computed exactly until all M.sub.k atoms have been found.

When all atoms have been coded, both m.sub.k and r.sub.k are added to the buffer, which is identical to the SIM3 buffering system.

The separable nature of the dictionary structures can be used to reduce the inner product search time significantly. To illustrate this, consider the task of finding the largest inner product in an S.times.S search window on a dictionary composed of B 1-D basis elements, each of length N. Ignoring separability altogether, the task requires one to compute B.sup.2 full 2-D inner products at each of S.sup.2 locations in the search window. Since each inner product is over an N.times.N pixel region, the total number of multiply-accumulate operations required is S.sup.2 B.sup.2 N.sup.2. With the values S=16, B=15, and N=16, finding a single atom takes a total of 14.7 million operations.

The separable basis set can be exploited to reduce this search time. Consider the 2-D inner product between a basis structure G.sub..alpha..beta. (i,j) and an image patch I(i,j). Since G is separable, the inner product can be written as: ##EQU11## This is equivalent to performing N 1-D vertical inner products using g.sub..beta. (j) and performing a single 1-D horizontal inner product between the resulting vector and g.sub..alpha.. The atom search requires an exhaustive computation of the inner product at each location using all combinations of .alpha. and .beta.. It is thus natural to precompute the necessary vertical 1-D inner products with a particular g.sub..beta., and then cycle through the horizontal 1-D inner products using all possible g.sub..alpha.. Furthermore, the results from the 1-D vertical pre-filtering also apply to the inner products at adjoining locations in the search window. This motivates the use of a large "prefiltering matrix" to store all the 1-D vertical filtering results for a particular g.sub..alpha.. The resulting search algorithm is shown in FIG. 14. The operation count for finding a single atom is thus BSN(S+N)+B.sup.2 S.sup.2 N. Using the same set of nominal values, 1.04 million operations are required to find each atom.

A few refinements can reduce search complexity. For example, 1-D basis elements with small scale values are essentially zero over much of their support. The search algorithm can be modified to compute inner products using only the significant "non-zero" region of each basis function. Parallelism can also be exploited. As an example, if B processing units are available, each can be assigned the computation of the vertical pre-filter matrix corresponding to one of the g.sub..beta. (j). This increases the speed of the search algorithm by a factor of B.

The following quantization and variable length coding techniques were used when implementing the invention. The positions (x,y) were specified with adaptive Huffman codes derived from the previous ten frames worth of position data. Since position data from previous frames is available at the decoder, no additional bits need be sent to describe the adaptation. One group of codewords specifies the horizontal displacement of the first atom on a position line. A second group of codes indicates the horizontal distance between atoms on the same position line, and also contains `end-of-line` codes which specify the vertical distance to the next active position line. The other parameters were coded using fixed Huffman codes based on statistics from training sequences. The basis structure was specified by horizontal and vertical components .alpha. and .beta. each of which is represented by an index equivalent to i in Table 1. Two code tables were used, one for the horizontal indices and one for the vertical. The continuous projection values p were quantized using a fixed linear quantizer. A variable length code specified the quantizer level, and a single bit was added for the sign.

Those skilled in the art will recognize a number of advantages associated with the disclosed technology. Initially, the system is not DCT based, and therefore does not suffer the shortcomings associated with DCT systems operating at low bit rates. In addition, by allowing functions to overlap arbitrarily, the artificial block segmentation used in DCT-based systems is avoided. Further, since functions can be selected to decay smoothly at the edges, basis edge artifacts are greatly reduced. As a result, the reconstructed image is more natural than comparable images generated by DCT-based systems.

The foregoing descriptions of specific embodiments of the present invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, obviously many modifications and variations are possible in view of the above teachings. For example, the Gabor basis set disclosed herein provides good results, but it is not necessarily optimal. The disclosed matching pursuit routine will function with any arbitrary set of 2-D basis functions. It is possible that a different set of bases will provide better results for specific classes of input sequences.

The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following Claims and their equivalents.