US 5,650,823 AGrant
Half Pel Motion Estimation Method for B Pictures
Issue Date:1997-07-22
•2 Claims
•5 Drawing Sheets
Abstract
Disclosed is a method of forming a bidirectionally coded picture, i.e., a B frame, from two reference pictures, e.g. I or P frame pictures. The method utilizes a single memory fetch of each reference picture, and interpolation of the estimated motion of each picture. This is accomplished by identifying a full pixel closest match from three full pixel boundary searches, calculating half pixel reference picture data therefrom, interpolating the half pixels to form bidirectionally coded pictures, and finding the closest match at the half pixel boundary. The intermediate results are stored in an 18.times.18.times.11 bit buffer holding a 7 bit partial sum for each pixel, said partial sum formed by adding the six most significant bits of corresponding I and P frame pixels. In this buffer four bits of each word in the buffer are the two Least Significant Bits of the corresponding I and P frame pixels.
Metadata
Assignee
- International Business Machines Corporation
Inventors
- Agnes Yee Ngai
- Ronald Steven Svec
Application Information
Application Number:US 4111001
Filing Date:1995-03-27
Priority Date:1995-03-27
Art Unit:265
Classifications
IPC:
H04N 732
Field of Search:
348415;394;402;409;416;714;715;716;717;718;719
Patent Drawings (5 sheets)
Description
Field of the Invention
This invention relates to video encoders, especially video encoders intended to encode and compress video signals, e.g., discrete cosine transform encoded video signals. The method, apparatus, and system of the invention are useful in encoding broadcast signals, cablecast signals, and digital network signals, as well as in high definition television, interactive television, multimedia, video on demand, video conferencing, and digital video recording.
Background of the Invention
The Moving Picture Experts' Group (MPEG) MPEG-2 Draft Standard is a compression/decompression standard for digital video applications. The standard describes an encoding result demonstrates a substantial bandwidth reduction by a subjective lossy compression followed by a lossless compression. The encoded, compressed digital video data is subsequently decompressed and decoded in an MPEG-2 Draft Standard compliant decoder.
The MPEG-2 Draft Standard is described in, e.g., C. A. Gonzales and E. Viscito, "Motion Video Adaptive Quantization In The Transform Domain," IEEE Trans Circuits Syst Video Technol, Volume 1, No. 4, December 1991, pp. 374-378, E. Viscito and C. A. Gonzales, "Encoding of Motion Video Sequences for the MPEG Environment Using Arithmetic Coding," SPIE, Vol. 1360, pp. 1572-1576, (1990), D. LeGall, "MPEG: A Video Compression Standard for Multimedia Applications," Communications of the ACM, Vol. 34, No. 4, (April 1991), pp. 46-58, S. Purcell and D. Galbi, "C Cube MPEG Video Processor," SPIE, v. 1659, (1992) pp. 24-29, and D. J. LeGall, "MPEG Video Compression Algorithm," Signal Process Image Commun, v. 4, n. 2, (1992), pp. 129-140, among others.
The MPEG-2 Draft Standard specifies a very high compression technique that achieves compression not achievable with intraframe coding alone, while preserving the random access advantages of pure intraframe coding. The combination of frequency domain intraframe encoding and interpolative/predictive interframe encoding of the MPEG-2 Draft Standard result in a balance between intraframe encoding alone and interframe encoding alone.
The MPEG-2 Draft Standard exploits temporal redundancy for motion compensated interpolative and predictive encoding. That is, the assumption is made that "locally" the current picture can be modelled as a translation of the picture at a previous and/or future time. "Locally" means that the amplitude and direction of the displacement are not the same everywhere in the picture.
MPEG-2 Draft Standard specifies predictive and interpolative interframe encoding and frequency domain intraframe encoding. It has block based motion compensation for the reduction of temporal redundancy, and Discrete Cosine Transform based compression for the reduction of spatial redundancy. Under MPEG-2 Draft Standard motion compensation is achieved by predictive coding, interpolative coding, and Variable Length Coded motion vectors. The information relative to motion is based on 16.times.16 blocks and is transmitted with the spatial information. It is compressed with Variable Length Codes, such as Huffman codes.
The MPEG-2 Draft Standard provides temporal redundancy reduction through the use of various predictive and interpolative tools. This is illustrated in FIG. 1. FIG. 1 shows three types of frames or pictures, "I" Intrapictures, "P" Predicted Pictures, and "B" Bidirectional Interpolated Pictures.
The "I" Intrapictures provide moderate compression, and are access points for random access, e.g., in the case of video tapes or CD ROMS. As a matter of convenience, one "I" Intrapicture is provided approximately every half second. The "I" Intrapicture only gets information from itself. It does not receive information from an "P" Predicted Picture or "B" Bidirectional Interpolated Picture. Scene cuts preferably occur at "I" Intrapictures.
"P" Predicted Pictures are coded with respect to a previous picture. "P" Predicted Pictures are used as the reference for future pictures, both "P" and "B" pictures.
"B" Bidirectional Coded pictures have the highest degree of compression. They require both a past picture and a future picture for reconstruction. "B" bidirectional pictures are never used as a reference.
Motion compensation goes to the redundancy between pictures. The formation of "P" Predicted Pictures from "I" Intrapictures and of "B" Bidirectional Coded Pictures from a pair of past and future pictures is a key feature of the MPEG-2 Draft Standard technique.
The motion compensation unit under the MPEG-2 Draft Standard is the Macroblock unit. The MPEG-2 Draft Standard Macroblocks are 16.times.16 pixel macroblocks. Motion information consists of one vector for forward predicted macroblocks, one vector for backward predicted macroblocks, and two vectors for bidirectionally predicted macroblocks. The motion information associated with each 16.times.16 macroblock is coded differentially with respect to the motion information present in the reference macroblock. In this way a 16.times.16 macroblock of pixels is predicted by a translation of a 16.times.16 macroblock of pixels from a past or future picture.
The difference between the source pixels and the predicted pixels is included in the corresponding bit stream. The decoder adds the correction term to the block of predicted pixels to produce the reconstructed block.
As described above and illustrated in FIG. 1, each 16.times.16 pixel block of a "P" Predicted Picture can be coded with respect to the closest previous "I" Intrapicture, or with respect to the closest previous "P" Predicted Picture.
Further, as described above and illustrated in FIG. 1, each 16.times.16 pixel block of a "B" Bidirectional Picture can be coded by forward prediction from the closest past "I" or "P" Picture, by backward prediction from the closest future "I" or "P" Picture, or bidirectionally, using both the closest past "I" or "P" picture and the closest "future "I" or "P" picture. Full bidirectional prediction is the least noisy prediction.
Motion information is sent with each 16.times.16 pixel block to show what part of the reference picture is to be used as a predictor.
As noted above, motion vectors are coded differentially with respect to motion vectors of the previous adjacent block. Variable Length Coding is used to code the differential motion vector so that only a small number of bits are needed to code the motion vector in the common case, where the motion vector for a block is nearly equal to the motion vector for a preceding block.
Spatial redundancy is the redundancy within a picture. Because of the block based nature of the motion compensation process, described above, it was desirable for the MPEG-2 Draft Standard to use a block based method of reducing spatial redundancy. The method of choice is the Discrete Cosine Transformation, and Discrete Cosine Transform coding of the picture. Discrete Cosine Transform coding is combined with weighted scalar quantization and run length coding to achieve still further levels of compression.
The Discrete Cosine Transformation is an orthogonal transformation. Orthogonal transformations, because they have a frequency domain interpretation, are filter bank oriented. The Discrete Cosine Transformation is also localized. That is, the encoding process samples on an 8.times.8 spatial window which is sufficient to compute 64 transform coefficients or sub-bands.
Another advantage of the Discrete Cosine Transformation is that fast encoding and decoding algorithms are available. Additionally, the sub-band decomposition of the Discrete Cosine Transformation is sufficiently well behaved to allow effective use of psychovisual criteria.
After transformation, many of the frequency coefficients are zero, especially the coefficients for high spatial frequencies. These coefficients are organized into a zig-zag, as shown in FIG. 2, and converted into run-amplitude (run-level) pairs. Each pair indicates the number of zero coefficients and the amplitude of the non-zero coefficient. This is coded in a Variable Length Code.
Discrete Cosine Transformation encoding is carried out in the three stages as shown in FIG. 2. The first stage is the computation of the Discrete Cosine Transformation coefficients. The second step is the quantization of the coefficients. The third step is the conversion of the quantized transform coefficients into {run-amplitude} pairs after reorganization of the data into zig-zag scanning order.
Quantization enables very high degrees of compression, and a high output bit rate, and retains high picture quality.
Quantization can be adaptive, with "I" Intrapictures having fine quantization to avoid "blocking." This is important because "I" Intrapictures contain energy at all frequencies. By way of contrast, "P" and "B" pictures contain predominantly high frequency energy and can be coded at a coarser quantization.
The MPEG-2 Draft Standard specifies a layered structure of syntax and bit stream. The bit stream is separated into logically distinct entities to prevent ambiguities and facilitate decoding. The six layers are shown in Table 1, below
Objects of the Invention
It is a primary object of the invention to provide a reduced requirement for memory bandwidth in bidirectionally coding pictures.
It is a further object of the invention to reduce the memory required for bidirectionally coded pictures.
Summary of the Invention
These and other objects are accomplished by a method of forming a bidirectionally coded picture, i.e., a B frame, from two reference pictures, e.g. I or P frame pictures. The method utilizes a single memory fetch of each reference picture, and interpolation of the estimated motion of each picture. This is accomplished by identifying a full pixel closest match from three full pixel boundary searches, calculating half pixel reference picture data therefrom, interpolating the half pixels to form bidirectionally coded pictures, and finding the closest match at the half pixel boundary. The intermediate results are stored in an 18.times.18.times.11 bit buffer holding a 7 bit partial sum for each pixel, said partial sum formed by adding the six most significant bits of corresponding I and P frame pixels. In this buffer four bits of each word in the buffer are the two Least Significant Bits of the corresponding I and P frame pixels.
Brief Description of the Figures
The invention may be understood by reference to the FIGURES appended hereto.
FIG. 1 shows the "I"-- Intrapicture Frame, the "P" Predicted Picture frames, and the "B" Bidirectional Frame in a Group of Pictures under the MPEG-2 Standard and is denominated Prior Art and is denominated Prior Art.
FIG. 2 shows the stages of carrying out the Discrete Cosine Transform spatial compression of the MPEG-2 Standard and is denominated Prior Art.
FIG. 3 shows a circuit for calculating horizontal or vertical pixels in B Bidirectionally Interpolated pictures according to the partial sum method of the invention.
FIG. 4 shows a circuit for calculating full half pixels in B Bidirectionally Interpolated pictures according to the partial sum method of the invention.
FIG. 5 shows a flow chart of the method of the invention.
Detailed Description of the Invention
As defined in the MPEG standard, video images can be compressed as one of three picture types: I, P or B. An I picture is compressed by removal of spatial redundancy within the picture itself. A P picture is compressed by removal of temporal redundancy with reference to a previously encoded (compressed) picture. A B picture is also compressed by removal of temporal redundancy, but with reference to two previously encoded pictures. The B picture can be compressed by interpolation of both reference pictures. This enables B pictures to achieve the highest compression among the three picture types.
Bidirectional interpolation in a B picture is defined as follows:
let x be a pixel from reference picture I and
let y be a pixel from reference picture P.
The bidirectionally interpolated reference pixel is
where / is division with rounding.
A pixel is defined as an 8 bit wide positive integer in the range of 0 to 255. Thus, rounding implies that if the most significant bit of the residue is 1 then a 1 is added to the least significant bit of the quotient. Only the quotient is kept as the result of division and the residue is discarded. This is easily implemented in hardware as a right shift followed by an increment.
In motion picture encoding, the temporal redundancy must be identified so that it can be removed. This is accomplished by a process called motion estimation. Comparison circuitry is used to find a closest match for the current picture within a search window. Thus one would require three motion estimations to find a closest match for a B picture: one motion estimation with each of the two reference pictures and one motion estimation with the interpolated reference.
Per the MPEG standard, motion estimation is performed on a macroblock. The video image is divided into units of 16.times.16 pixels called a macroblock. For this reason the size of the closest match macroblock must also be 16.times.16 pixels. An 18.times.18 pixel area is required to form all possible half pixels around the identified closest match (16.times.16) macroblock. The types of half pixels used in motion estimation are explained later.
Motion estimation of a B picture involves memory fetches. A picture (720.times.480 pixels) requires 346K bytes of luminance data and is typically stored in memory external to the ASIC.
One method, utilized in the prior art, is to fetch one reference picture from external memory and to perform motion estimation with it. Then the second reference picture is fetched and motion estimation is performed in the same manner as the first reference picture. Then the closest match reference data (18.times.18 pixel block) from the first and second reference picture are fetched again and the motion estimation is performed on the interpolated picture.
The method of our invention is to fetch each reference picture only once as shown in the flow chart of FIG. 5. The closest match reference data from each reference picture is saved in an on-chip buffer. Subsequently, an interpolated motion estimation is performed using this buffered data. This method reduces the memory bandwidth requirement over the method of the prior art.
Once a closest match is identified from these three full pel boundary searches, half pixel reference data must be calculated and the motion estimation performed again to find the closest match reference data at the half pel boundary. There are three types of half pel interpolation used to calculate the half pixel reference data: horizontal half, vertical half and full half.
Let a, b, c and d be the four adjacent pixels in a reference picture as follows:
a b
c d
The horizontal half pixels are formed by:
where / is division with rounding. Vertical half pixels are formed by:
where / is division with rounding. This is illustrated in FIG. 4. Full half pixels are formed by:
where / is division with rounding.
Half pixels must first be formed from the buffered data for each reference picture. Then the interpolated half pixels for B pictures can be formed from these results to complete the motion estimation. Because of rounding, the order of operation must be maintained to correctly produce the interpolated result. In the prior art, this requires 2 sets of 18.times.18.times.8 bit buffers for a B picture, one for each reference picture.
According to the method the invention described herein the buffer requirement can be reduced to a single 18.times.18.times.11 bit buffer, as shown in the flow chart of FIG. 5. This buffer holds a 7-bit partial sum for each pixel, formed by adding the 6 most significant bits of corresponding I and P picture pixels. The other 4 bits of each word in the buffer consist of the 2 least significant bits of the corresponding I and P pixels.
As mentioned before, half pel motion estimation of a B picture requires half pel interpolation on each reference picture followed by interpolating those half pel results across the two reference pictures.
The following pictures are used with respect to the I frame. Let the following denote the pixels in the reference I picture:
where x denotes the bit position of each pixel and is an integer between 1 and 8.
The equation for horizontal half pel is
Considering only the two least significant bits of the pixel pair yields the following: ##EQU1##
Note that I.sub.S0x (8) is the `round` term for this half pel calculation since we perform division by 2. With this in mind the horizontal half pel equation for the I picture can be modified to: ##EQU2##
With similar deduction, the vertical half pel equation for the I picture can be written as: ##EQU3## where I.sub.Cx0 (6), I.sub.Sx0 (7), and I.sub.Sx0 (8) are obtained as follows: ##EQU4##
Again, note that I.sub.Sx0 (8) is the `round` term for this calculation since the division is by 2.
With similar deduction, the full half pel equation can be written as: ##EQU5## where I.sub.C (5), I.sub.C (6), and I.sub.S (7) are obtained from the following equation: ##EQU6##
Note that for this calculation, the `round` term is I.sub.S (7) since we divide by 4 and that I.sub.S (8) is discarded.
Half pixel calculations for the P picture can be handled in the same manner. Let the following denote the pixels in the reference P picture:
where x denotes the bit position of each pixel and is an integer between 1 and 8.
With the same method as in the I picture, the equation for horizontal half interpolation in the P picture could be written as follows: ##EQU7## where P.sub.C0x (6), P.sub.S0x (7), and P.sub.S0x (8) are obtained from: ##EQU8##
The equation for vertical half interpolation in the P picture can be written as follows: ##EQU9## where P.sub.Cx0 (6), P.sub.Sx0 (7), and P.sub.Sx0 (8) are obtained from: ##EQU10##
And finally, the equation for full half interpolation in the P picture could be written as follows: ##EQU11## where P.sub.C (5) P.sub.C (6) and P.sub.S (7) are obtained from the following equation: ##EQU12##
Bidirectionally interpolated horizontal half pixels in the B picture are formed by the interpolation of the horizontal half pixels from the I and P reference pictures. As such, the equations for I.sub.HH and P.sub.HH can be combined with the equation for bidirectional interpolation. This yields the following equation for an interpolated horizontal half pixel: ##EQU13## where IP.sub.C0x (4), IP.sub.C0x (5), IP.sub.C0x (6), and IP.sub.S0x (7) are formed as follows: ##EQU14## and I.sub.C0x (6), I.sub.S0x (7), I.sub.S0x (8), P.sub.C0x (6), P.sub.S0x (7), and P.sub.S0x (8) are as previously defined in the horizontal half pel equations for I and P pictures respectively.
The HH equation can then be rearranged as follows: ##EQU15## where IP.sub.00 n denotes the partial sum formed from I.sub.00 (n) and P.sub.00 (n) where IP.sub.01 n denotes the partial sum formed from I.sub.01 (n) and P.sub.01 (n) with `n` denoting the bit position in the pixel byte and is in the range of 0 through 6 only.
Also note that:
IP.sub.00 (0) is the carry out of I.sub.00 (1:6)+P.sub.00 (1:6)
IP.sub.01 (0) is the carry out of I.sub.01 (1:6)+P.sub.01 (1:6)
Thus, the interpolated horizontal half pixel results can be calculated with partial sums (IP.sub.00 (n) and IP.sub.01 (n)) from the I and P reference pictures in combination with the two least significant bits of each pixel from the I and P reference pictures.
The same rearrangement can also be done on the VH and FH equations. Thus the VH becomes: ##EQU16## where IP.sub.00 (n) denotes the partial sum formed from I.sub.00 (n) and P.sub.00 (n) where IP.sub.01 (n) denotes the partial sum formed from I.sub.10 (n) and P.sub.01 (n) with `n` denoting the bit position in the pixel byte and in the range of 0 through 6 only.
Again, note that:
IP.sub.00 (0) is the carry out of I.sub.00 (1:6)+P.sub.00 (1:6)
IP.sub.01 (0) is the carry out of I.sub.10 (1:6)+P.sub.10 (1:6)
and IP.sub.Cx0 (4) IP.sub.Cx0 (5) IP.sub.Cx0 (6) and IP.sub.Sx0 (7) are formed as follows: ##EQU17## where I.sub.Cx0 (6), I.sub.Sx0 (7), I.sub.Sx0 (8), P.sub.Cx0 (6), P.sub.Sx0 (7), and P.sub.Sx0 (8) are as previously defined.
With the similar deduction the FH equation becomes: ##EQU18## where IP.sub.00 (n) denotes the partial sum formed from I.sub.00 (n) and P.sub.00 (n), where IP.sub.01 (n) denotes the partial sum formed from I.sub.01 (n) and P.sub.01 (n), where IP.sub.01 (n) denotes the partial sum formed from I.sub.10 (n) and P.sub.10 (n), where IP.sub.11 (n) denotes the partial sum formed from I.sub.11 (n) and P.sub.11 (n), and n denotes the bit position in the pixel byte and is in the range of 0 through 6 only.
IP.sub.00 (0) is the carry out of I.sub.00 (1:6)+P.sub.00 (1:6). IP.sub.01 (0) is the carry out of I.sub.01 (1:6)+P.sub.01 (1:6). IP.sub.01 (0) is the carry out of I.sub.10 (1:6)+P.sub.01 (1:6) and IP.sub.11 (0) is the carry out of I.sub.11 (1:6)+P.sub.11 (1:6).
IP.sub.C (3), IP.sub.C (4), IP.sub.C (5), and IP.sub.C (6) in the previous equation are formed by the following: ##EQU19## with I.sub.C (5), I.sub.C (6), I.sub.S (7), P.sub.C (5), P.sub.C (6), and P.sub.S (7) as previously defined.
Also note that for this case the least significant bit of each partial sum (IP.sub.00 (6), IP.sub.01 (6), IP.sub.10 (6), and IP.sub.11 (6)) must be accounted for in the above equation.
The above equations clearly show that the half pixels required for B picture motion estimation can be formed by interpolation of the pixels in the corresponding byte positions of the two reference pictures first and then calculating the half pixel values from those interpolated pixels.
The above equations clearly show that the half pixels required for B picture motion estimation can be formed from a `reduced` set of data rather than the two full 18.times.18 pixel blocks normally required. This reduced set of data consists of an 18.times.18.times.11 array of 7-bit partial sums formed from the 6 high-order bits of corresponding pixels in the I and P pictures, along with the 2 least significant bits of each of those pixels. Using this method, the on chip buffer space can be reduced from 5184 bits (2.times.18.times.18.times.8 ) to 3564 bits (18.times.18.times.11), for a 31% improvement in required buffer area.
Circuitry useful for carrying out the method of the invention is shown in FIG. 3. The circuitry 11 includes an instruction store 21, an instruction fetch and decode unit 23, sending control signals 29 to an ALU controller 30, a branch processor 25, an arithmetic logic unit (ALU) 31, comprised of one byte arithmetic logic units (ALU's) 33, 35, 37, and 39, a two level general purpose register 41, a special purpose register 43, a condition register 45, a register/memory interface unit 47, a data bus 48, and a bus 49 to external registers and memory.
While the invention has been described with respect to certain preferred embodiments and exemplifications, it is not intended to limit the scope of the invention thereby, but solely by the claims appended hereto.
Claims
We claim:
1. A method of forming a bidirectionally coded picture from two reference pictures with a single memory fetch of each reference picture and interpolation of the estimated motion of each picture comprising the steps of: a. identifying a full pixel closest match from three full pixel boundary searches; b. calculating half pixel reference picture data from said full pixel closest match; c. storing the half pixel reference picture data in an 18.times.18.times.11 bit buffer holding a 7 bit partial sum for each pixel, said partial sum formed by adding the six most significant bits of corresponding I and P frame pixels; d. interpolating the half pixels to form bidirectionally coded pictures; and e. finding the closest match at the half pixel boundary.
2. The method of claim 1 wherein four bits of each word in the buffer are the two Least Significant Bits of the corresponding I and P frame pixels.
Patent Citations (13)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4958226 | 1990-09-01 | Haskell et al. | |
| US5278647 | 1994-01-01 | Hingorani et al. | |
| US5311305 | 1994-05-01 | Mahadevan et al. | |
| US5311307 | 1994-05-01 | Yoshimoto | |
| US5321508 | 1994-06-01 | Veltman et al. | |
| US5351095 | 1994-09-01 | Kerdranvat | |
| US5379351 | 1995-01-01 | Fandrianto et al. | |
| US5386234 | 1995-01-01 | Veltman et al. | |
| US5414469 | 1995-05-01 | Gonzales et al. | |
| US5526054 | 1996-06-01 | Greenfield et al. | |
| US5528315 | 1996-06-01 | Sugiyama | |
| US5539466 | 1996-07-01 | Igarashi et al. | |
| US5539467 | 1996-07-01 | Song et al. |