Picture Element Processor for a Memory Management System

This invention concerns a digital video processor. Specifically, the invention concerns a system for encoding and decoding image representative picture elements (pixels or pels) in a block based image processor.

Memory management and memory reduction are important aspects in the design and operation of image processors. For example, consumer products such as television systems may use image processors including MPEG-2 signal processing. The MPEG (Motion Picture Experts Group) signal compression standard (ISO/IEC 13181-2, May 10, 1994) is a widely accepted image processing standard which is particularly attractive for use with satellite, cable and terrestrial broadcast systems employing high definition television (HDTV) processing among other forms of image processing. Products using high definition displays require 96 Mbits or more of memory to temporarily store decoded MPEG frames prior to display. An MPEG processor requires these frames for motion estimation and compensation to reconstruct accurate images for display.

Systems which reconstruct images from decoded MPEG information typically employ Differential Pulse Coded Modulation (DPCM). In DPCM processing as typically used in an MPEG decoder, a processor generates a prediction which anticipates the next pixel value. A summation network subtracts the prediction from the actual pixel value resulting in a difference. This difference, known as prediction error, is generally smaller than the original pixel or prediction values, so quantizing and storing the difference rather than the original pixel value saves memory. Ang et al., "Video Compression Makes Big Gains," IEEE Spectrum, October 1991, describes an MPEG encoder and decoder.

During decoding, a dequantizer regenerates substantially the same prediction from previously decoded pixels. Only a difference value and the prediction are needed to decode and reconstruct the current pixel. The prediction is often partially or wholly based on the previous pixel, which itself was decoded and reconstructed from the pixel preceding it. For a more complete description of such predictors and their operation, see Jain, A., Fundamentals of Digital Image Processing, Prentice-Hall, Inc., p. 484 (1989); and Gonzalez et al., Digital Image Processing, Addison-Wesley Publishing Company, Inc., pp. 358-368 (1992).

Accurately representing the first pixel in a block during image block compression avoids propagating prediction error throughout the entire block of pixels. In smooth areas, (e.g. a display area which has subtle changes in color or objects or the like), noise contamination of the first pixel may produce artifacts which a viewer may find unacceptable. Therefore, the first pixel processed should be represented by a sufficient number of bits when stored in memory to ensure accurate image reconstruction.

The present inventors recognize the desirability of providing a data reduction system with reduced hardware and software requirements which will save memory without introducing error into reconstructed data. A system according to the present invention satisfies these objectives.

In accordance with the principles of the present invention, a "first pixel" processor compresses a predetermined pixel of a pixel block as a function of a pixel block parameter.

In a disclosed embodiment of the invention a television receiver includes an MPEG decoder. An MPEG decoded signal is provided to a data reduction network which quantizes data prior to storage in memory. The network employs a so-called first pixel processor which compresses (quantizes) a reference pixel of a pixel block as a function of the pixel block minimum value. After decompression, the reconstructed value of the reference pixel is substantially equal to its value prior to encoding by the first pixel processor. Thus, the reference pixel may be used in a predictor network to reconstruct other quantized pixels in the image data block without concern for propagating significant error throughout a reconstructed pixel block.

FIG. 1 is a block diagram of a pixel block processor including apparatus according to the present invention.

FIG. 2 shows details of an encoder portion of the FIG. 1 system preceding storage in memory.

FIG. 3 is a flow chart of a reference pixel encoding process.

FIG. 4 depicts a decoder portion of the FIG. 1 system after storage in memory.

FIG. 5 is a flow chart of a reference pixel decoding process.

FIG. 6 is a block diagram of an MPEG compatible television system employing the present invention.

In FIG. 1, an MPEG decoder provides a block of MPEG decoded pixel data to a compressor 12. Compressor 12 includes a predictor 18, a quantizer 20, and a combiner 22. Predictor 18 may be of the type described in Jain, A., Fundamentals of Digital Image Processing, Prentice-Hall, Inc., p. 484 (1989), for example.

Quantizer 20 quantizes the pixel block values and provides a data reduced pixel block to memory 14. When the display processor (not shown) accesses the reduced data block in memory 14 for image display, decompressor 16 reconstructs the original pixel block. Decompressor 16 includes predictor 24 and dequantizer 26 to retrieve reduced data from memory 14 and to reconstruct the reduced data block. Quantizer 20 and dequantizer 26 are arranged according to the principles of the present invention as will be discussed. Predictor 24 is of the same type as predictor 18.

Input 10 of compressor 12 receives a pixel block from an MPEG decoder, which will be discussed in connection with FIG. 6. The pixel block is in the spatial domain, and comprises an 8.times.8 block of image pixels, for example. Input 10 supplies the pixel block data to a non-inverting input of combiner 22 and to quantizer 20. The output of predictor 18 supplies pixel prediction data to an inverting input of combiner 22 and to quantizer 20. Combiner 22 combines signals from its inverting and non-inverting inputs and provides the difference to quantizer 20. Quantizer 20 outputs quantized pixel values to predictor 18. Quantizer 20 also outputs quantized pixel difference values for storage in memory 14.

FIG. 2 illustrates quantizer 20 in greater detail. The same reference numbers identify elements common to FIGS. 1 and 2. Specifically, quantizer 20 includes first pixel encoder 30, multiplexer 32 and quantizer processor 28. Input 10 provides the block pixel data to first pixel encoder 30, which operates upon a predetermined reference pixel of the pixel block. The term "first pixel" does not specifically refer to any particular preferred pixel position. "First pixel" refers to that pixel within a block of pixels which will be used initially in a predictor for reconstructing the pixel block. Therefore, the pixel becomes a reference pixel for all subsequently reconstructed pixels of a compression network employing a predictor.

First pixel encoder 30 has two primary functions. The most important function is to quantize the reference pixel. The second function is to reduce the storage requirements of frame memory 14 by reducing the number of bits required to represent the quantized value of the reference pixel. These functions are discussed later. If quantization of the reference pixel is lossless or near lossless, the pixel can be retrieved from memory 14, reconstructed with little or no error and used as a prediction value for the remaining pixels in the quantized block. Using an accurate prediction value initially in the prediction process prevents a prediction bias error from propagating throughout the entire reconstructed pixel block.

Reducing the number of bits required to store the quantized reference pixel reduces the size and cost of memory 14. Encoder 30 reduces the number of bits required to represent the quantized reference pixel by one to five bits in this example. Although this amount may seem insignificant, when considered in the context of a high definition television image frame which may contain more than 32,600 8.times.8 pixel blocks, every bit saved results in a significant savings in memory. These savings become more significant when multiplied by the number of television receivers produced by a manufacturer.

A potential conflict exists between accurately representing the reference pixel value and representing all pixels including the reference pixel with as few bits as possible. An obvious savings occurs if memory is reduced by fifty percent, which translates into eight-bit pixel words being compressed to four bits. Using four bits to represent the reference pixel may induce significant error in the course of prediction processing. Since an accurate reference pixel representation is important to beginning the prediction process, using more bits, when necessary, than the average number of bits used to represent the remaining pixels is worthwhile. Therefore memory reduction is secondary to accuracy relative to the reference pixel.

For the remaining pixels in a block, memory reduction is more important than the accuracy of any one pixel. Encoder 30 may represent each pixel processed as a word containing from three to seven bits. It is impracticable to attempt to determine the amount of frame memory that will actually be used to store image frames for any given broadcast signal because of the randomness involved. Therefore memory would have to accommodate the maximum word length (seven bits) for each pixel processed by encoder 30 if it processes all pixels.

First pixel encoder 30 receives the minimum block pixel value from quantizer processor 28. Quantizer 28 receives pixel difference data from combiner 22 and predictor 18, and sends quantized data to predictor 18. Both quantizer 28 and encoder 30 send quantized data to multiplexer 32. Multiplexer 32 sends sequenced quantized data to memory 14 (FIG. 1) as will be discussed subsequently.

FIG. 3 is a flow chart of the first pixel encoding process performed by encoder 30. In step 31, encoder 30 processes the original first pixel value (X.sub.0) from input 10, and the quantized minimum pixel block value (Q.sub.min) from unit 28 to produce a quantized first pixel value (QX.sub.0). Step 31 operates according to the following equation:

Pixels values received at input 10 (FIG. 1) from an MPEG decoder are integers. Therefore, quantizer processor 20 and first pixel encoder 30 receive and output data representative of integers, including QX.sub.0. The advantage of using integers is faster data transfer between elements, faster processing within processors, and less complicated hardware and software. Since X.sub.0 began as an integer when received by first pixel encoder 30 from the MPEG decoded signal, data loss will occur only if X.sub.0 -Q.sub.min is an odd number. An odd number divided by two always has a remainder of 0.5. If the system stores only the integer portion, then the remainder is lost and is unavailable for reconstructing an image.

First pixel encoder 30 divides by two because a divisor of two immediately decreases by one the number of pixels required to represent the integer portion of the result. Using a larger devisor may save more bits but induces greater error. It has been observed that dividing by two results in a displayed image which is substantially unaffected by the above compression technique.

Encoding the first pixel reduces the eight-bit original pixel by one to five bits in step 33. Determining how many bits to use to represent the encoded first pixel depends upon the value of the quantized range, QR. Quantizer 28 (FIG. 2) calculates QR using the expression:

where Q.sub.max represents the quantized maximum value in the pixel block. The quantized range QR is chosen as an indicator for setting the number of bits allocated to represent the value of the encoded first pixel because the number of bits needed to represent the value of the encoded first pixel is less than the number of bits needed to represent the value of the quantized range, as is now explained.

Quantizer processor 28 produces a quantized minimum value which is less than the original pixel block minimum value, and a quantized maximum value which is greater than the original pixel block maximum value. Therefore, the following relationship exists:

where X.sub.0 represents any pixel value within the pixel block currently being processed. Equation (3) may be restated as follows by incorporating equation (2):

Subtracting Q.sub.min from all three parts of equation (4) produces the following equation:

Since encoded first pixel QX.sub.0 is one half of X.sub.0 -Q.sub.min (equation (1)), QX.sub.0 requires one less bit to storage it in memory 14 (in a binary system) than the quantized range QR requires. Therefore, quantizer 28 allocates one less bit than is needed to represent QR when determining the word size required to represent and store the encoded first pixel value.

Continuing with FIG. 3, first pixel encoder 30 initially constructs QX.sub.0 as an eight-bit word in step 31. In step 33 encoder 30 masks QX.sub.0 by the appropriate number of bits for the current range and transfers the desired bits to multiplexer 32. The masked bits of QX.sub.0 from step 33 are sent to multiplexer 32 which holds QX.sub.0 until it is transferred to memory 14. Control and timing for this operation are discussed in association with FIG. 6.

FIG. 4 shows the arrangement of first pixel decoder 38, demultiplexer 34 and dequantizer processor 36 within dequantizer 26. Demultiplexer 34 conveys the data from memory 14 to first pixel decoder 38 and to dequantizer processor 36. First pixel decoder 38 receives the encoded first pixel value from demultiplexer 34, and receives the minimum block pixel value from dequantizer processor 36. First pixel decoder 38 decodes the first pixel to its quantized value as supplied to input 10 (FIG. 1) from the MPEG decoded signal. The reconstructed first pixel is conveyed to utilization circuits on a data bus along with the other reconstructed pixels of the pixel block from dequantizer 36.

Dequantizer 36 receives prediction data from predictor 24 (FIG. 1), and the other quantized pixel values from demultiplexer 34. Dequantizer 36 reconstructs the remaining pixel block value to match the original bits per pixel size provided to input 10 (FIG. 1) and outputs the reconstructed values. Timing control from a local microcontroller causes the decoded reference pixel from first pixel decoder 38 and reconstructed pixel data from dequantizer 36 to appear on a data bus at the appropriate time as needed by a display processor. This will be seen in FIG. 6.

Within any DPCM prediction network, the accuracy of the first data point used by the predictor, such as predictor 24, is critical to the accuracy of all data which is subsequently produced by the prediction network for a block of data. Each data point within the block is built upon preceding data points by the prediction network. The dequantized data represents the difference between the preceding data point and the current data point. During reconstruction, this difference is added to the preceding reconstructed data point to attain the current data point. Therefore, any error introduced in the first data point will propagate as a prediction error for subsequent data points within the same block.

FIG. 5 is a flow chart of the first pixel decoding process. In step 35, decoder 38 performs a decoding operation in accordance with the expression :

where RX.sub.0 represents the decoded first pixel value. QX.sub.0 and Q.sub.min are the same as in equation (1).

Equation (6) represents the inverse of the operation represented by equation (1). Equation (1) is a divide by two operation and equation (6) is a multiply by two operation. Since first pixel decoder 38 multiplies the encoded first pixel by two during decoding, the maximum error in the reconstructed first pixel value is numerically one less than the original value of the first pixel. This occurs only if X.sub.0 -Q.sub.min is an odd number, because first pixel encoder 30 keeps only the integer portion of the pixel value during its encoding process. If X.sub.0 -Q.sub.min is an even number, the decoded first pixel value equals the original first pixel value.

After first pixel decoder 38 calculates the decoded first pixel value RX.sub.0 in step 35, RX.sub.0 is padded with leading zeros to attain the pixel word size required by the display processor in step 37. Generally, this is eight bits. In step 37, first pixel decoder 38 adds zeros to the left of the most significant bit of RX.sub.0 until it is an eight bit word. All bits added during this operation are zero (0) so as not to introduce error to the value. Reconstructed first pixel value RX.sub.0 is output to the display processor.

Because compressor 12 and decompressor 16 perform inverse operations system architecture and construction are simplified. Also, encoder 30 in step 31 and decoder 38 in step 35 employ inverse operations which may be implemented by simple bit shifting using known bit shifting techniques.

Referring again to FIG. 1, memory 14 stores the encoded first pixel value until it is no longer needed for pixel reconstruction and display. During the time the encoded first pixel resides in memory 14, it may be accessed and decoded by a subsequent display processor via decompressor 16. Compressor 12 and decompressor 16 reside as a common integrated circuit. Memory 14 advantageously resides outside the integrated circuit thereby allowing the size of memory 14 to be selected as needed to accommodate the signal processing requirements of a particular system. This results in manufacturing cost savings, e.g., in the case of a reduced cost consumer television receiver using a reduced resolution display requiring less frame memory for the MPEG decoder.

FIG. 6 illustrates portions of a practical digital signal processing system in a television receiver including apparatus according to the present invention as previously discussed. The digital television system of FIG. 6 is simplified so as not to burden the drawing with excessive detail. For example, not shown are FIFO input and output buffers associated with various elements, read/write controls, clock generator networks, and control signals for interfacing to external memories which can be of the extended data out type (EDO) or synchronous type (SDRAM).

Common elements in FIG. 1 and FIG. 6 have the same identifier. Elements in signal processor 72, except for compensator 70, correspond to elements found in the STi 3500A MPEG-2/CCIR 600 Video Decoder integrated circuit commercially available from SGS-Thomson Microelectronics. Motion compensation unit 70 may employ the STi 3220 Motion Estimator Processor integrated circuit available from the same manufacturer.

Briefly, the system of FIG. 6 includes microprocessor 40, bus interface unit 42 and controller 44 coupled to an internal control bus 46. In this example, microprocessor 40 is located external to the integrated circuit containing MPEG decoder 72. A 192 bit wide internal memory bus 48 is a conduit for data to and from compressor 12, similar decompressors 16 and 50, and external frame memory 14. Units 12, 16 and 50 receive compression and decompression factor control signals from microprocessor 40 via controller 44, along with enable control signals. Microprocessor 40 also partitions memory 14 into frame storage, buffer, and on screen display bit map sections for MPEG decoding and display processing. Also included is a local memory control unit 52 which receives Request inputs and provides Acknowledge outputs as well as memory Address output, Read Enable and Write Enable outputs. Memory control unit 52 generates real time address and control signals for controlling memory 14. Memory control unit 52 also provides output clock signals Clock In and Clock Out in response to input clock signals Clock In from a local clock generator (not shown). Microprocessor 40 partitions memory 14 into bit buffers, video frame store sections and frame storage buffers for MPEG decoding, and display processing and on-screen display maps.

Display processor 54 includes horizontal and vertical resampling filters as needed to convert a decompressed image format to a predetermined common format for display by an image reproducing display device 56. For example, the system may receive and decode image sequences corresponding to formats such as 525 line interlaced, 1125 line interlaced or 720 line progressive scan. A television receiver will likely use a common display format for all receiver formats.

External interface networks 58 convey control and configuration information between the MPEG decoder and external microprocessor 40 in addition to input compressed video data for processing by the MPEG decoder. The MPEG decoder system resembles a coprocessor-processor for microprocessor 40, e.g. microprocessor 40 issues a decode command to the MPEG decoder for each frame to be decoded. The decoder locates the associated header information, which in turn is read by microprocessor 40. With this information microprocessor 40 issues data for configuring the decoder, e.g. with respect to frame type, quantization matrices, etc., after which the decoder issues appropriate decode commands. Technical specification material for the SGS-Thomson STi 3500A and 3220 integrated circuit devices noted above provides additional information concerning such manner of MPEG decoder operation.

Microprocessor 40 conveys mode control data, programmed by the receiver manufacturer, to memory controller 52 for controlling the operation of multiplexer 32 (FIG. 2) and demultiplexer 34 (FIG. 4), and for establishing the compression/decompression factors for units 12, 16, and 50 as required. The disclosed system can be used with all Profiles and all Levels of the MPEG specification in the context of various digital data processing schemes, such as may be associated with terrestrial broadcast, cable and satellite transmission systems, for example.

FIG. 6 also depicts a portion of a digital video signal processor 72 such as may be found in a television receiver for processing an input high definition video signal. Signal processor 72 includes a conventional MPEG decoder constituted by blocks 60, 62, 64, 66, 68 and 70 with frame memory 14. Ang et al. "Video Compression Makes Big Gains," IEEE Spectrum, October 1991, describes an example of the operation of an MPEG encoder and decoder.

Signal processor 72 receives a controlled data stream of MPEG coded data from a preceding input processor (not shown), e.g., a transport decoder which separates data packets after input signal demodulation. In this example, the received input data stream represents high definition image material (1920.times.1088 pixels) as specified in the Grand Alliance specification for the United States high definition terrestrial television broadcast system. The input data stream is in the form of data blocks representing 8.times.8 pixels. The data blocks represent compressed, coded intraframe and interframe information. The intraframe information comprises I-frame anchor frames. Generally, the interframe information comprises predictive motion coded residual information representing the image difference between adjacent picture frames. The interframe motion coding involves generating motion vectors representing the offset between a current block being processed and a block in a prior reconstructed image. The motion vector which represents the best match between the current and prior blocks is coded and transmitted. Also, the difference (residual) between each motion compensated 8.times.8 block and the prior reconstructed block is Discrete Cosine Transformed (DCT), quantized and variable length coded (VLC) before being transmitted. Various publications, including Ang, et al. above, describe motion compensated coding processes in greater detail.

Buffer 60 accepts the input compressed pixel data blocks before being variable length decoded by VLD 62. Buffer 60 exhibits a storage capacity of 1.75 Mbits in the case of a main level, main profile MPEG data stream. Inverse quantizer 64 and DCT 66 decompress decoded compressed data from VLD 62. Output data from DCT 66 is coupled to one input of adder 68.

A signal from buffer 60 controls the quantization step size of inverse quantizer 64 to assure smooth data flow. VLD 62 provides decoded motion vectors to motion compensation unit 70 as will be discussed. VLD 62 also produces an inter/intra frame mode select control signal as known (not shown for simplification). The operation performed by units 62, 64, and 66 are the inverse of corresponding operations of an encoder located at a transmitter.

By summing the residual image data from unit 66 with predicted image data provided from the output of unit 70, adder 68 provides a reconstructed pixel based on the contents of video frame memory 14. When signal processor 72 has processed an entire frame of pixel blocks, frame memory 14 stores the resulting reconstructed image. In the interframe mode, motion vectors obtained from VLD 62 provide the location of the predicted blocks from unit 70.

The image reconstruction process involving adder 68, memory 14 and motion compensation unit 70 advantageously exhibits significantly reduced memory requirements due to the use of block compressor 12 prior to data storage in frame memory 14. The size of frame memory 14 may be reduced by up to fifty percent (50%), for example, when a 50% compression factor is used. Unit 50 performs the inverse function of unit 12, and is similar to decompressor 16 described above. Compressor 12 and decompressors 16 and 50 are constructed in accordance with the principles of the present invention as previously discussed and shown in FIGS. 1, 2, 4 and 5.