US 5,675,382 AGrant
Spatial Compression and Decompression for Video
Issue Date:1997-10-07
•8 Claims
•12 Drawing Sheets
Abstract
A spatial compression and decompression system for color video data wherein a video stream is generated by a charge-coupled device with red, green and blue pixel elements. In a first compression stage, the 5-bit color data from each element is processed in turn. This step involves reading out from a lookup memory a 3-bit category (or compressed value) and 5-bit output value based on the 5-bit color data, a 5-bit previous value that represents same color data for a neighboring pixel and a 1-bit table select signal that determines in which of two similar tables the lookup is performed. The 3-bit category data for a pixel's three elements are combined and used as a 9-bit index into a Huffman table that performs the second stage of compression. Each Huffman table entry includes a variable-length Huffman code that represents more common categories with fewer bits, which provides additional compression over the first stage. The Huffman table reads out the code of the particular 9-bit category used for the lookup. This code is packed into an output buffer and transmitted to the decompressor that will unpack for display or store the resulting compressed data stream. In the first decompression stage, each Huffman code in the compressed data stream is decoded using no more than two compares and one table lookup into an inverse Huffman decode table. The resulting 9-bit category is used as an index into one of two similar inverse lookup tables, which outputs the appropriate 16-bit uncompressed data value.
Metadata
Assignee
- Connectix Corporation
Inventor
- Roger A. Bauchspies
Application Information
Application Number:US 6304311
Filing Date:1996-04-08
Priority Date:1996-04-08
Art Unit:265
Classifications
IPC:
H04N 1419
Field of Search:
348358382384;387;390;393;394;397;405261.1;427;523;524166;232;246;281
Patent Drawings (12 sheets)
Description
The present invention relates generally to video compression systems, and particularly to systems that perform spatial video compression (i.e., compression within a single video frame) for color video data.
BACKGROUND OF THE INVENTION
It takes an enormous amount of bandwidth to process, store or transmit high-resolution, full-color digital video data at acceptable frame rates. For example, assuming a video source with a resolution of 320 by 240 pixels (corresponding to VGA resolution) and 16-bit pixel data, one frame of color video data takes up 1,228.8 Kbits (320.times.240.times.16). Assuming the video source provides 12 frames per second at this resolution, systems that display or store the video data must have a bandwidth of at least 14.746 Mbits per second. Common transmission systems (telephone lines and 28.8 Kbit per second modems) and storage media (hard disk drives with typical real-world storage rates of no more than 10 Mbits per second) do not have this capability. Consequently, there is a need for systems that compress high-resolution video data so that it can be stored and transmitted more easily.
There are two classes of video compression systems. Systems that perform temporal compression shrink the amount of video data by detecting similarities between corresponding pixels in subsequent video frames and encoding the redundant information so it takes up less space on transmission or storage. In contrast, systems that perform spatial compression reduce the amount of data needed to represent a single frame of video by detecting regions within the frame with similar pixel data and compressing the video data corresponding to those regions. The present invention does not perform temporal compression; therefore we will only address prior art spatial compression systems.
Prior art spatial compression systems take a variety of approaches to generating compressed video data. However, before performing any data compression, the vast majority of spatial compression systems first quantify the similarity between each pixel and its neighboring pixels (the neighboring pixels can be one or more pixels). When the video data is monochrome, this first step involves a simple comparison of the grey levels of the neighboring pixels. When the video being compressed is color, where each pixel is made up of three elements responsive to at least three complementary colors (e.g., red, green and blue), the prior art evaluates the similarities between individual element data rather than between pixels as a whole. These prior art systems then separately compress the element data based on the similarities between the individual elements.
For example, a class of video compression systems that perform run-length encoding (RLE) map the similarity data for each pixel element into an encoded value whose length is a function of the similarity data (i.e., similar elements are assigned encoded values with small lengths, resulting in data compression). Another class of compression systems use discrete cosine transforms (DCTs) to represent regions of elements with small frequencies (i.e., regions where there is little change across elements) with small values that generally have fewer bits than the same regions in the uncompressed image.
One problem with these prior art systems is that they tend to produce compressed data streams that, when decoded and displayed, result in images that have pronounced color banding. This is because the prior art spatial compression systems force groups of neighboring pixels to the same color, even if there is a color gradient across those pixels. This is partly because the prior art systems use only one function or table to encode each element's similarity data. Therefore, there is also a need for a color video compression system that does not produce color banding in images displayed from decompressed data.
Another problem with the prior compression systems (especially the RLE systems) is that for random images (i.e., images with little or no correlation between neighboring pixel elements), they are likely to produce a "compressed" video stream that is actually longer than the original video stream. This is the case because the lack of similarity between random neighboring elements causes the RLE systems to generate uniformly long codes for all of the elements. This can even occur in RLE systems for real-world data. Therefore, there is also a need for a video compression system that is guaranteed to produce compressed data streams that are no bigger than the initial video stream for all possible images.
Finally, even if the prior art systems were able to provide the aforementioned features, they would still be too slow to compress and then decode highly detailed video data at acceptable frame rates. This is because the encoded data provided by the prior art compression systems is structured so that it can only be decoded bit-by-bit or in time-consuming pattern matching (in the RLE case). For example, one prior art method of compressing data, called Huffman coding, relies on a statistical analysis of the similarity data to produce encoded (compressed) data where each encoded datum has a length that is related to the likelihood of occurrence of its associated similarity data. That is, if a certain piece of similarity data occurs frequently in a set of images, it would be associated with a Huffman code of relatively few bits. This Huffman coding process generates a compact compressed video stream that preserves much of the detail in the original images (even though it too can be longer than the original data stream for random images). However, systems that employ Huffman coding decode the stream of Huffman codes bit by bit, using a Huffman decode tree that represents the space of possible Huffman codes and their associated decoded values. Thus when a particular Huffman code happens to be many bits long (e.g, 16-bits), decoding that code using a tree approach would take equally many (e.g., 16) steps.
Therefore, there is also a need for a spatial compression and decompression system for color video that provides the aforementioned advantages and is also fast enough to provide the necessary decompression bandwidth.
SUMMARY OF THE INVENTION
The present invention is an improved system for compressing and decompressing color video data that meets the needs set out above. More particularly, the present invention is a video compression system for compressing a digital video stream of N-bit datums, each of which corresponds to one pixel in a video image with an associated column index (i) and row index (j) and including Q M-bit subcomponents representing respective source values of Q elements of the one pixel. The elements of the present invention include two lookup tables that are used to perform the first stage of data compression, an address circuit, an output circuit and a state machine that controls the other elements.
The first lookup table includes at least 2.sup.2M entries, each of which stores a P-bit first category value and a M-bit first output value and is uniquely associated with a M-bit previous value and a M-bit source value. This table is configured to read out a first particular entry's first category and output values upon receiving a readout address assembled from the first particular entry's associated previous and source values and a first table select signal. The first category read out of the first lookup table represents a compressed version of the source value used in the lookup. Like the first table, the second lookup table has at least 2.sup.2M entries, each of which stores a P-bit second category value and a M-bit second output value and is uniquely associated with a M-bit previous value and a M-bit source value. As with the first table, the second lookup table is configured to read out a second particular entry's second category and output values upon receiving as a readout address the second particular entry's associated previous and source values and a second table select signal. Note that the rows of the second lookup table are a reflection of corresponding rows of the first lookup table about the columns of those corresponding first table rows for which the first category is 0. The second category read out from the second table represents a compressed version of the associated source value used for the lookup.
The address circuit is coupled to the lookup tables and computes the previous value that is input to those tables as part of the lookup address. The address circuit computes the previous value as a function of the output values of selected, previously-processed pixel elements according to a first formula when the first table select signal is available and according to a second formula when the second table select signal is available. The output circuit is also coupled to the lookup tables and is configured to receive and store the P-bit category values for all of the Q elements of the one pixel currently being processed. This output circuit is also configured to output a single Q.times.P-bit combined category that encapsulates the Q category values for the current pixel. Given that Q.times.P is smaller than N, which is the size of one pixel's worth of video data, the combined category is guaranteed to be a compressed version of the original pixel.
These elements are controlled by the first state machine, which, among other things, provides the first table select signal and the second table select signal so that the first and second lookup tables and the first and second formulas are used to readout the category and output values for alternating pixels. This use of different lookup tables and formulas for alternating pixels reduces color banding in video images that are reconstructed from the combined category information. The first state machine also triggers the output circuit to store and output the Q.times.P-bit combined category for the one pixel being currently processed.
The video compression system of the present invention can also include a Huffman encode table having at least 2.sup.Q.times.P entries, which performs the second stage of data compression. Each of the Huffman encode table entries includes a Huffman data field for storing a variable-length Huffman value for one of the combined categories and a Huffman bits field for storing a Huffman bit count representing the total bits in the associated Huffman value. The Huffman encode table, which is coupled to the output circuit, reads out the Huffman value and Huffman bit count associated with the entry whose address matches the combined category output by the output circuit. The Huffman values are allocated to the combined categories so that the combined categories that are more likely to occur in a video frame are allocated to Huffman values having fewer bits. By reducing the size of the most likely combined categories in this fashion, the Huffman table is able to provide additional data compression of data above that provided by the first lookup table. Additionally, the Huffman encode table is structured so that it never outputs a Huffman value that is longer than the N-bits in a video stream datum, which ensures that the compressed video stream is no larger, even in the worst case of random images, than the original video stream.
The compressed data stream generated by the first and second compression stages is decompressed in two stages. The first stage of decompression is performed by a Huffman decode table that receives and decodes the compressed data stream in X-bit chunks, where X represents the number of bits in the longest possible Huffman value. The compressed data stream includes plural compressed datums, each of which corresponds to one of the Huffman values read out from the Huffman table in the second compression stage. Each chunk is structured so that it includes at least one of these Huffman values aligned with the most significant bit position of the chunk.
The Huffman decode table has at least 2.sup.X entries, each including combined category and Huffman bits data associated with a Huffman value that is mapped to the entry's X-bit address. The Huffman decode table is configured to read out this combined category and Huffman bits data in whatever entry is indexed by the fixed-length chunk input to the Huffman decode table. The Huffman decode table is structured so that a set of Huffman decode table entries with X-bit addresses whose most significant Z bits match a Z-bit Huffman value includes the same combined category data and Huffman bits data. This same data is identical to the combined category and Huffman bits data associated with the Z-bit Huffman value in the Huffman table that performs the first stage of compression. Thus, the same data is read out from any one of the entries when the most significant Z bits of the fixed length chunk match the Z bits, thereby allowing Huffman data to be decoded in a single lookup.
The present invention also includes a final decompression stage wherein two inverse lookup tables are used to convert individually the Q, P-bit categories output by the Huffman decode table into a corresponding output value based on a previous pixel value. The two inverse lookup tables correspond to the two lookup tables used in the first stage of compression and are respectively used to decompress category data for alternating pixels.
BRIEF DESCRIPTION OF THE DRAWINGS
Additional objects and features of the invention will be more readily apparent from the following detailed description and appended claims when taken in conjunction with the drawings, in which:
FIG. 1 is a block diagram of the video compression-decompression system of the present invention;
FIG. 2 is a block diagram of the video-compression-decompression system of the present invention as implemented in a digital video camera;
FIG. 3 depicts the structure of a typical video image and a corresponding video stream that is input to the video compression hardware;
FIG. 4A is a functional block diagram of the video compression system of the present invention;
FIG. 4B is a hardware block diagram of the video compression system of the present invention;
FIG. 5A illustrates the effects on the video stream of 50% sub-sampling;
FIG. 5B illustrates the effects on the video stream of 25% sub-sampling;
FIG. 6 is a block diagram illustrating the structure of the two lookup tables 210A, 210B stored in the lookup table memory 200 of FIG. 3;
FIG. 7 is a block diagram illustrating the structure of the last value storage element 112 and one-row storage element 114 of FIG. 3 and their cooperation with the averager 124;
FIG. 8 is a block diagram illustrating the structure and cooperation of the 9-bit output register 280 and the Huffman table 300 of FIG. 3;
FIG. 8B illustrates a Huffman decode tree used in the prior art to decode variable-length Huffman codes.
FIG. 9 is a diagram showing the structure of a preferred system for generating the Huffman data stored in the Huffman table of FIG. 7.
FIG. 10 is a flow diagram of the video data compression method implemented by the present invention.
FIG. 11A is a functional block diagram of video decompression system of the present invention; and
FIG. 11B is a block diagram illustrating the structure of the decompression system of the present invention.
DESCRIPTION OF THE PREFERRED EMBODIMENT
Referring to FIG. 1, there is shown a high-level block diagram that illustrates the environment in which the present invention operates. A conventional rasterizing video source 102 (such as a TV or video camera) generates a video data stream 103 that is input to a video compressor 104. The compressor 104 compresses the data in the data stream 103 (i.e., reduces the number of bits required to represent each frame of the original video image) and generates a corresponding compressed data stream 105 that can be transmitted or stored much more efficiently than the uncompressed video stream 103. For example, the compressed data stream 105 can be written to a storage device 108 for later use. The compressed stream 105 can also be transmitted to a decompressor 106 that reverses (to some extent) the compression process and generates a decompressed data stream 107 that approximates the original data stream 103. A display can make use of the decompressed data stream 107 to display an approximation of the original image scanned by the raster source 102.
Referring to FIG. 2, there is shown a block diagram that illustrates a preferred embodiment of the present invention in which the raster source 102 is a charge-coupled device (CCD) and in which the video compressor 104 is combined with the CCD 102 in a video camera 80 and the optional storage 108, decompressor 106 and display are combined in a separate computer 90 coupled to the camera. In this embodiment, the CCD 102 scans an image (image) and outputs a corresponding 16-bit digital video stream 103 to the compressor 104. The operations of the CCD 102 and the structure of the 16-bit video stream 103 it generates are now described in greater detail in reference to FIG. 3.
As shown in FIG. 3, the CCD 102 scans an image using a grid of m by n pixels (short for picture cells), each of which has an associated index (i,j). For the purposes of this application, pixel(0,0) is in the top left corner of the CCD 102 and pixel(m,n) is in the lower right corner. Each of the pixels is made up of multiple elements, each sensing a different part of the visual spectrum. In the preferred embodiment, each pixel of the CCD 102 includes three pixel elements sensitive, respectively, to red (R), green (G) and blue (B) light. However, the present invention could function equally well with other types of CCDs, such as a CCD 102 having multiple pixel elements sensitive, respectively, to cyan (C), yellow (Y), and magenta (M) wavelengths; or a monochrome CCD.
The CCD 102 represents the signal detected at each pixel using a 16-bit video stream datum 103i. Five bits of each datum 103i are used to encode the signal strength from a particular pixel element (R, G or B). The final bit (b15) is reserved for other purposes and is not used in the present invention. The CCD 102 rasterizes the scanned image by generating a video stream 103 wherein the individual 16-bit video datums 103i appear in a fixed order. For example, in the preferred embodiment, the CCD 102 reads out its scanned data row by row starting from pixel(0,0), corresponding to a datum 103-0, and ending with pixel(m,n), corresponding to a datum 103-(m)(n). The CCD 102 repeatedly scans and reads out image data at a fixed frame rate (FR). However, because the present invention is concerned only with spatial video compression (i.e., intra-frame data compression), this application will not focus on frame to frame processing. Also, depending on the processing capacity of the end user, the resolution (i.e., n and m) of the CCD 102 can be varied.
Referring again to FIG. 2, from the video stream 103, the compressor 104 generates a compressed data stream 105 wherein each datum 105i comprises an indeterminate number of bits and corresponds to one of the 16-bit datums 103i from the video stream 103. Even for the worst case scenario (a image in which there is no correlation between adjacent datums 103i), the compressor 104 is guaranteed to generate a compressed data stream 105 that uses no more bits to represent a given video frame than the original video image, which is not the case in prior art video compression systems.
The compressed data stream 105 generated by the compressor 104 is output by the camera 80 to the computer 90, where it can be processed by the decompressor 106 and/or stored in the video storage unit 108. From the compressed data stream 105, the decompressor 106 generates a 16-bit decompressed video stream 107 that approximates the original video stream 103. Because of the structure of the individual datums 105i, which is a key feature of the present invention, the decompressor 106 is able to generate the decompressed video stream 107 with speed and efficiency that are not possible with prior art decompression techniques. Based on the decompressed video stream 107, the display can display a redisplayed image (image') that resembles the original image (image). How well the redisplayed image (image') resembles the original image (image) is inversely related to the level of compression performed by the compressor 104. I.e., high compression levels result in degraded image quality and vice-versa. However, high compression levels are also related to increased efficiencies in video stream transmission and storage. The compressor 104 of the present invention balances these opposing factors by providing the highest possible compression at the greatest possible speeds while retaining acceptable redisplayed image quality. For example, the compressor 104 of the present invention can achieve 50 percent compression levels with absolutely no loss of video quality, either perceived (meaning what the user sees) or actual (meaning the content of the decompressed video stream 107 as compared to the original video stream 103). The compressor 104 can also achieve even higher levels of compression (e.g., up to five to one) with little loss in apparent quality to the viewer of the reconstructed image (image'). How this is accomplished in the present invention is now described in reference to FIG. 4A.
Referring to FIG. 4A, there is shown a functional block diagram of the compressor 104, which performs video compression in two stages. The first compression stage (S1) compresses the 16-bit data stream 103 from the CCD 102 into a stage 1-compressed stream 281 in which the datums 281i (each corresponding to one 16-bit datum 103i) are only 9-bits in length. Thus, the first stage S1 compresses the input data stream 103 by a ratio of 16 to 9 (16:9). In the second stage (S2), the 9-bit datums 281i are converted to Huffman datums 301i with a variable length that is inversely related to the likelihood of their occurrence. I.e., 9-bit datums 281i whose bit pattern occurs frequently are represented by S2 as Huffman datums 301i with a small number of bits (e.g, as few as 1 or 2 bits), and datums 281i whose content occurs infrequently are represented by S2 as Huffman datums 301i using a larger number of bits (e.g, as many as 13 to 16 bits). This means that some of the 9-bit datums 218i are actually expanded in the second compression stage (S2). However, because the most likely 9-bit patterns in the datums 281i are associated with Huffman datums 301i of very few bits, for typical images (where there is high degree of correlation between adjacent pixels) high levels of compression are achieved by stage 2 compression (S2). Moreover, because a Huffman datum 301i can have no more than 16 bits, it is guaranteed that the stage 2-compressed stream 301 generated by the second compression stage S2 will never be larger than the input data stream 103 for a given frame of video data. These two stages of video compression are performed under the control of a compression state machine (hereinafter referred to as SM1) 250 using multiple control signals 251.
The Huffman datums 301i are output to a set of output control registers 380, 390, which collectively pack the datums 301i into 8-bit chunks 105i that form the compressed output data stream 105. Under control of the output state machine (SM2) 350, these 8-bit chunks 105i are output to the decompressor 106 as needed. Because of the variability in length of the Huffman datums 301i, each 8-bit chunk could include multiple datums 301i (e.g., 4 datums 301, each 2 bits in length) or fewer than one datum (e.g., when the compressed datum 301i is longer than 8 bits). These functional blocks are now described in greater detail in reference to FIG. 4B.
Referring to FIG. 4B, there is shown a hardware block diagram of the compressor 104. The elements of FIG. 4B that carry out the stage 1 compression described in reference to FIG. 4A include a sub-sampler 116, one-row storage 114, last value storage 112, multiplexers 118, 120, 122, an averager 124, a lookup table memory 200 and a 9-bit register 280. The operation of these elements is controlled by the compression state machine SM1 250 using multiple control signals 251. These signals include the following:
External inputs supplied to the first stage elements include the video stream 103 to be compressed, which is coupled to the sub-sampler 116, and a color video control signal 257 from the CCD 102 controller (not shown) that indicates to the SM1 250 when a new video stream datum 103i is available from the CCD 102. Outputs supplied by the first stage elements include the 9-bit datums 281i from the 9-bit register 280. The operation of these first stage compression elements is now briefly described. A fuller description is provided below, in reference to FIGS. 5-8.
The sub-sampler 116 samples the 5-bit red, green and blue data for a particular video stream datum 103i (representing one pixel) and outputs the sampled data 117 to the multiplexer 122 as three selectable color groups 117R (red data), 117G (green data), 117B (blue data). The multiplexer 122 selects one of the groups 117R, 117G or 117B and outputs the 5-bits of selected data to the lookup table memory 200 as a 5-bit source value 123. This 5-bit source value 123 is combined with a 5-bit previous value 125 output by the averager 124 and a 1-bit table select signal 251F output by the SM1 to form an 11-bit address that is coupled to the address lines of the lookup table memory 200.
The lookup table memory 200 includes two lookup tables 210A and 210B, each storing 2.sup.10, 8-bit entries, each of which includes a 3-bit output category value 203 and a corresponding 5-bit output value 201. Upon receiving a new 11-bit address, the lookup table memory 200 selects the table 210A, or 210B designated by the 1-bit table select signal 251F and reads out from the selected table 210A or 210B the 5-bit output value 201 and 3-bit category value 203 stored in the entry addressed by the combined 5-bit source and previous values 123, 125.
The 3-bit category value 230 read out by the lookup memory 200 is output to the 9-bit register 280, which stores the 3-bit category 203 in a three bit segment designated by the red, green or blue register select signals 251G generated by the SM1 250.
The 5-bit output value 201 read out by the lookup memory 200 is output to the last value register 112 and the one-row storage 114. The last value register 112 is a 15-bit register with three 5-bit segments for storing red (R), green (G) and blue (B) data for a single pixel P(i,j). The last value register 112 stores the 5-bit output value 201 in the R, G or B segment designated by the SM1 in the 3-bit read/write strobe 251C. The last value register 112 reads out all 15 bits whenever the SM1 sets the read/write strobe 251 to all zeros.
The one-row storage 114 is a 15-bit memory with m entries (m=320 in the preferred embodiment), each of which stores the 5-bit red, green and blue output values 201 for the most recently processed row of pixels. For example, assuming that the latest sampled datum 117i corresponds to a pixel P(i,j), the row of pixels stored by the one-row storage 114 comprises all of the sampled datums 117i representing pixels between the pixel P(i,j-1) and the pixel P(i-1, j). The one-row storage 114 stores the 5-bit output value 201 at the location designated by the 9-bit address 251B output by the SM1. This 9-bit address 251B corresponds to the column index (i) of the latest sampled Pixel P(i,j). The 2-bit read/write strobe 251C asserted by the SM1 determines in which segment (R, G or B) of the addressed location the 5-bit output value 201 is stored. As with the last value storage register 112, the one row storage 114 outputs all 15-bits of the entry at the 9-bit address 251B when the SM1 sets the read/write strobe 251B to zero. The read is stored before the write so the old data is used before it is overwritten by the new 201.
The 15-bit value 115 read out by the one-row storage 114 provides the red, green and blue data for the previously scanned pixel that lies directly above (or as close to directly above as possible) the pixel whose video data 103i is being processed. For example, assuming every video datum 103i is sampled by the sub-sampler 116 and the pixel being processed is the pixel P(i,j), the data 115 output from the last value storage would be from the pixel P(i, j-1)). The 15-bit last value 113 read out by the last value register 112 corresponds to the red, green and blue data for the pixel processed immediately before the current pixel. For example, assuming the same scenario as above, the last value 113 would be from the pixel P(i-1, j) (i.e., the pixel to the immediate left of the pixel being processed). One color is selected from these 15-bit values by the MUXes 118, 120 in response to the color select signal 251D provided by the SM1. The resulting 5-bit values 119, 121 output by the MUXes are averaged (i.e., added and then divided by two) by the averager 124, which outputs the result as the 5-bit previous value 125. The averager 124 is also coupled to the carry-in signal 251E provided by the SM1 250. When the SM1 250 asserts the carry-in signal, instead of computing the simple average just described, the averager 124 adds 1 to the sum of the values 119 and 121 before dividing by two.
The Huffman table 300, under control of the SM1 250, carries out the stage 2 compression described in reference to FIG. 3A. The Huffman table 300 is a memory with 2.sup.9, 20-bit entries, each of which provides 4-bits for a Huffman bit count 303 and 16-bits for a Huffman value 305. The 4-bit count 303 indicates how many of the other 16-bits are actually used for the Huffman value 305. The address lines of the Huffman table 300 are coupled to the 9-bit category data 281, which provides the address of the 20-bit Huffman data to be read out by the Huffman table 300. The 20-bit Huffman data 303, 305 is read out in 8-bit chunks 301 to the holding register 380 in the order indicated by the 2-bit read control signal 251H, which is provided by the SM1 250. For example, the SM1 250 cycles the 2-bit read signal 251H through three different values that cause the Huffman table 300 to output the low eight bits of the Huffman data 305L, the high eight bits of the Huffman data 305H and finally the Huffman bit count 303. When the Huffman data 301 is about to be read out, the SM1 250 outputs a Huffman data ready signal 251I to the output state machine SM2 350 that tells the SM2 350 to initiate an output step.
The elements that perform the output step described in reference to FIG. 3A are the 20-bit holding register 380, which receives and stores the 8-bit Huffman data 301i, and the 8-bit output register 390, which packages the variable length Huffman values 305 for inclusion in the compressed video stream 105. The elements 380, 390 are controlled by the output state machine (SM2) 350 via the following control signals 351A, 351B:
External data control signals that flow between the SM2 350 and the microprocessor (not shown) that controls the camera 80 include a data.sub.-- ready.sub.-- signal 357 output by the SM2 and an acknowledge (ACK) signal 359 output by the microprocessor. When the output register 390 is full, the SM2 350 asserts the data.sub.-- ready.sub.-- signal 357. In response, after the data has been accepted, the microprocessor asserts the ACK signal 359, upon which the SM2 350 empties the output register 390.
The holding register 380 loads the 8-bit datum 301-i from the Huffman table into the register segment indicated in the 3-bits of the 3-bit holding register (HR) load select signal 351A. The HR load signals 351A are timed by the SM2 350 so the holding register 380 segments 380A, 380B, 380C are loaded in the same order that the 8-bit Huffman data chunks 301 are read out. I.e., the Huffman bit count 303 and the Huffman high and low segments 305H, 305L are respectively loaded into the holding register segments 380A, 380B and 380C. The SM2 350 uses the fourth bit of the HR load signal 351A to tell the holding register 380 to shift out one bit 181 of the Huffman value 305 stored in the segments 380B, 380C. The 8-bit output register 390 shifts-in the one bit 181 in response to the shift-in control signal 351B and outputs its contents as a compressed datum 105i.
Having generally described the functionality and interaction of the elements of FIG. 4B, additional details are now provided for certain of these elements, including the sub-sampler 116, the lookup tables 210A and 210B, the averager 124 and the Huffman table 300.
Sub-Sampler 116
The sub-sampler 116 samples the 16-bit video data stream 103 and outputs a 15-bit sampled datum 117i that includes the 15 low-order bits of a sampled video stream datum 103i. The 15-bits of the sampled datum 117i comprise three 5-bit color groups 117R, 117G and 117B that correspond, respectively, to the red, green and blue color data from the sampled datum 103i. The decimate signal 251A from the SM1 250 determines when the sub-sampler 116 samples the video stream 103. In the preferred embodiment, the SM1 250 uses this decimate signal 251A to achieve 100 percent, 50 percent and 25 percent sub-sampling where, respectively, every datum, every other datum, or every fourth datum 103i in the video stream are output by the sub-sampler 116 as a sampled datum 117i. A depiction of which image pixels from the CCD 102 are used by the compressor 104 in the 50 and 25 percent sub-sampling cases is shown in FIGS. 5A and 5B, respectively.
FIG. 5A shows the set of CCD 102 pixels (indicated by the symbol "##") whose data is represented in the sub-sampled video stream 117 for 50 percent sub-sampling. Similarly, FIG. 5A shows the set of CCD 102 pixels (again indicated by the symbol "##") whose data is represented in the sub-sampled video stream 117 for 25 percent sub-sampling.
The color group data 117Ri, 117Bi, 117Gi output by the sub-sampler 116 are coupled to the MUX 122 as three selectable input groups. Based on the state of a color selection signal 251D provided by the SM1, the MUX 122 outputs data from one of the color groups 117Ri, 117Bi, 117Gi as a 5-bit source value 123i to be compressed (note: in the preferred embodiment, the processing sequence for each pixel is green, red and blue). The value 123i is coupled to the lookup table memory 200 as 5 bits of an 11-bit lookup address. As mentioned above, the other components of the lookup address are a 5-bit previous value 125i-1 output by the averager 124 and a 1-bit lookup table select value 251E output by the SM1.
Lookup Memory 200
Based on the 1-bit lookup table select value 251E provided by the SM1 250, the lookup table memory 200 selects one of its lookup tables 210A or 210B and then performs a lookup in the selected table 210 on the lookup table entry 210p,s!, where "p" represents the 5-bit previous value 125 output by the averager 124 and "s" represents the 5-bit source value 123 from the sub-sampler 116 via the MUX 122. The structure of a lookup table 210 is now described in reference to FIG. 6.
Referring to FIG. 6, each entry 210p,s!in the lookup tables 210A, 210B includes a 3-bit category 203 and a 5-bit output value 201. The 3-bit category 203 for an entry 210p,s! captures the difference between the 5-bit source value (s) 123 and the 5-bit previous value (p) 125. Thus, for the set of entries 210p,s! where "p" and "s" are identical or similar (e.g., if "p" and "s" are 16; or if "p" is 6 and "s" is 7) the 3-bit category 203 stored in those entries would be 0. In contrast, for entries 210p,s! where the difference between "p" and "s" is large (e.g., if one is 0 and the other is 31), the 3-bit category 203 for those entries would be 7, or as large as possible. The 5-bit output value 201 represents a quantized version of the source value 123 that is used as a proxy for the actual source value 123 for all compression and decompression stages subsequent to the initial lookup. This quantized 5-bit value 201 is the same for all lookup table entries 210p,s! with the same previous value (p) 125 and category value 203.
For example, consider one row (for p=16) of a lookup table 210A used in the preferred embodiment:
Note that there are more common output values 201 for high-numbered categories 203 than for low-numbered categories. This is due to the fact that the source pixel and previous pixel values 123, 125 represent the values of the same color element for neighboring pixels on the CCD 102, Because viewers of the image ultimately displayed from the decompressed data 107 are less likely to notice small alterations between the values of neighboring pixel elements when the original differences between those pixel elements was large, the lookup tables 210 use finer resolution in categories 203 and output values 201 for neighboring pixel elements (e.g., source 123 and previous 125 values) that were originally very close in value. Thus, the above example of a lookup table row assigns seven different source values 123 (i.e., 0-6), which are as different from the previous value "16" as is possible, the same category "6" and output value "3". In contrast, there are only two source values (15, 16), which are as similar to the previous value "16" as possible. These source values 123 are assigned the category "0" and output value "16". This design of the lookup tables 210A, 210B is extremely important as it allows an image to be compressed in such a way that important details are not lost.
Note that the categories are assigned so that the 0 category is at the point in the row where the previous value 125 and the source value 123 are most similar, and the other categories are assigned in increasing order on alternating sides of the 0 category. Another goal is to assign odd numbers of source values to the higher categories so that the difference between the source value and its associated output category 201 is kept as small as possible within a given category. Generally, the largest difference between the source and output value will be no larger than the category modulo 2. For example, none of the source values assigned the category 5 (i.e., the source values 22, 23, 24, 25, 26) differ by more than plus or minus 2 from their associated output category 24. In fact, for categories other than categories 6 and 7, the output value (i.e., the displayed value) will differ from its corresponding source value by no more than plus or minus 2. For the smaller categories (e.g., 0 and 1) the difference between the source value and its output value is never greater than 1. This means that the redisplayed image is very close in color to the original image.
Both of the tables 210A and 210B are constructed according to these principles; however there is a subtle but important difference between the contents of the two lookup tables 210A and 210B. The rows of 210B have fundamentally the same values as the rows of the table 210A; but represent a reflection of the rows of the table 210A about the values of s for which the category is 0. For example, consider the following row for the previous row value of "16" that is taken from an actual second lookup table 210B that is paired with the table 210A whose comparable row is shown above:
As will be described later, the SM1 uses the two tables 210A, 210B for alternating lookups. That is, when performing a lookup on a source value for a pixel P(i,j), the SM1 might use the table 210A; but when performing a lookup for the next pixel P(i+1, j), the SM1 would use the other table 210B. Because of the subtle differences between the mirrored tables, even if the previous value 125 and source values 123 are identical for the two pixels, the two tables might generate different output categories and values 203, 201. For example, for "p" equaling 16 and "s" equaling 15, the first table would read out a category 203 and output value 201 of 0 and 16, whereas the second table 210B would read out a 1 and a 15, respectively. However, because the differences between the tables is subtle, some p and s pairs result in the same category and output values being read out of the tables 210A and 210B (e.g. in the exemplary set of tables, p=16 and s=16 always are associated with category 203=0 and output value=16).
As a result of this two-table lookup process, some identical, adjacent pixels are compressed to different respective 9-bit category values 281 in the first stage of compression, which means that those adjacent pixels would be re-displayed as different values following decompression. This seems like a flaw in the present invention; but is actually an advantage that prevents color banding of the decompressed video data 107 that is generated by other video compression systems. This color banding occurs in the prior art due to the fact that these systems force many slightly differing neighboring pixels to the same exact value. Because even small groups of pixels with the same color data are readily detectible, color banding in the decompressed images is perceived by viewers. This problem is avoided by the present invention's use of two lookup tables 210A, 210B for alternating lookups.
Complete lookup tables 210A, 210B are shown respectively in Appendices A and B. In these table listings, each row includes all table values for a single previous value 125, which in the tables is called "prevRGB". Other information in these tables include the source value 123 ("inRGB") and the output value 201 and output category 203 that correspond to each pair of previous value 125 and output value 123. Note that these tables are used to compress the red, green, and blue color data of a sampled datum 117. Also note that some of the rows of the table 210B are not a perfect reflection of the corresponding rows of table 210A. These deviations from perfect reflectivity were made to locally optimize the tables for a specific implementation of the preferred embodiment; however, the present invention does not require these local optimizations.
Averager 124
Referring to FIG. 7, there is shown a block diagram of the averager 124, the last value register 112 and the one-row storage 114. In the lower right of this figure there is also included a depiction of one section of the CCD 102 that shows the pixel being currently processed P(i,j) and the pixels P(i-1,j) and P(i, j-1) that are adjacent to the pixel P(i,j). These "adjacent" pixels are the pixels whose associated 5-bit RGB output values are averaged to produce the 5-bit previous value 125. Control signals 251 input to these elements include the R/VV strobe 251B, which is coupled to the last value register 112 and the one-row storage 114, the color select signal 251D, which is coupled to the MUXes 118, 120, and the carry-in signal 251E, which is tied to the averager 124. All of these control signals are generated by the SM1 at appropriate times during a stage 1 compression step.
Before sampling a new pixel P(i,j), the SM1 sets the 9-bit address 251B to i (i.e., the column index of the datum 103i being sampled) and outputs a read signal using the R/W strobe 251C. This causes the one-row storage 114 to output the RGB values 115 associated with the pixel P(i,j-1) (i.e, the pixel above the pixel P(i,j)) and causes the last value register 112 to output the RGB values 113 associated with the last sampled pixel P(i-1,j). The SM1 250 then sets the 2-bit color select signal 251D to cause the MUXes 118, 120 to select one color from their respective 15-bit inputs 113, 115. The 5-bit data 119, 121 output from the MUXes 118, 120 is input to the averager 124, which computes the average of the two values with or without a carry-in, depending on the state of the carry-in signal 241E provided by the SM1 250. For example, with a carry-in, the average of the two values 119 and 121 would be:
Without a carry-in, the average of the two values would be:
The SM1 250 alternates the state of the carry-in signal 241E for alternating sampled pixels. Thus, if the SM1 causes the averager 124 to use a carry-in to average the RGB data for the pixel P(i-1,j), it would not set the carry-in for the pixel P(i,j). This is another advantage of the present invention, which, along with the use of alternative lookup tables 210A, 210B, prevents the color banding that occurs in the prior art. In the preferred embodiment, the SM1 sets the carry-in 251E and the table select signal 251F so the carry-in is always used in conjunction with lookups into the second table 210B.
The embodiment shown in FIG. 7 presumes that the sub-sampler 116 has sampled every datum 103i in the video data stream 103. In other cases, e.g., where the sub-sampler samples only 50 percent or 25 percent of the video datums 103i, the adjacent pixel values 119 and 120 that are averaged do not represent the logical pixels directly above and to the left of the pixel whose data 103i was sampled (refer to FIGS. 5A and 5B). This fact is immaterial to the present invention, which averages whatever data 113 is stored in the last value register 112 (which represents the last sampled datum 117) and whatever data 115 is stored in the one-row storage with an address that is closest to the currently sampled pixel's column address (which represents the datum from the previous sampled row that is closest to being above the last sampled datum 117).
Because the previous value 125 computed by the averager 124 represents an element value for a pixel that neighbors the pixel providing the source value 123, the resulting 3-bit category 203 read out by the lookup table 210 represents the categorical difference between the same color elements of two neighboring pixels. For example, assuming that the source value 123 corresponds to the red data value of a sampled pixel P1 and the previous value 125 corresponds to the red data value computed for a neighboring pixel P2, the 3-bit category 203 output by the lookup table represents the categorical difference between P1's and P2's red data values. This categorization of the degree of difference between adjacent pixel elements is an important first step in video compression, where high levels of compression are achieved by representing the similarities between adjacent pixels with compressed video data.
Once the 15-bit data from the last value register 112 and the one-row storage 114 is read out, the SM1 250 sets the R/W strobe 251C so that the 5-bit output value read out from the lookup memory 200 overwrites the appropriate 5-bits of the one-row storage at the 9-bit address (i) 251B and also overwrites the appropriate 5-bits of the last value segment.
To avoid problems in computing the previous value 125 for pixel data 103i associated with pixels that lie in the first row or first column of the CCD 102, at the beginning of a frame, the SM1 loads each 5-bit segment of the last value register 112 and the one-row storage 114 with the default value of 16 (the mid-point of the 5-bit color values). Similarly, at the beginning of a line of data other than the first line of a particular image, the SM1 250 loads each of the 5-bit RGB segments of the last value register 112 with 16.
Huffman Table 300
Referring to FIG. 8, there is shown a block diagram illustrating the structure of a Huffman table 300 as implemented in the present invention and the 9-bit register 280 that provides the 9-bit category data 281 that is used to perform the Huffman table lookup.
This 9-bit category data 281 represents combined category data for all three elements from a single pixel P(i,j). Among other things, each Huffman table lookup produces a variable-length Huffman value 305 that characterizes the 9-bit category 281 used for the lookup. In the preferred embodiment, the longest Huffman value is 13-bits, although Huffman values of as many as 16-bits could be used. Thus, each Huffman table lookup simultaneously encodes the combined category data for three pixel elements. This simultaneous processing (i.e., compression) of all three elements of a pixel is unknown in prior art video compression systems (e.g., systems using run-length encoding (RLE) and discrete cosine transforms (DCT)), where each pixel element is separately encoded. These combined categories 281 provide additional levels of compression because, in typical images, all three pixel elements tend to change very little from pixel to pixel for large portions of the image.
In the Huffman table 300, each of the 2.sup.9 possible 9-bit categories 281 is associated with a Huffman code, or value 305 whose total number of bits is inversely related to the likelihood of the occurrence of that 9-bit category 281. For example, because repetition is very common in video data (i.e., it is common for large areas of contiguous pixels to have the same color data), the Huffman code 305 with the fewest bits is typically associated in the present invention with the 9-bit value that is represented in octal as 000 (indicating there was no or nearly no difference between neighboring pixels' red, green and blue values). Similarly, the Huffman code 305 with the most bits is typically associated with the 9-bit value that is represented in octal as 777 (indicating there was the largest possible difference between neighboring pixels' red, green and blue values).
Beyond the general rule that the Huffman codes for more frequent categories have fewer bits than the Huffman codes for the less frequent categories, the present invention generally imposes no strict rules on the ordering of Huffman values that have the same number of bits. The only exception to this rule applies to the situation where, for processing purposes, two Huffman decode tables 430 (rather than one) are used to decode the Huffman data stream 105. This situation is discussed in reference to FIG. 11B.
The present invention can distribute the possible Huffman values (e.g., in the preferred embodiment there are 2.sup.13 possible Huffman values) among the 9-bit categories in a variety of ways. However, some ways of allocating Huffman codes are less conservative than others. For example, the more frequent 9-bit categories could be assigned Huffman values in such a way that there are too few remaining Huffman codes for the less frequent 9-bit categories. In such a situation, the present invention assigns the remaining 9-bit categories respective Huffman codes whose length is the maximum Huffman code length (e.g., 13-bits), whose numerical value is smaller than any of the Huffman codes for the more frequent categories (after left-justifying all of the Huffman codes to the maximum Huffman code length), and whose least significant 9-bits comprise the 9-bit category 281. Thus, when such a code is encountered during the decompression step, it can immediately be decoded to the correct 9-bit category. In the preferred embodiment this feature is disabled as each 9-bit category 281 is assigned one of the available 2.sup.13 Huffman values.
Each Huffman table entry 300i also has a 4-bit field that includes a Huffman bit count 303i. The Huffman bit count indicates to the SM1 250 how many of the 16 bits that could be used to represent Huffman values are actually used for the Huffman value 305i. Based on this Huffman bit count 303i, which is stored in the counter register 380A as part of the output step, described above, the SM2 knows how many Huffman value bits to shift into the output register 390. In the preferred embodiment, the Huffman data is left-justified in the data registers 380H and 380L so that it can be easily shifted out by the SM1 250.
A full example of a Huffman table 430 used in an implementation of the preferred embodiment is shown in Appendix C. This example shows the Huffman values ("Huff") assigned to each of the 512 categories 281 ("array val") of the present invention and the number of bits ("bits") associated with those Huffman values. Note that all of the values shown in Appendix C are decimal values. Thus, the array values 1, 8 and 64 all correspond to category values where only one of the constituent 3-bit categories is 1. Consequently, these entries are associated with a small number of bits. The longest Huffman value in this table is 13 bits (associated with many values), while the shortest Huffman values is 2-bits (associated with the 9-bit category whose decimal value is 0).
Various methods of implementing Huffman tables for use in Huffman coding and decoding are well-known. These prior art methods typically use Huffman trees to decode the streams of variable length Huffman data output from the Huffman table. This is a slow process that limits the amount (i.e. resolution) of video data that can be processed in real time. The present invention completely eliminates the need for this slow, tree-based Huffman decoding by encoding as Huffman values 305 the 9-bit categories in such a way that only one or two tables lookups need be performed by the decoder 106 to decode the Huffman data 105.
For an illustration of the present invention's Huffman encoding technique, consider a situation where there are only four values Vi to be encoded (think of these values Vi as representing the 2.sup.9, 9-bit categories). Assume for the present example that the values V1, V2, V3 and V4 are most likely to occur in the order given (i.e., V1 is more likely to occur than V2 and so on). Thus, consistent with well-known Huffman coding principles, the Huffman value 305V1 for the input value V1 should have no more bits than the Huffman value 305V2, and so on. Subject to these constraints, the Huffman values 305 could be assigned in many different ways. One possible Huffman table 300 encoding the values Vi is as follows:
Given this Huffman table 300, a traditional Huffman decoder would decode the resulting compressed video stream 105 using a three-level Huffman decode tree wherein the various Huffman values are distributed at different tree levels based on their total bits and contents. For example, the traditional decode tree corresponding to the above Huffman table is shown in FIG. 8B. Given the decode tree of FIG. 8B, the traditional decoder would analyze the video data stream 105 bit-by-bit, starting at the root node of the tree and taking the applicable tree branch based on the state (1 or 0) of the current bit. The value corresponding to the Huffman value being decoded is only discovered when the decoder reaches a leaf node. For example, to decode the Huffman value 001, the traditional Huffman decoder would perform the following steps:
1) read the first bit (0)
2) take the right hand branch;
3) read the next bit (0);
4) take the right hand branch;
5) read the next bit (1);
6) take the left hand branch (to a leaf); and
7) return the leaf value (V3).
In contrast to the prior art, the present invention represents the information needed to decode Huffman values 305 generated by a particular Huffman table 300 in a Huffman decode table 430. As will be discussed in reference to FIG. 11B, the Huffman decode table 430 is a logical construct that can be subdivided into two tables 430A and 430B; however, for the purposes of this discussion, it shall be assumed that the Huffman decode table 430 is a single table. The Huffman decode table 430 has 2.sup.q entries, where q is the maximum length of a Huffman value (q is 13 in the preferred embodiment). Each entry 430i in the decode table 430 is associated with one Huffman value 305 and includes a Huffman bits field 432 that represents the number of bits in the associated Huffman value 305 and a decoded value field 434 that represents the decoded value or category associated with the same Huffman value 305. Because there are more Huffman table entries (2.sup.13 in the preferred embodiment) than values 281 to be decoded (2.sup.9 in the preferred embodiment), some of the Huffman values 305 are associated with more than one of the decode table entries. Each of the redundant entries associated with a particular Huffman value 305i has identical information (e.g, number of bits and decoded value) and has a lookup address that in its most significant bits matches the associated Huffman value 305i. For example, the Huffman decode table 430 related to the 4-entry Huffman table presented above (where q is 3) would look as follows:
Given this type of Huffman decode table 430, the decompressor 106 is able to decode compressed data chunks 105i q bits at a time (i.e., in a single lookup). For example, using the table 430 shown above, the decompressor 106 decodes any q-bit chunk that begins with a "1" to the input value V1 (because all of the entries associated with Huffman codes beginning with "1" are mapped to the same decoded value V1, the Huffman bits other than the first bit are irrelevant to the value returned by the lookup). Similarly, any q-bit chunk that begins with a "01" pattern is decoded to V2. Once it has decoded a q-bit chunk, the decompressor 106 shifts-out the number of bits used by that decode operation and does a lookup on the remaining bits of the chunk 105i until there are no unprocessed bits left. For example, the decompressor 106 would decompress the variable-length Huffman data stream 105i (101001) in the following set of steps:
1) datum 105i=101001;
2) do a lookup into decode table using first three bits of the datum 105i (101) and output the "decoded value" returned from table (V1);
3) shift the datum 105i contents left by "number of bits" returned from table (1);
4) do a lookup into decode table using first three bits of the datum 105i (010) and output the "decoded value" returned from table (V2);
5) shift the datum 105i contents left by "number of bits" returned from table (2);
6) do a lookup into decode table using first three bits of the datum 105i (010) and output the "decoded value" returned from table (V2); and
7) shift the datum 105i contents left by "number of bits" returned from table (2).
The Huffman decode table 450 and decompression steps performed by the decompressor 106 will be discussed in greater depth below, in reference to FIGS. 11A and 11B.
Referring to FIG. 9, there is shown an automated system that can be used to generate the Huffman tables 300 used in the present invention. This system includes a frame storage device 370 that stores the category data 281 from multiple video frames. The frame storage device is coupled to a frame data processor 372, which computes frequency statistics 374 (i.e., the number of occurrences) for each the 29 possible 9-bit categories 281 based on the category data from the frame storage. Based on the category statistics and the present invention's rules for generating Huffman values, the Huffman table generator 376 assigns low-bit Huffman values to the most-frequent 9-bit categories and high-bit Huffman values to the most infrequent 9-bit categories and writes the resulting total bit and Huffman value to a Huffman table 300 that can be used by the compressor 104. Of course, the inverse Huffman tables 430 used to decode the Huffman values defined in the Huffman table 300 can also be generated in this same process.
The category statistics tables used to generate the Huffman table shown in Appendix C is shown in Appendix D. This frequency table shows the octal values of each 9-bit category 281 ("octal") and the relative likelihood ("cnt") of that 9-bit category occurring in an average frame. The other columns of this table show the corresponding decimal value of the 9-bit category ("dec real"), the number of huffman bits used to encode the category ("huff bit depth") and the huffman value ("huff value"). As expected, the "octal" 000 category is the most frequent, and the categories with only one bit equal to 1 (e.g, 001, 010, or 100) are among the next most frequent. Similarly, the value for the 767 category is the least frequent value.
Compression Method
Referring to FIGS. 10-1 and 10-2, there is shown a flow chart of the video compression method of the present invention, which is implemented in the SM1 250. Because the structure of the elements involved in the compression and output steps have already been described, the present discussion will mostly focus on the overall flow of the data method.
The first step of the compression process is initiated when a new frame of data becomes available from the CCD 102, which is indicated to the SM1 via the color video control signal 257 (502). Upon receiving this signal 257, the SM1 initializes all entries 114i (R, G and B) of the one-row storage to 16 (the midpoint of the 5-bit color values) (504) and initializes the three, 5-bit last values 112R, 112G, and 112B in the last value register 112 to 16 (506). Once these values are initialized, the SM1 250 is ready to begin processing the color data for the first pixel in the data stream 103.
As the first step in processing a pixel datum 103i, the SM1 250 initializes a read of the value 115 from the one-row storage 114 that is just above the current pixel being processed (508) and the last value 113 from the last value register 112. As discussed above, the value 115 read from the one-row storage 114 corresponds to a previous pixel that lies above the pixel that is to be processed in the current compression step. At this point, the SM1 250 issues the color select signal 251D to the MUXes 118, 120, causing them respectively to select one 5-bit element of the last value 113 and one-row value 115 (510). These selected values 119, 121 are then averaged by the averager 124 (512) using the value of the carry-in signal 251E output by the SM1, which maintains the same carry-in signal value 251E for all color datums from a single pixel.
Once the previous value 125 has been output by the averager 124, the SM1 250 asserts the decimate signal 251, which causes the sub-sampler 116 to sample the video stream datum 103i and output a sampled datum 117i to the MUX 122 (516). The SM1 250 then asserts the color select signal 251D coupled to the MUX 122, upon which the MUX 122 outputs the selected 5-bit color data to the lookup table memory 200 as the 5-bit source value 123 (518). Simultaneously, the SM1 250 asserts the lookup table select signal 251F, which, with the source value 123 and the previous value 125 forms the 11-bit address of the 3-bit output category 203 and 5-bit output value 201 read out of the lookup table memory 200 (520). The SM1 250 then asserts the signal 251G, which causes the 9-bit register to store the 3-bit category read out by the lookup table memory 250 in the appropriate 3-bits of the 9-bit register 280 (522). Next, the SM1 250 asserts the read/write strobe 251C so the 5-bit output value 201 read out from the lookup table memory 200 is stored in the appropriate 5-bit segments of the last value register 112 and one-row storage 114 to be used to compress the next row of video data 103 (524). At this point, the SM1 250 repeats the steps (510) to (524) if there are remaining 5-bit color elements of the same pixel to be processed (526-NO).
Once all of the element data for the current pixel has been processed (526-YES), the SM1 250 asserts the output signal 251G, upon which the 9-bit register 280 outputs the 9-bit category 281 to the Huffman table 300 (528). The Huffman table 300 then reads out the Huffman value 305 and Huffman bit count 303 from the entry addressed by the 9-bit category (529) in three 8-bit chunks in response to the read control signal 251H asserted by the SM1 250, which, prior to the first 8-bit read out, also issues the Huffman data ready signal 251H to the output state machine 350 (530).
Upon receiving the Huffman data ready signal 251I, the output state machine 350 extracts the Huffman bit count 303 from the counter register 380C (532) and shifts out the bits from the Huffman value 305 stored in the Holding register segment 380B and 380C (534). The SM2 350 then asserts the shift in control signal 351B, which tells the output register 390 to shift-in the Huffman value bit from the Holding register 380 (534). After each bit that is shifted into the output register 390, the SM2 350 updates its count of the available space in the output register (536). If the output register is not yet full (536-NO), the SM2 keeps shifting data into the output register 390 until the holding register 380 is empty. Once the output register 390 is full (536-YES) the SM2 350 issues the data ready signal 357 to the microprocessor that controls the camera 80 (537). If the camera's microprocessor is ready for the data, it returns the ACK signal 359, upon which the SM2 350 empties the output register, resets its count of available space in the output register 390 and continues to shift Huffman values into the output register 390 one-bit at a time (538). After processing the current pixel (560-YES), the SM1 250 resets the lookup select signal 251F and carry-in 251E to change the lookup table 210 and average function that are used to compress the next sampled pixel's data 117, increments the 9-bit address 251B to match the column index of the next pixel and updates internal state data (562). If sub-sampling at rates other than 100 percent is to be used, the SM2 also determines whether the decimate signal 251A is to be set for the next available datum 103i (562). The SM1 and SM2 repeat this process until all of the datums 103i for a particular frame have been processed.
Having discussed the method and system of the compressor 104, the decompressor 106 of the present invention is now discussed.
Decompressor 106
Referring to FIG. 11A, there is shown a functional block diagram of the decompression operation as it is implemented by the decompressor 106 of the present invention. In the first decompression stage 106A, the decompressor 106 reverses the Huffman coding that took place in the second stage of data compression by taking in the compressed video datums 105i in 8-bit chunks and converting different numbers of bits (corresponding to the variable Huffman values) from the compressed datums 105i into corresponding 9-bit category values 401i. Then, in the second decompression stage 106B, the decompressor 106, using inverse Huffman tables 410A and 410B outputs for each 9-bit category datum 401i an associated 15+1(=16)-bit video data value 107i that can be displayed as part of the reconstructed video stream 107. Note that the decompressor 106 of the present invention can be implemented in hardware or software. However, for the purposes of the present application, it shall be assumed that the decompressor 106 is being implemented in software running inside a computer that controls the display of the reconstructed data 107.
Referring to FIG. 11B, there is shown a block diagram of the decompressor 106 that performs the two decompression stages described in reference to FIG. 11A. This decompressor 106 includes a 24-bit input buffer 420, a Huffman lookup processor 422, inverse Huffman tables 430A and 430B, a stage 1 decompression processor 440 and stage 1 decompression tables 460A and 460B.
The elements of FIG. 11B that perform the Huffman decompression steps are the input buffer 420, the inverse Huffman tables 430A, 430B and the Huffman lookup processor 422, which controls the other two elements. The input buffer 420 is a 24-bit buffer that receives the compressed datums 105i from the output buffer and various control signals 423 from the Huffman lookup processor 422. These control signals include a bits.sub.-- used signal 423a and a send.sub.-- bits signal 423b. The Huffman lookup processor 422 also has access to several status variables, including: bits.sub.-- used 424a, bits.sub.-- avail 424b, T1.sub.-- threshold 424c, T2.sub.-- threshold 424d and max.sub.-- bits 424e.
At the beginning of a decompression step, the input buffer 420 supplies the Huffman lookup processor 422 with an appropriate (what is meant by "appropriate" is described below) chunk of compressed video data 421 of length "max.sub.-- bits" 424e, which corresponds to the length of the longest Huffman value 305, whenever it receives the send.sub.-- bits signal 423b. In the preferred embodiment max.sub.-- bits 424e is set to 13-bits, but in other embodiments could vary between 9-bits (which would provide no compression) and 16-bits. In a process that is similar to that described in reference to FIG. 8, the Huffman lookup processor 422 decompresses the chunk 421 returned by the input buffer 420 by performing a lookup into one of the inverse Huffman tables 430A or 430B, which together compose the single logical Huffman decode table 430 described in reference to FIG. 8.
As already described, the Huffman decompression method of the present invention uses a lookup technique wherein a chunk 421 of compressed data 105 is used to read out a 9-bit category value from an inverse Huffman table 430. This compressed data 105 is stored in the input buffer 420. Because the Huffman values of the present invention can be as long as 16-bits and because the output register 390 can package together Huffman data from two different pixels (e.g., the 8-bit output register might hold a 7-bit Huffman value and 1-bit from a second, 16-bit Huffman value), the input buffer 420 needs to be at least as large as the sum of the longest Huffman value (e.g., 16 bits) and the width of the output register minus 1 (e.g, 7-bits). These values ultimately depend on the architecture of the computer hosting the decompressor 106.
In the preferred embodiment, the table 430A includes 2.sup.9 entries and the table 430B includes 2.sup.10 entries. As previously described, each of these entries includes a number.sub.-- of.sub.-- bits field 432 that indicates the number of bits in the Huffman value that corresponds to that entry and a 9.sub.-- bit.sub.-- category field 434 that provides the Huffman value's associated 9-bit category 401. The present invention employs these two tables (with a total of 2.sup.9 +2.sup.10 =1536 entries) to decode the 13-bit Huffman values 305 rather than a single 13-bit table (with a total of 2.sup.43 entries) as described in connection with FIG. 8 to save storage. These savings are important as they ensure that, regardless of the configuration of the computer executing the decompressor 106, all Huffman decoding data (i.e., the Huffman decode tables 430A and 430B) will be able to fit into the computer's cache memory, allowing for extremely fast lookups. The reason these two tables 430A and 430B can be substituted for a larger single table 430 is due to the way the present invention allocates the 2.sup.13 possible lookup addresses associated with 13-bit Huffman values to the 2.sup.9 possible 9-bit categories 281. How this allocation is accomplished will become apparent in conjunction with the following description of Huffman value decoding using the two tables 430A and 430B.
Given two Huffman decode tables 430A and 430B, the Huffman processor 422 performs a lookup on a chunk of compressed data 421 in three stages. First, the Huffman processor 422 determines whether the numerical value of the max-bits chunk 421 is greater than or equal to the Table 1 threshold (T1.sub.-- thresh) 424c, which represents the smallest lookup address (i.e., Huffman value) that is in Table 430A. Note that for this and other comparisons, all Huffman values are assumed to be max-bits in length. Thus, each of the 9-bit addresses in the table 430A are evaluated as if they are left shifted and padded to 13-bits (max.sub.-- bits) using zeros in the 4-least significant bit positions. I.e., the 9-bit addresses in the first table 430A represent the most significant 9-bits of a 13-bit Huffman value. For example, the 9-bit address "101010101" would be evaluated as if it were "1010101010000".
If the max-bits chunk 421 is larger than the Table 1 threshold 424c, the Huffman processor 422 performs a table lookup in the table 430A using the nine most significant bits 421H of the chunk 421 as the lookup address. As described in reference to FIG. 8, the returned value 401 is the 9-bit value 281 associated with the Huffman value that maps to that address 421.
If the numerical value of the max.sub.-- bits chunk 421 is less than the Table 1 threshold 424c and is greater than or equal to a similar Table 2 threshold 424d, the Huffman processor 422 does a 13-bit lookup in the Table 430B using the least significant 10-bits 421L of the 13-bit chunk 421. This is only possible because, when two Huffman decode tables 430A, 430B are used, the present invention requires that Huffman values 305 be allocated to 9-bit categories in such a way that all of the 9-bit categories that fall into the second table have Huffman values whose "r" most significant bits (where "r" is the difference between max.sub.-- bits 424e and the number of bits in the first table--e.g., 3 in the preferred embodiment) are all zeros. When this is true the 13-bit lookup can be performed into the 10-bit address space of the table 430B. As in the previous lookup, the returned 9-bit category 401 is associated with the Huffman value/address used for the lookup.
Actual examples of the 9-bit and 10-bit lookup tables 430A, 430B used in the preferred embodiment are shown in Appendix E. In these examples, the respective 9 or 10-bit lookup addresses of the tables 430A and 430B are represented by the "array val" field. Thus, the "array vals" for the table 430 are between 0 and 511 (corresponding to the 29 entries in table 430A) and the "array vals" for the table 430B are between 0 and 1023. The number of bits field 432 and 9-bit category field 434 from the respective tables 430A, 430B are encoded as a single decimal value "dec value". This "dec value" can be decoded by converting it to its 16-bit binary representation, wherein the least significant 9 bits (b0 to b8) are the 9-bit category 281, the next bit (b9) is a place holder, the next 5-bits (b10 to b14) provide the number of bits 432 and the MSB (b15) is not used. Thus, using Table 430A, for "array val"=64 (corresponding to a 9-bit address 001000000), "dec val"=16411, which has the binary representation: 0 10000 0 000011011. From this binary representation, it can be determined that the number of bits in the corresponding Huffman value is 8 ("01000") and the associated 9-bit category is 27 ("000011011"). One can verify this result by referring back to the Huffman table 300 shown in Appendix C.
When the numerical value of the max.sub.-- bits chunk 421 is less than the Table 2 threshold 424d, the Huffman lookup processor 422 outputs the least significant 9-bits of the truncated chunk. As described in connection with FIG. 8, these 9-bits are written into the max-length Huffman 305 whenever the compressor cannot assign a Huffman value to an infrequent 9-bit category. Note that in the preferred embodiment, the Table 2 threshold 424d is set to 0, meaning that all of the 9-bit categories have Huffman values in one of the two tables 430A, 430B.
The Huffman lookup processor 422 is a state machine that keeps track of and controls the progress of the Huffman decompression operation based partly on stored status data. One particularly important item of status data is the bits.sub.-- used variable 424a that is continually updated by the Huffman lookup processor 422 to indicate how many bits of the compressed data currently stored in the input buffer 420 have already been processed (i.e., decompressed). The status data also includes a bits.sub.-- avail variable 424b that indicates how many of the bits in the input buffer 420 remain to be processed.
Because the Huffman decompression method of the present invention uses a lookup technique wherein a multi-bit block of compressed data is used to read out a 9-bit category value from an inverse Huffman table 430, the Huffman lookup processor cannot perform a lookup until there is enough data in the input buffer 430 for the worst case Huffman value, which could be as long as 16-bits. Moreover, there is no guarantee that, just because the input buffer 420 has max.sub.-- bits of stored data, that all of those bits belong to a single Huffman value. This is because the output register 390 can package multiple Huffman values together and can also package incomplete Huffman values. For example, assuming the output buffer 390 provides 8-bits of data at a time, one buffer of compressed data 105 might include a 7-bit Huffman value and one bit from a 16-bit Huffman value that corresponds to the next pixel.
Consequently after each lookup, the Huffman lookup processor 422 updates the bits.sub.-- used and bits.sub.-- avail variables 424a, 424b and issues the bits.sub.-- used message 423a to the input buffer 420. Upon receiving this message, the input buffer makes the bits.sub.-- used 424 unavailable for further processing. If at this point, bits.sub.-- avail 424 is still greater than or equal to max.sub.-- bits 424e, the Huffman lookup processor 422 performs another lookup on the remaining bits in the lookup buffer. If bits.sub.-- avail 424b is less than max.sub.-- bits 424e, the Huffman lookup processor 422 issues the get.sub.-- bits message 423b to the input buffer, upon which the input buffer 420 loads another 8-bit chunk 105 from the output buffer 390 (in alternative embodiments, this could also be a 4 or 12 bit chunk) and passes the new bits on to the Huffman lookup processor 422 along with the other available bits in the input buffer 420.
Once it has decompressed a Huffman datum from the compressed bit stream 105, the Huffman lookup processor 422 outputs the corresponding 9-bit category 401 to the stage 1 decompression processor 440. The decompression processor 440 then performs a lookup into the appropriate inverse category lookup table 460A, 460B using 3-bits of the 9-bit category 401 and 5-bits of the 15-bit previous value (computed by the decompressor 106 in the previous processing interval). Note that the inverse lookup tables 460A and 460B are respectively associated with the lookup tables 210A and 210B and are used in the same alternating pattern. As with the lookup tables 210, the value of the 5-bit previous pixel is set to 16 when the pixels being decoded are in the first row or first column of a frame. Using respective ones of the 5-bit previous values and 3-bit categories from the 9-bit combined category 401, the decompression processor 440 is able to generate the 5-bit RGB values for a decompressed video datum 107i, which is packaged as a 16-bit value that can be displayed as part of a video image.
While the present invention has been described with reference to a few specific embodiments, the description is illustrative of the invention and is not to be construed as limiting the invention. Various modifications may occur to those skilled in the art without departing from the true spirit and scope of the invention as defined by the appended claims. ##SPC1##
Claims
What is claimed is:
1. A video compression system for compressing a digital video stream of N-bit datums, each of said datums corresponding to one pixel in a video image with an associated column index (i) and row index (j) and including Q M-bit subcomponents representing respective source values of Q elements of said one pixel, said datums appearing in said stream in raster order, said, system comprising: a first lookup table with at least 2.sup.2M first entries, each of said first entries storing a P-bit first category value and a M-bit first output value and being uniquely associated with a M-bit previous value and M-bit source value, said first lookup table being configured to read out a first particular entry's first category and output values upon receiving as a readout address said first particular entry's associated previous and source values and a first table select signal, said read out first category representing a compressed version of said associated source value; a second lookup table with at least 2.sup.2M second entries, each of said second entries storing a P-bit second category value and a M-bit second output value and being uniquely associated with a M-bit previous value and M-bit source value, said second lookup table being configured to read out a second particular entry's second category and output values upon receiving as a readout address said second particular entry's associated previous and source values and a second table select signal, the rows of said second lookup table being a reflection of corresponding rows of said first lookup table about the columns of said corresponding first table rows for which said first category is 0; said read out second category representing a compressed version of said associated source value; an address circuit coupled to said lookup tables that computes and provides said previous value, said address circuit being configured to compute said previous value as a function of output values of selected previous pixel elements according to a first formula when said first table select signal is available and according to a second formula when said second table select signal is available; an output circuit coupled to said lookup tables that is configured to receive and store said P-bit category values for all of said Q elements of said one pixel, said output circuit also being configured to output a single Q.times.P-bit combined category representing said Q category values for said one pixel; and a first state machine that controls and coordinates operations of said lookup tables, said address circuit, and said output circuit, said first state machine being configured to provide said first table select signal and said second table select signal so that said first and second lookup tables and said first and second formulas are used to readout said category and output values for alternating pixels, thereby reducing color banding in said video images reconstructed from said combined category information, and to cause said output circuit to store and output said Q.times.P-bit combined category for said one pixel.
2. The video compression system of claim 1, wherein said address circuit comprises: a last value register with Q last value segments coupled to said lookup tables, said last value register being configured to write into a selected one of said Q last value segments said output value read out from either one of said tables and read out said last value segments; one-row storage with at least i entries, each having Q one-row segments, said one-row storage being configured to write into a selected one of said i entries at a selected one of said Q one-row segments said output value read out from either one of said tables; a selector coupled to said last value register and said one-row storage that selects said one of said Q one-row segments and said one of said Q last value segments; an averager that averages a M-bit last value selected from one of said read out last value segments and a particular 5-bit value read out by said one row storage, said particular value corresponding to the output value associated with one selected element of a pixel having the same column index (i) as and a row index (j-1) that is one less than the row index (j) of the most recent pixel represented in said data stream, said last value and said particular value representing the same pixel element type.
3. The video compression system of claim 1, wherein said elements comprise red, green and blue pixel elements and each of said M-bit subcomponents encodes the intensity of a respective one of said pixel elements, further comprising: a set of multiplexers, each of said multiplexers being coupled to three inputs carrying M-bit red, green and blue information and a RGB select signal generated by said first state machine that determines which of said information signals is selected and output by said multiplexer; said set of multiplexers including: a first multiplexer coupled to the output of said last value register that provides said M-bit last value; and a second multiplexer coupled to the output of said one-row storage that provides said M-bit particular output value.
4. The video compression system of claim 3, further comprising: a sub-sampler that is configured to eliminate a fixed percentage of datums from said video stream before said datums are processed, said sub-sampler having an input coupled to said video stream and an output coupled to a third multiplexer; said third multiplexer having an additional input coupled to said RGB select signals and an output coupled to said input table so that said third multiplexer passes one of the red, green or blue information as said source value.
5. The video compression system of claim 1, further comprising: a Huffman encode table having at least 2.sup.Q.times.P entries, each of said entries including: a Huffman data field for storing a variable-length Huffman value for one of said combined categories; and a Huffman bits field for storing a Huffman bit count representing the total bits in said Huffman value; said Huffman encode table being coupled to said output circuit and being configured to read out a Huffman value and Huffman bit count associated with said combined category output by said output circuit; said Huffman values being allocated to ones of said combined categories so that said ones more likely to occur in a video frame are allocated in said Huffman encode table with Huffman values having fewer bits, said Huffman encode table thereby providing additional data compression of data in said video frame above that provided by said first lookup table.
6. The video compression system of claim 5, further comprising: a Huffman decode table that is configured to receive and decode a compressed data stream in X-bit chunks, where X represents the number of bits in the longest of said Huffman values, said compressed data stream including plural compressed datums, each of which corresponds to one of said Huffman values read out from said Huffman encode table, each said chunk including at least one Huffman value aligned with the most significant bit position of said chunk; said Huffman decode table having at least 2.sup.X entries, each including said combined category and said Huffman bits data associated with a Huffman value that is mapped to said entry's X-bit address; said Huffman decode table also being configured to read out said combined category and said Huffman bits data in said entry indexed by said fixed-length chunk input to said Huffman decode table, said Huffman decode table being structured so that a set of Huffman decode table entries with a said X-bit address whose most significant Z bits matches a Z-bit Huffman value include the same said combined data and said Huffman bits data, said same data being associated in said Huffman encode table with said Z-bit Huffman value; such that said same data is read out from any one of said entries when said most significant Z bits of said fixed length chunk match said Z bits, thereby allowing Huffman data to be decoded in a single lookup.
7. The compression system of claim 6, wherein said Huffman encode table comprises a first and second Huffman decode table; said first Huffman decode table having 2.sup.B entries addressable by B-bit addresses that correspond to the most significant B bits of said X-bit lookup addresses, each of said B-bit addresses addressing a respective first Huffman decode table entry whose combined data and Huffman bits data are the same as said entry in said Huffman encode table addressed by an X-bit lookup address with the same most significant B bits; and said second Huffman decode table having 2.sup.C entries addressable by C-bit addresses that correspond to the least significant C bits of said X-bit lookup addresses; each of said C-bit addresses addressing a respective second Huffman decode table entry whose combined data and Huffman bits data are the same as said entry in said Huffman encode table addressed by an X-bit lookup address with the same least significant C bits and most significant (X-C) bits that are all zero, C being less than X, and the sum of 2.sup.C and 2.sup.B being less than 2.sup.X ; said first and second Huffman decode tables thereby occupying less space than said Huffman decode table; said Huffman values being selected so that all Huffman values that are mapped to X-bit addresses in said second table have (X-C) most significant bits that are all zero.
8. The compression system of claim 6, further comprising: first and second lookup decode tables corresponding to said first and second lookup tables, said lookup decode tables being configured to output an M-bit output value to a lookup address including a P-bit category output by said Huffman decode table as part of said combined categories and a corresponding M-bit output value from a previously-decompressed neighboring pixel; said first and second lookup decode tables respectively being used to convert said P-bit categories for alternating pixels, said lookup decode tables corresponding respectively to said first and second lookup tables; said M-bit output values for a single pixel being collected into a N-bit reconstructed pixel datum that can be displayed as part of a reconstructed video stream.
Patent Citations (10)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4598372 | 1986-07-01 | McRoberts | |
| US4785349 | 1988-11-01 | Keith et al. | |
| US4835607 | 1989-05-01 | Keith et al. | |
| US4868653 | 1989-09-01 | Golin et al. | |
| US4939583 | 1990-07-01 | Tsuboi et al. | |
| US5123059 | 1992-06-01 | Hirosawa et al. | |
| US5220325 | 1993-06-01 | Ackland et al. | |
| US5389965 | 1995-02-01 | Kuzma | |
| US5400075 | 1995-03-01 | Savatier | |
| US5528628 | 1996-06-01 | Park et al. |