Method and System for Compression and Decompression of Data

This invention relates to data compression and decompression and, more particularly, to a method and system for encoding that is a combination of fixed-length and variable-length encoding.

The electronic collection and storage of information presents problems in many environments. For example, when clinical data is collected from a patient through a patient monitor, vast amounts of data can be collected during a short interval. A computer system typically stores this data in computer memory for later processing. The computer memory has a limited capacity and can quickly fill up with the collected data.

To reduce the memory requirements of collected data, some computer systems compress data before storing it in the computer memory. When the collected data is needed for processing, the computer system decompresses the data. Thus, while the data is not needed, the memory requirements are minimized. Similarly, to reduce data transmission time, data is transmitted from one computer to another in compressed format. The receiving computer then decompresses the data before processing.

Many compression and decompression methods are known. Various methods have a variety of advantages. For example, some methods generate a large reduction in data size. Other methods preserve all the original data through the compression and decompression process. These methods are referred to as being lossless because no data is lost. Other methods compress data very quickly or decompress data very quickly. These various methods also have a variety of disadvantages. For example, some methods do not generate a large reduction in data size. Other methods do not preserve all the original data through the compression and decompression process; that is, the decompressed data is only an approximation of the original data. Also, some methods compress or decompress slowly. Some methods use considerable computer memory when compressing or decompressing. Some methods are usable for specific types of data, e.g., ASCII text or electrocardiogram wave forms.

Some compression and decompression methods use a fixed-length encoding, while others use a variable-length encoding. A fixed-length encoding method represents each symbol by the same number of bits. For example, the American Standard Code Information for Interchange (ASCII) is a fixed-length encoding. The ASCII standard specifies that each character is represented by 8 bits. A variable-length encoding method represents the symbols by a varying number of bits. For example, the well-known Morse code is a variable-length encoding.

Fixed-length and variable-length encoding methods each have advantages and disadvantages. A major advantage of a variable-length encoding is that it produces a smaller encoded data buffer than a fixed-length encoding. Variable-length encoding methods typically assign shorter codes to more frequently used symbols. A major advantage of fixed-length encoding is that the encoding and decoding are particularly efficient when performed by a computer system. For example, when the code length is a multiple of a byte length, then bit manipulations (e.g., shifting) that are necessary for variable-length codes are not needed.

It would be desirable to have an encoding scheme that is not dependent on the type of data, that produces small compressed data buffers, and that allows for efficient decoding.

It is an object of the present invention to provide a data compression and decompression method that combines both a variable-length and fixed-length encoding.

It is another object of the present invention to provide an encoding method based on modulo arithmetic in which a number is encoded as a variable-length quotient portion and a fixed-length remainder portion.

These and other objects, which will become apparent as the invention is fully described below, are obtained by an improved method and system for compressing and decompressing data. In preferred embodiments, a computer system compresses a data buffer that comprises a plurality of symbols wherein each symbol has one or more occurrences. The system generates a symbol table containing each symbol and the count of the occurrences in the data buffer and having an index for each symbol. For each occurrence of a symbol in the data buffer, the system determines the index for that symbol into the symbol table and encodes the index into a format with a variable-length portion, a delimiter, and a fixed-length portion. The system then outputs the encodings as the compressed data buffer.

FIG. 1 is an overview of the compression and decompression system in a preferred embodiment of the present invention.

FIG. 2 shows sample encodings for index 13 in a preferred embodiment of the present invention.

FIG. 3 shows sample data of the compression and decompression system in a preferred embodiment.

FIG. 4 is a flow diagram of a routine to select the suffix length for a data buffer in a preferred embodiment.

FIG. 5 shows the compressed buffer sizes for the sample data buffer of Table A for suffix lengths 0, 1, 2, and 3

FIG. 6 is a flow diagram of a routine that encodes a data buffer in a preferred embodiment.

FIG. 7 is a flow diagram of a routine that decodes a compressed data buffer in a preferred embodiment.

FIG. 8 is an overview flow diagram of the compression system.

FIG. 1 is an overview of the compression and decompression system in a preferred embodiment of the present invention. Data buffer 101 contains data to be compressed. The data is stored in the data buffer 101 in the form of symbols. Each symbol comprises contiguous bits, preferably, forming one or more bytes. To compress the data, a compression module 102 inputs from the data buffer 101 and outputs to a compressed data buffer 103, symbol table 104, and suffix length 105. Compression module 102 may be implemented on a properly programmed computer. To decompress the compressed data, the decompression module 106 inputs the compressed data buffer 103, the symbol table 104, and the suffix length 105 and outputs the decompressed data buffer 107. The decompression module may be implemented on a properly programmed computer.

To compress the data in the data buffer 101, the compression module 102 first generates a count table (not shown) containing an entry for each unique symbol in the data buffer 101. The entries contain the symbol and the count of the number of occurrences of the symbol in data buffer 101. The count table is preferably sorted in descending order based on the count of the number of occurrences of each symbol in the data buffer 101. The symbol table 104, which is sent to the decompression module 106, is a table of symbols sorted in descending order based on the count of occurrences. The symbol table 104 is generated after the count table. FIG. 3 shows sample data of the compression and decompression system in a preferred embodiment. The sample data buffer 301 contains 20 symbols that are to be encoded. The count table 309 contains one entry for each unique symbol (A through F) in data buffer 301 sorted based on the count. Because symbol C occurs more times in data buffer 301 than the other symbol (its count is 7), it is the first entry in count table 309.

After the count table is completed, the compressed data buffer 103 is generated. The compression module 102 retrieves each symbol serially from data buffer 101, generates the symbol encoding, and then adds the encoding to the compressed data buffer 103. Referring now to FIG. 3, as shown in the encoding table 308, symbol C is encoded as "000" and symbol E is encoded as "1001." In a preferred embodiment, an encoding table is not actually generated. Rather, a symbol encoding is generated each time a symbol is received from the data buffer. The encoding process is described below in detail. Again referring to FIG. 3, the data compression module 302 first retrieves the symbol C from the data buffer 301, generates the encoding, and adds the encoding ("000") to the compressed data buffer 303 at 350. The data compression module 302 then processes the next symbol in the data buffer 301, which is also the symbol C, generates the encoding, and adds the encoding to the compressed data buffer 303 at 351. The data compression module 302 then retrieves the symbol A from the data buffer 301, generates the encoding, and adds the encoding ("001") to the compressed data buffer 303 at 352. The compression module 302 continues with processing all the symbols in the data buffer 301 to generate the compressed data buffer 303.

The compression module 102 encodes the symbols using a variable-length encoding. As shown in FIG. 3, the encoding for symbol C is 3 bits, while the encoding for symbol D is 4 bits. The encodings represent the indices into the symbol table. For example, the encoding "000" represents the index 0 (position of symbol C) into the symbol table, while encoding "1000" represents index 4 (position of symbol D). The encodings have a variable-length prefix, a delimiter, and a fixed-length suffix. The variable-length prefix may have a length of zero. However, if the length of the prefix is greater than zero, it will consist of a string of 1s. The delimiter is a "0," which separates the prefix from the suffix, and the suffix is an integer value, which may have a length of zero. If the suffix has a length of zero, then its integer value is considered to be zero. For example, symbol D is encoded as a "1000." The first bit ("1") is the prefix; the next bit ("0") is the delimiter; and the last two bits ("00") are the suffix. The length of the suffix is 2 bits as shown in FIG. 3 at 305. The selection of a suffix length for a data buffer is described below in detail. Symbol A is encoded as a "001." This encoding has a zero-length prefix since there are no leading 1s. The delimiter is the first bit ("0") and the suffix is the last two bits ("01").

The encoding method of the compression module can be best understood by illustrating the conversion of an encoding to an index during decompression. An index is derived from an encoding by multiplying the number of 1s in the prefix by 4 (when the suffix length is 2) and adding the integer value of the suffix. For example, "010" represents the index 2, because there are no leading 1s (0*4), the delimiter is "0," and the integer value of the suffix ("10") is 2. The encoding "1001" represents the index 5 (1*4+1) , because there is a leading "1" (1*4), the delimiter is "0," and the integer value of the suffix ("01") is 1. An index is encoded by the reverse process. The encoding is formed by creating a string where the prefix contains a number of ones equal to the quotient of the index divided by 4 (when the suffix length is 2), the delimiter is "0," and the prefix is the remainder of index divided by 4 (when the suffix length is 2) represented in a binary integer format. For example, the index 2 is encoded as "010,38 because the prefix length is "0 (2/4), the delimiter is "0," and the suffix is "10" (remainder of 2/4). The index of 5 is encoded as "1001," because 5 divided by 4 is 1 (prefix of "1"), the delimiter is "0," and the remainder of 5 divided by 4 is 1 (the suffix of "01").

To decompress the compressed data buffer 103, the decompression module 106 parses the compressed data buffer 103 into encodings for symbols, converts each encoding into an index into the symbol table, retrieves the indexed symbol from the symbol table, and adds the retrieved symbol to the decompressed data buffer 107. For example, as shown in FIG. 3, decompression module 306 first retrieves the string "000" from the compressed data buffer 303, converts the string to index 0, retrieves the symbol ("C") at index 0 from the symbol table, and adds the symbol C to the decompressed data buffer 307. The decompression module 306 continues parsing the encodings and adding the indexed symbols to the decompressed data buffer 307. The decompression module 106 parses the compressed data buffer 103 by counting leading 1s, skipping the delimiter, and saving the fixed-length suffix. The index is calculated by multiplying the count of leading 1s by 2 to the power of the suffix length (2.sup.suffixlength) and adding in the saved suffix value. The indexed symbol is then retrieved from the symbol table 104 and stored in the decompressed data buffer 107.

In the example of FIG. 3, the suffix length is 2. In a preferred embodiment, the suffix length can vary from data buffer to data buffer. The suffix length for a data buffer is selected to minimize the size of the resulting compressed data buffer as explained in detail below. The indices are encoded in modulo arithmetic format based on the suffix length. In modulo arithmetic format, a number is represented by the integer portion of a quotient, a remainder, and a base. For example, the number 51 is represented in modulo base 4 format as (12, 3), that is, an integer portion of 12 and remainder of 3 (12*4+3=51) and represented in modulo base 5 as (10, 1), that is, an integer portion of 10 and a remainder of 1 (10*5+1=51). In modulo arithmetic format, the base is the divisor and the number to be represented is the dividend. In the present invention, the index is the number to be represented (the dividend) and 2 to the power of the suffix length (2.sup.suffixlength) is the base (divisor). The prefix represents the quotient, and the suffix represents the remainder. The number 1s in the prefix is equal to the quotient. For example, if quotient is 3, then the prefix is "111." The remainder is stored in binary arithmetic format. For example, if the remainder is 5 and the suffix length is 4, then the suffix is "0101." The divisor is defined by the selection of the suffix length. The divisor is 2 to the power of the suffix length. For example, if the suffix length is 4 then the divisor is 16(2.sup.4).

FIG. 2 shows sample encodings for index 13 in a preferred embodiment of the present invention. The encodings vary based on the suffix length. In this example, if the suffix length is 0, then the length of the encoding is 14 bits as shown by 205. If the suffix length is 1, then the length of the encoding is 6, as shown by 204. The encoding for index 13 with a suffix length of 2 is "111001." The quotient of 3 is represented by the string "111" in the prefix 201. The first "0" is the delimiter 202. The remainder, which is 1, is represented by the binary value "01" in the suffix 203.

Table A shows the symbols, counts, and encodings for a sample data buffer. The column entitled "symbol" contains the symbol. The column entitled "count" contains the count of the occurrences of the corresponding symbol in a sample buffer. The column entitled "index" represents indices into the symbol table. The columns entitled "suffix length=2" and "suffix length=3" contain the encodings for each of the indices into the symbol table. In this example, the symbol 119 is the fourteenth entry (index=13) into the symbol table and occurs 165 times in the data buffer. The encoding for index 13 is "111001" when the suffix length is 2 and "10101" when the suffix length is 3.

FIG. 8 is an overview flow diagram of the compression method. The input to this system is a data buffer of symbols, and the output is a compressed data buffer, a symbol table, and a suffix length. In step 801, the system generates the symbol table for the data buffer. The symbol table contains each unique symbol in the data buffer sorted by the number of occurrences of the symbol in the data buffer. In a preferred embodiment, the system also generates a count table containing the number of occurrences of each symbol in the data buffer. In step 802, the system selects a suffix length for the compressed data. This selection method is described below in detail. The suffix length is selected to minimize the size of the resulting compressed data buffer. In step 803, the system generates an encoding for each symbol and generates the compressed data buffer. This method is described below in detail.

FIG. 4 is a flow diagram of a routine to select the suffix length for a data buffer in a preferred embodiment. This routine determines which suffix length will produce the smallest compressed file. The routine determines the size of the compressed data buffer for successive suffix lengths. When a suffix length would produce a compressed data buffer size that is larger than that for the previous suffix length determined, then that previous suffix length produces the smallest compressed buffer. FIG. 5 shows the compressed buffer sizes for the sample data buffer of Table A for suffix lengths 0, 1, 2, and 3. The compressed buffer size for suffix length 0 is 66300, 1 is 43553, 2 is 36372, and 3 is 37338. Because the buffer size for suffix length 3 is larger than the buffer size for suffix length 2, suffix length 2 is preferably selected for the encoding. FIG. 5 is one table with portion 500B being a continuation of portion 500A. Columns 501 and 502 comprise the symbol table. Column 501 contains the indices into the symbol table, and column 502 contains the symbols. Column 503 contains the counts of occurrences of the symbols. Columns 504 through 511 show the number of bits needed to store all occurrences of the corresponding symbol in the compressed data buffer. Column 504 shows the number of bits needed to encode the corresponding symbol when the suffix length is 0; column 505 shows the number of bits needed to encode all occurrences of the corresponding symbol when the suffix length is 0. Columns 506 and 507, columns 508 and 509, and columns 510 and 511 show the corresponding information for suffix lengths 1, 2, and 3, respectively. For example, it takes 1130 bits to encode all occurrences of symbol 129 (index 0) when the suffix length is 0, and it takes 2260 bits to encode when the suffix length is 1. At the bottom of columns 505, 507, 509, and 511, the values 512, 513, 514, and 515 indicate the total size of the compressed buffer when the suffix length is 0, 1, 2, and 3 respectively. In steps 401 through 407, the routine loops determining the size of the compressed buffer for each suffix length until the minimum size is found. In step 401, the routine initializes variables for the looping. The routine sets variable newtotal to the maximum integer value and variable suffixlength to -1. Variable newtotal accumulates the total number of bits in the compressed buffer for a given suffix length. Variable suffixlength indicates the suffix length currently being processed. In step 402, the routine sets variable oldtotal equal to variable newtotal. Variable oldtotal contains the compressed buffer size for the last suffix length processed. The routine sets variable newtotal to 0 and increments variable suffixlength to indicate the next suffix length. In steps 403 through 406, the routine loops accumulating the total number of bits in the compressed buffer for the suffix length indicated in variable suffixlength. In step 403, the routine initializes index i to 0. Index i is an index into the symbol table. In step 404, the routine increases variable newtotal by the number of bits needed to encode all occurrence of the symbol indexed by index i. The length of the prefix is index i divided by 2 to the power of variable suffixlength (i/2.sup.suffixlength). The length of the delimiter is 1. The length of the suffix is the value in variable suffixlength. Thus, the encoding length for all occurrences of the symbol indexed by index i is given by the length of each occurrence times the number of occurrences as stored in the count table ((i/2.sup.suffixlength +1+suffixlength)* count[i]). In step 405, the routine increments index i to index the next symbol in the symbol table. In step 406, if index i equals the number of symbols in the symbol table, then all the symbols have been processed and the routine continues at step 407, else the routine loops to step 404 to process the next symbol. In step 407, if variable newtotal is greater than or equal to variable oldtotal, then the previous suffix length processed allowed for the minimum buffer size and the routine decrements variable suffixlength to point to the previous suffix length and returns, else the routine loops to step 402 to process the next suffix length.

FIG. 6 is a flow diagram of a routine than encodes a data buffer in a preferred embodiment. The input to this routine is the data buffer, the symbol table, and the suffix length. In steps 601 through 608, the routine loops processing each symbol in the data buffer. In step 601, the routine selects the next symbol in the data buffer, starting with the first. In step 602, if all the symbols have been processed, then the encoding is done and the routine returns, else the routine continues at step 603. In step 603, the routine determines the symbol table index for the selected symbol. In a preferred embodiment, the routine searches the symbol table for the selected symbol. Alternatively, the cross reference table of indices that is indexed by the symbol could be generated. In step 604, the routine determines the length of the prefix. The prefix length is the index into the symbol table divided by 2 to the power of the suffix length (index/2.sup.suffixlength). In step 605, the routine determines the value of the suffix. The suffix value is the remainder of the symbol table index divided by 2 to the power of the suffix length (index % 2.sup.suffixlength). In step 606, the routine adds the prefix to the compressed data buffer. The prefix is a string of 1s of the length defined by the prefix length. In step 607, the routine adds a 0 as a delimiter to the compressed data buffer. In step 608, the routine adds the suffix to the compressed data buffer and loops to step 601 to process the next occurrence of a symbol

FIG. 7 is a flow diagram of a routine that decodes a compressed data buffer in a preferred embodiment. The input to this routine is the compressed data buffer, the symbol table, and suffix length. The output is the decompressed data buffer. In steps 701 through 705, the routine loops searching for the next delimiter bit in the compressed data buffer and determining the length of the prefix. In step 701, the routine initializes the variable prefixlength to 0. In step 702, the routine selects the next bit from the compressed data buffer, starting with the first bit. In step 703, if all the bits have been processed, then the decoding is complete, else the routine continues at step 704. In step 704, if the selected bit equals 0, then it is a delimiter and the routine continues at step 706 to determine the suffix, else the routine increments the variable prefixlength in step 705 and loops to step 702. In step 706, the routine selects the next number of bits indicated by the suffix length. In step 707, the routine sets variable suffix equal to the integer value of the selected bits. In step 708, the routine converts the encoding to the symbol table index. The variable index is set to the variable prefixlength times 2 to the power of the suffix length plus the variable suffix. In step 709, the routine adds the symbol table value indicated by the variable index to the decompressed data buffer and loops to step 702 to decode the next index.

Although the present invention has been described in terms of a preferred embodiment, modifications within the spirit of the invention will be apparent to those skilled in the art. For example, one skilled in the art would appreciate that the encoding could be rearranged to have the fixed-length portion first, followed by the variable-length portion, and then the delimiter, or that the delimiter could be a 1 and the variable-length portion could be a string of 0s. The scope of the present invention is defined by the claims which follow.