Data Compression System for Successively Applying at Least Two Data Compression Methods to an Input Data Stream

This application is related to patent application Ser. No. 308,767 filed Feb. 9, 1989 for John T. O'Brien et al.

This invention relates to data processing systems and, in particular, to an improved data compression apparatus that efficiently reduces the size of the data file for transmission or storage on the data processing system.

It is a problem in the field of data processing systems to reduce the size of the data files that are stored in memory. These data files typically contain a significant amount of redundancy in the data. There are four basic types of data redundancy. The first type of data redundancy is character distribution, wherein in a typical character string some characters are used more frequently than others. A second type of data redundancy is character repetition, wherein a string of repetitions of a single character occurs. In this case, the message can usually be encoded more compactly than by just repeating the character symbol. A third type of data redundancy is high usage patterns, wherein certain sequences of characters appear with relatively high frequency. These sequences of characters can be represented with relatively fewer bits for a net saving in both time and space. The fourth type of data redundancy is positional redundancy, wherein certain characters appear consistently at a predictable place in each block of data. These four types of data redundancy enable data compression algorithms to reduce the size of the data file for data storage purposes.

Lempel and Ziv, in numerous published papers, describe the method of compressing data in which the compressor maintains a dictionary of character sequences which have already been encountered in the input data stream. When a sequence of characters is being compressed and that sequence of characters has already been encountered and stored in the dictionary, the compressor causes a reference value (sometimes called a pointer, code word or token) to be output. The reference value identifies the string in the dictionary that is the same as the sequence of characters being compressed. Generally, the number of bits required to identify the entry in the dictionary that contains this sequence of characters is smaller than the number of bits that would have been required had the characters that make up the string been output. Thus, the data is compressed.

This process of compressing data by replacing sequences of characters with a reference value which identifies an entry in a dictionary is called textual substitution. The reconstruction of the original data by a decompressor is made possible by having the decompressor build the same dictionary of strings which the compressor built as it was compressing the data. Because modifications to the dictionary are based only on strings which have been previously encountered, the decompressor is capable of performing the same updates to the dictionary as it is decompressing the data as the compressor performed as it was compressing the data.

Many variations on this basic data compression technique have been considered and implemented. Among the components of the basic method which may be changed are: the method of initializing the dictionary; the method of matching input character sequences with strings contained in the dictionary; the method of selecting which strings of characters should be added to the dictionary and where in the dictionary they should be added; and if the dictionary is limited to a finite size, the method of choosing strings to be deleted from the dictionary.

For example, U.S. Pat. No. 4,814,746 issued Mar. 21, 1989 to V. S. Miller et al. teaches a data compression method which modifies the data compression method of Lempel and Ziv by addition of new character and new string extensions to improve the compression ratio. This method also includes a routine to delete the least recently used string to limit the encoding tables to a fixed size to thereby significantly improve data transmission efficiency.

Another data compression patent is U.S. Pat. No. 4,558,302 issued Dec. 10, 1985 to P. A. Welch that discloses a data compression method which searches the input data stream to determine the longest match of a stored string. Each stored string comprises a prefix string and an extension character where the extension character is the last character in the string and the prefix string comprises all but the extension character. When the longest match between the input data character string in the stored strings is determined, the code signal for the longest match is transmitted as the compressed code signal for the encountered string of characters and an extension string is stored in the string table. The prefix of the extended string is the longest match and the extension character of the extended string is the next input data character following the longest match.

A characteristic of all these compression techniques is that they do not handle sequences of the same character repeated several times in a row any differently than other sequences of characters even though repetitions of a single character are commonly encountered in the data stored or transmitted by a computer.

Run length limiting is a form of data compression which is widely employed to cause the representation of the repetition of a character to require fewer bits than the sequence of repeated characters originally required. It is generally accomplished by choosing a character to serve as an escape character. When a character is found to be repeated several times in the input data stream, the repeated characters are replaced with the single instance of the character followed by the escape character and a repeat count. For simplicity in implementation, the repeat count is usually expressed in a form which occupies the same amount of space as a character occupies. Because of the common occurrence of character repetitions in computer data, this run length limiting technique is effective, but it has several disadvantages.

One problem arises when the character chosen to be the escape character also occurs as data in the input data stream. If the escape character is placed in the output data stream unmodified, the decompression of the data fails to reproduce the original data because the escape character is interpreted as indicating that the character following it is a repeat count. Many solutions to this problem exist, though they usually require producing a two character sequence in the output data stream whenever the escape character appears in the input data stream. Therefore, if the input data stream contains many instances of the escape character, the output data stream may actually occupy more space than the input data stream. Another problem with traditional run length limited data compression techniques is that the specification of the character, its repeat count and the escape character usually requires the same space that three characters require. Thus, compression of the data only occurs if the runs consist of four or more characters.

While run length limited data compression is useful and easy to implement, it does not yield compressed data streams which are as small as those yielded by other data compression techniques. Generally, these other techniques perform a mapping of all the input characters into output symbols. This reduces the size of the output data stream and also provides the opportunity to define additional output symbols to serve the purpose of the escape character of the run length limiting data compression technique. This eliminates the problems which arise when the escape character appears in the input data stream. Furthermore, these data compression techniques commonly use output symbols which vary in size. This means that provisions for handling varying width symbols have also been incorporated into these compression techniques and it is practical to consider using repeat counts which vary in width according to the value of the repeat count. An effective technique for identifying the width of a repeat count in the output data stream is to use a separate escape code output symbol for each possible width of a repeat count.

The problem with these aforementioned methods of data compression is that each one is directed to a particular aspect of data compression and they do not very efficiently handle both run length encoding and textual substitution.

The above described problems are solved and a technical advance achieved field by the improved data compression system for concurrently processing both textual substitution and strings of repeated characters. In this system, the performance of data compression techniques based on textual substitution are improved by the use of a compact representation for identifying instances in which a character in the input data stream is repeated. A further performance improvement is obtained by expanding the alphabet of symbols stored in the compressor's dictionary to include both the characters of the input data stream and repeat counts which indicate the repetition of a character. The handling of these repeat counts by the textual substitution based compression technique may be no different than the handling of characters, or certain modifications can be made in the handling of repeat counts. Therefore, data compression effectiveness is improved not only by using a compact representation for sequences of repeated characters, but also by storing character sequences which include repeated characters in the dictionary so that the entire sequence is encoded in a single reference value.

Textual substitution data compression techniques lead to the definition of strings in the dictionary which contain sequences of repeated characters. However, the strings in the dictionary are built a character at a time or a string at a time, and many instances of a long sequence of repeated characters would have to be encountered before a string is defined which allows the entire sequence to be encoded in a single reference value. The compression efficiency of this textual substitution system is improved by nesting a run length encoding system in the textual substitution system. This structure adds the recognition of runs of a repeated character before the data compression system performs the textual substitution data compression operation. The dictionary in this system is capable of storing strings made up of a combination of repeat counts and characters from the input data stream. In the implementation of the dictionary, the number of bits available to store each element of the string must be greater than the number of bits which are required to store only characters. Thus, simply changing the basic textual substitution algorithm by increasing the size of the alphabet of input characters to include repeat counts yields improved compression effectiveness.

FIGS. 1 and 2 illustrate in, block diagram form, the of architecture the present improved data compression system; and

FIGS. 3 and 4 illustrate in, flow diagram form, the operational steps taken by a typical data compression system and by the improved data compression system to compress data, respectively.

The improved data compression system more efficiently processes strings of repeated characters than previous data compression methods. In this system, the performance of data compression technique based on textual substitution are improved by the use of a compact representation for identifying instances in which a character in the input data stream is repeated. This is accomplished by nesting a run length encoding system in the textual substitution system. This structure adds the recognition of runs of a repeated character before the data compression system performs the textual substitution data compression operation.

A further performance improvement is obtained by expanding the alphabet over symbols stored in the compressors dictionary to include both the characters of the input data stream and repeat counts which indicate the repetition of a character. The handling of these repeat counts by the textural substitution based compression technique may be no different than the handling of characters, or certain modifications can be made in the handling of repeat counts. Therefore, data compression effectiveness is improved not only by using a compact representation for sequences of repeated characters, but also by storing character sequences which include repeated characters in the dictionary so that the entire sequence is encoded in a single reference value.

Textual substitution data compression techniques lead to the definition of strings in the dictionary which contain sequences of repeated characters. However, the strings in the dictionary are built a character at a time or a string at a time, and many instances of a long sequence of repeated characters would have to be encountered before a string is defined which allows the entire sequence to be encoded in a single reference value. The compression efficiency of this textual substitution system is improved by adding the recognition of runs of a repeated character before the data compression system performs the textual substitution data compression operation. This requires that the dictionary be capable of storing strings made up of a combination of repeat counts and characters from the input data stream. In the implementation of the dictionary, this requires that the number of bits available to store each element of the string must be greater than the number of bits which are required to store only characters. Thus, simply changing the basic textual substitution algorithm by increasing the size of the alphabet of input characters to include repeat counts yields improved compression effectiveness.

The method of operation of a typical data compression system is illustrated in FIG. 3. At step 301, the dictionary of strings is initialized. The details of this initialization process are a function of the particular data compression algorithm that is selected. The dictionary can be initialized to be empty or may contain some or all of the possible single character strings or may even contain multiple character strings which are expected to be encountered in the input data stream of uncompressed characters. Once the dictionary is initialized, a character pointer is initialized at step 303 to point to the first character in the input data stream to be compressed.

The data compression process is implemented by the loop consisting of steps 304-314 which operates until no uncompressed characters remain in the input data stream. At step 304, a determination is made whether any more characters remain to be compressed. If no characters remain, the data compression process exits at step 315. If the character pointer is not beyond the last character to be compressed, processing advances to step 305. The string dictionary is searched at step 305 to find a string which matches the uncompressed characters in the input data stream, starting at the position identified by the character pointer. The result of this search is tested at step 306. If a string was found in the string dictionary which matches one or more of the uncompressed characters in the input data stream, a string reference value is output at step 307 to identify the matching dictionary string. The character pointer is then advanced at step 308 by the number of characters contained in the matching string.

If, at step 306, no matching string was found in the string dictionary, then a field is output at step 310 to identify the character presently pointed to by the character pointer. The character pointer is then advanced at step 312 by one character.

The additional steps 313, 314 are shown to illustrate the step of deleting (313) one or more strings from the string dictionary and then to possibly add (314) one or more strings to the string dictionary. There are many different techniques proposed to implement these steps and the specific technique used is not relevant to this discussion. Therefore, these steps are simply mentioned for consistency with existing systems. Once step 314 is executed, processing returns to step 304 to test for additional uncompressed characters in the input data stream.

FIG. 1 depicts the architecture improved data compression system 100 in which repetitions of uncompressed characters are encoded more efficiently and Appendix A attached hereto discloses a program to implement the steps of the data compression and decompression processes. The data compression system 100 is represented as consisting of three subproceses (110, 111 and 112) which communicate through flows of data (102, 103 and 104). These processes may be implemented in hardware, software or a combination of both.

The Replace Runs With Repeat Counts process 110 examines the input data stream that consists of uncompressed characters 101 and produces a stream of characters and repeat counts 102. Repeat counts are produced whenever there are three or more repetitions of a single uncompressed character. In the data produced by the Replace Runs With Repeat Counts process 110, these runs of a single character in the input character stream are replaced with the character followed by a repeat count. The storing of the characters and repeat counts 102 in symbol buffer 121 is not required, but has been done to allow the description of the Compress Data Using Textual Substitution process 111 to be simplified. It should be understood that the choice may be made in implementing the improved data compression system to have the Compress Data Using Textual Substitution process 111 immediately process characters and repeat counts 102 without the information first being first stored in symbol in symbol buffer 121.

The Compress Data Using Textual Substitution process 111 examines the stream of input characters and repeat counts 103 which were previously stored in symbol buffer 121, and replaces sequences of input characters and repeat counts which have been previously encountered and stored in the dictionary 120 with string reference values. It also adds to the dictionary 120 (and possibly deletes from the dictionary 120) strings which consist of previously encountered sequences of input characters and repeat counts. If the Compress Data Using Textual Substitution process 111 encounters a repeat count in input data stream 103, it is encoded with a string reference value if a matching string which contains the repeat count is found in the dictionary 120. If no matching string which contains the repeat count is found, the repeat count is output rather than a string reference value representing a string containing the repeat count which are stored in symbol buffer 121.

The Encode Repeat Counts process 112 produces a compressed data stream 105 from a stream of string reference values and repeat counts 104. As described above, this can be done by replacing each repeat count with a reserved reference value followed by an encoded form of the repeat count. This allows the data decompression process to distinguish repeat counts from string reference values.

FIG. 2 shows data decompression system 200 which decompresses the data compressed by the data compression process 100 in FIG. 1. The Decode Repeat Counts process 210 translates instances of reserved reference values followed by repeat counts in the compressed data 201 into repeat counts. Its output 202 then consists of string reference values and repeat counts.

The Decompress Data Using Textual Substitution process 211 replaces the string reference values in the stream of string reference values and repeat counts 202 with a sequence of characters and repeat counts. The repeat counts in this process's input data stream 202 are not replaced by this process. The dictionary 220 contains the sequence of characters and repeat counts encoded by each of the string reference values. The Decompress Data Using Textual Substitution process 211 also adds to the dictionary 220 (and possibly deletes from the dictionary 220) strings which consist of previously encountered sequences of characters and repeat counts. The same strings are added to the dictionary 220 or deleted from the dictionary 220 that were added or deleted by the Compress Data Using Textual Substitution process 111.

The Replace Repeat Counts With Runs process 212 replaces repeat counts in its input data stream 203 with a number of copies of the character most recently output by this process. The number of repetitions is specified by the repeat count.

The data decompression process 200 reverses each of the modifications performed on the input characters 101 received by the data compression process 100. Thus, the characters 204 produced by the data decompression process 200 are the same as those presented as input characters to the data compression process 100.

The method of operation of the present improved data compression system 100 is illustrated in FIG. 4. At step 401, the string dictionary 120 is initialized. Again, the details of this initialization process are a function of the particular data compression algorithm that is selected. The string dictionary 120 can be initialized to be empty or may contain some or all of the possible single symbol strings or may even contain multiple symbol strings which are expected to be encountered in the input data stream 103 of symbols. Once the string dictionary 120 is initialized, Replace Runs With Repeat Counts process 110 at step 402 loads a symbol buffer 121 with symbols, representing uncompressed characters and repeat counts. Rather than compressing characters, the ensuring steps perform textual substitution data compression of these symbols. Process 110 replaces instances of three or more repetitions of the same character by one symbol which is the character followed by another symbol which is a repeat count indicating how many times the character was repeated. Once the symbol buffer 121 is loaded, a symbol pointer is initialized at step 403 to point to the first symbol in the symbol buffer 121 to be compressed.

The data compression process is implemented by the loop consisting of steps 404-414 which operates until no symbols remain in symbol buffer 121. At step 404, a determination is made whether any more symbols remain to be compressed. If no symbols remain, the data compression process 100 exits at step 315. If the symbol pointer is not beyond the last symbol in symbol buffer 121, processing advances to step 405. The string dictionary 120 is searched at step 405 to find a string which matches the symbols in the symbol buffer 121, starting at the position identified by the symbol pointer. The result of this search is tested at step 406. If a string was found in the string dictionary 120 which matches one or more of the symbols in symbol buffer 121, a string reference value is output at step 407 to identify the matching dictionary string. The symbol pointer is then advanced at step 408 by the number of symbols contained in the matching string.

If, at step 406, no matching string was found in string dictionary 120, then at step 409 the symbol pointed to by the symbol pointer is examined to determine whether it is a character or a repeat count. If the symbol is a character, then a field is output at step 410 which identifies the character. If, at step 409, the symbol is determined to be a repeat count, then a field is output at step 411 which identifies the repeat count. The symbol pointer is then advanced at step 412 by one symbol.

The additional steps 413, 414 are shown to illustrate the step of deleting (413) one or more strings from the string dictionary 120 and then to possibly add (414) one or more strings to the string dictionary 120. There are many different techniques proposed to implement these steps and the specific technique used is not relevant to this discussion. Once step 414 is executed, processing returns to step 404 to test for additional symbols in symbol buffer 121.

It is obvious that step 402 may or may not be included in the data compression loop (steps 404-414) simply as a matter of design choice. The symbol buffer 121 can operate prior to or concurrently with the textual substitution process 111. Additionally, symbol buffer 121 can operate on all or only a segment of the input data stream.

The method of operation of the decompression process is not illustrated in detail herein. This decompression method is the reverse of the compression method and is obvious to one skilled in the art in view of the above description of the improved data compression method.

Compression effectiveness can sometimes be improved by avoiding the definition of strings which start with a repeat count. This produces increased data compression efficiency if the characteristics of the data being compressed are such that strings which start with a repeat count would rarely be referenced. When definition of a string which starts with a repeat count is suppressed and the algorithm used for defining new strings adds the first character of the next string onto the end of the previous string, a further modification to the textual substitution algorithm can be made. Rather than defining a string consisting of a repeat count followed by the next input character, a string can be defined consisting of the entire previously defined string (which ends in a repeat count) followed by the next input character.

Take, for example, the sequence of input characters AAAB, with the dictionary 120 only containing entries for the input characters A and B. Replace Runs With Repeat Counts process 110 represents this sequence as A, Repeat 2, B before textual substitution takes place. Because A, Repeat 2 does not exist in the dictionary 120, a string consisting of A, Repeat 2 is added to the dictionary 120. An unmodified textual substitution algorithm would then search for Repeat 2, B and, because that string is not in the dictionary 120, the string Repeat 2, B would be added to the dictionary 120. However, if definition of strings which start with a repeat count is suppressed, there is no point in searching for a string consisting of Repeat 2, B. In addition, the Repeat 2, B string should not be added to the dictionary 120. Instead, a search for the string A, Repeat 2, B could be performed and, because the string has not yet been defined, the string A, Repeat 2, B is added to the dictionary 120.

In this example, the search for the string A, Repeat 2, B fails because the string A, Repeat 2 was just added to the dictionary 120 and no string which starts with A, Repeat 2 could possibly have been previously defined. Clearly, the search for the string A, Repeat 2, B could have been skipped. However, it is generally simpler to implement the data compression algorithm by always performing a search for a string before defining it.

There are two ways of handling runs which are longer than can be represented by an input character followed by the maximum repeat count. One way of encoding a run of this length is to produce the sequence: Char, Max Repeat Count, Char, Max Repeat Count, . . . The other way is to produce the sequence: Char, Max Repeat Count, Max Repeat Count, . . . The second approach produces fewer entities to be encoded by the textual substitution algorithm. For many textual substitution algorithms, this yields greater compression effectiveness. However, if the textual substitution algorithm has been modified as just described, the first approach allows a long run to be compressed into fewer bits than if the second approach were used.

If this modified textual substitution algorithm were used to compress the sequence Char, Max Repeat Count, Max Repeat Count, . . . then each time a new repeat count was received, a search would be performed for the string just defined followed by the new repeat count. As described before, this search always fails, so the representation of the maximum repeat count is output and the process repeated. The compressed output would then consist of the sequence: Reference Value of Char, Encoded Max Repeat Count, Encoded Max Repeat Count, . . . Even though strings are being defined in the dictionary, no reference to previously defined strings takes place because there are no strings in the dictionary which start with a repeat count and each entity being processed is a repeat count.

The sequence Char, Max Repeat Count, Char, Max Repeat Count, . . . provides the modified textual substitution algorithm the opportunity to reference the strings which were just defined. The first few instances of Char, Max Repeat Count lead to the definition of strings containing repetitions of Char, Max Repeat Count. As subsequent instances of Char, Max Repeat Count are encountered, matching strings are found in the dictionary and reference values are output which encode several repetitions of Char, Max Repeat Count. Thus, the size of the compressed representation of the run is reduced.

The following program listings, set forth in the C language, embody the steps of the present invention when used with a simple form of a textual substitution data compression algorithm. This program can be run on any computer which uses 8 bit bytes and a binary representation of data and which is equipped with a compatible C compiler.

The compression 100 and decompression 200 processes are each implemented as functions to be called in a main program that would manage the input and output data. The compress and decompress functions each refer to constant values defined in the common header file "compress.h". The contents of this header file are shown in the first portion of the following program listing.

The compression 100 and decompression 200 processes must output or input data fields which vary in width. The functions Pack() and UnPack() perform the storing or retrieving of these variable width fields. Many techniques are suitable for packing bit fields into computer memory words and unpacking bit fields from computer memory words and these techniques are well known in the data processing art.

Whenever input characters and repeat counts are stored or transmitted together, it is necessary to be able to distinguish input characters from repeat counts. In these programs, such a distinction is made by storing input characters as positive values and storing repeat counts as negative values. This allows the computer executing the programs to rapidly distinguish between input characters and repeat counts when necessary. In the compress function, the array rll--buffer[] contains data stored in this fashion. This array corresponds to symbol buffer 121 in FIG. 1. In the decompress function, the array stack[] (which is accessed through the push() and pop() macros) also contains data stored in this fashion. This array corresponds to flow 203 in FIG. 2. In a hardware implementation of these processes, characters and repeat counts could both be stored as positive (or unsigned) values and a flag bit used to distinguish a repeat count from a character. ##SPC1## ##SPC2##