Collaborative Judgement in Statistical Coding

The present invention relates to the field of statistical coding; more particularly, the present invention relates to combining multiple probability estimates into a collaborative judgment for coding information.

Statistical coding is an example of a coding technique that is a useful tool for storing and transmitting large amounts of data. For example, the time required to transmit an image, such as a facsimile transmission of a document, is reduced when compression is used to decrease the number of bits required to recreate the image.

In statistical coding, one develops a probability estimate from conditioning data. Typically, the set of all coding decisions is partitioned based on the conditioning data and statistics are developed for each such partition member, or context. For example, coding decisions might be conditioned on neighboring pixels. An example of such a scheme is shown in FIG. 1A and 1B. Referring to FIG. 1A, 10 prior pixels are used to condition pixel 101 (identified by the "?"). If each pixel has two possible values, there are 1024 different combinations that are possible for the 10 pixels. Thus, there are 1024 different contexts that can be used to condition the coding of pixel 101. Referring to FIG. 1B, a lookup table (LUT) 102 contains a probability estimate of how likely a pixel is to be in its most probable state (or least probable state) for each possible combination of values for the 10 pixels. Therefore, LUT 102 contains 1024 context bins, each context bin having a probability estimate associated with it. LUT 102 receives the 10 prior pixels at its input 103 and outputs probability estimate 104 in response thereto.

Two or more partitionings may be made to obtain two or more probability estimates for the same pixel. For example, one estimate might correspond to pixels neighboring towards the left of the pixel being conditioned and another estimate might correspond to pixels neighboring above the pixel being conditioned. The two estimates are combined into a single probability estimate (i.e., a collaborative judgment). FIG. 2 is a block diagram illustrating such a system.

Referring to FIG. 2, two probability estimates are conditioned on two different groups of 8 pixels. One of the probability estimates, P.sub.1, indicating the probability that pixel 200 is in its most probable state, is output from LUT 201 based on the values of the 8 pixels shown in pixel group 203. The other probability estimate, P.sub.2, indicating the probability that pixel 200 is in its most probable state, is output from LUT 202 based on the values of the 8 pixels shown in pixel group 204. Because pixel groups 203 and 204 have 5 common pixels, 5 of the inputs to LUTs 201 and 202 are the same and are shared. The two probability estimates are combined by combining block 205 to generate a collaborative probability estimate 206.

The main advantage of using a collaborative judgment is that smaller conditioning partitions can be used. The use of smaller conditioning partitions implies faster adaptation because contexts are not "diluted" and the model is not "overfitted" with too many contexts. Smaller partitions also imply less hardware to store the probability estimates. In other words, the memory for storing the statistics and context (i.e., the context memory) may be smaller.

In the case where two probability estimates are being combined, one might assume that an average of the two probability estimates would achieve the most accurate probability estimate. However, this simple combining technique is often only applicable when the conditioning upon which one of probability estimates is based is strongly related to the conditioning upon which the other probability estimate is based.

In the prior art, estimates are combined via the geometric mean of the odds and the result is exaggerated if there is a consensus. See Kris Popat and Rosalind W. Picard, "Exaggerated Consensus in Lossless Image Compression," In Proceedings of IEEE Int'l Conf of Image Processing, Vol. III, pgs. 846-850, November 1994. That is, when the probability estimates are both greater than 50% (i.e., there is a consensus) and the resulting combined probability estimate is greater than either of the two component probability estimates, the resulting probability estimate is said to be exaggerated.

In Popat-Picard, the collaborative judgment, referred to as P.sub.NET, is calculated based on raw probability, referred to as Q, according to the following:

where A equals:

P.sub.NET is generated according to the following equation:

where P.sub.1 and P.sub.2 are the two probability estimates and B measures the divergence between probability estimates P.sub.1 and P.sub.2. The value Y(B) is an exaggeration exponent implemented with a table-lookup developed from training images. The function mapping consensus to an exaggeration exponent is set by training using an ensemble of test data.

A method of generating a collaborative judgment for use in coding a symbol is described. One embodiment of the method comprises generating multiple probability estimates for a symbol to be coded, selecting an exaggeration exponent and weighting information indicative of weighting between the probability estimates, and combining the probability estimates into the collaborative probability estimate based on an exaggeration function using the exaggeration exponent and the weighting information.

The present invention will be understood more fully from the detailed description given below and from the accompanying drawings of various embodiments of the invention, which, however, should not be taken to limit the invention to the specific embodiments, but are for explanation and understanding only.

FIGS. 1A and 1B illustrate a prior art partitioning scheme.

FIG. 2 is a block diagram of a system in which two context models are combined.

FIG. 3 is a flow diagram of one embodiment of a process of creating a collaborative probability estimate.

FIG. 4 illustrates a two-dimensional map having individual regions associated with different pairs of probability estimates.

FIG. 5 is a block diagram of one embodiment of a coding system.

A statistical coding scheme using a collaboration judgment is described. In the following description, numerous details are set forth, such as, for example, numbers of probability estimates, contexts, bits and memories, etc. It will be apparent, however, to one skilled in the art, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.

Some portions of the detailed descriptions described below are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" or the like, may refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

Also as discussed below, the present invention also relates to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMS, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. The algorithms presented herein are not inherently related to any particular computer or other apparatus. Various general purpose machines may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these machines will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.

Overview

A method and apparatus for generating a collaborative judgment for use in coding a symbol is described. The symbol may comprise a bit, decision, token or other datum that is to undergo coding.

A In one embodiment, a method for generating a collaborative judgment includes generating multiple probability estimates for each symbol that is to be coded. These probability estimates may represent low-order conditional probability distributions. These probability estimates are combined to create a single higher-order estimate, which is based on their agreement (i.e., their consensus). For instance, if graphs of two probability distributions have the same shape, the combination of individual probability estimates may result in an exaggeration of the shape. Thus, a collaborative judgment is created by combining multiple probability estimates where the combination results in an exaggeration of the probability estimates of any individual one of the probability estimates.

In one embodiment, the collaboration judgment is the result of applying a function to probability estimates. This function uses an exaggeration exponent and weighting information which are selected based on the probability estimates. The weighting information is indicative of the weighting between the probability estimates. In one embodiment, the weighting information may comprise an efficiency weight that may indicate the extent to which each of the probability estimates influences the collaborative probability estimate. In another embodiment, the weighting information may comprise a set of weights, each weight associated with one of the probability estimates. The exaggeration exponent indirectly measures the degree of independence among the conditioning data used by the separate probability estimates.

The probability estimates are then combined into a collaborative probability estimate based on the exaggeration function using the exaggeration exponent and the weighting information.

FIG. 3 illustrates one embodiment of the process for combining multiple probability estimates into a collaborative probability estimate. The process set forth in FIG. 3 calculates the collaborative probability estimate using a function (e.g., an equation) that includes an exaggeration exponent and an efficiency weight.

Processing logic may perform the combining process. The processing logic may comprise software such as, for example, that which runs on a general purpose or dedicated processing device or computer system, hardware such as, for example, dedicated logic or circuitry, or a combination of both.

Referring to FIG. 3, the process begins with processing logic generating multiple probability estimates for each symbol to be coded (processing block 301). In the case where the symbol comprises a decision, each probability estimate may be generated in response to a context. That is, data being coded is input into multiple context models. Each context model translates the input data into a set or sequence of binary decisions and provides a context bin for each decision. Both the sequence of binary decisions and their associated context bins are output to a probability estimation module, which generates a probability estimate for each binary decision in a manner well known in the art.

In the present invention, the context models are different. In one embodiment, neighboring pixels are used as the context to condition the coding of a decision. A predetermined number of pixels may be used. For instance, in one embodiment, ten neighboring pixels are used as one context, while other contexts may be formed by using different groups or numbers of neighboring pixels. In one embodiment, when an individual bit of a pixel is being coded, only those bits in the same bit position within the neighboring pixels forming the context are used to condition that individual bit. Note that in other embodiments, other bits within one or more of the pixels may form the context for a particular bit (e.g., decision).

Other context models may be employed to generate contexts that result in a probability estimate being created when input to a probability estimation machine.

In one embodiment, two probability estimates are generated, one for each context. These are referred to herein as probability estimates P.sub.1 and P.sub.2. The number of probability estimates generated may instead be three or more.

After the probability estimates have been generated, processing logic classifies them into a region R (processing block 302). FIG. 4 illustrates one embodiment of a two-dimensional (2-D) consensus map 400 having regions I-VIII. One of the regions in map 400 is selected based on the values of probability estimates P.sub.1 and P.sub.2. Each region is associated with two parameters, an efficiency weight, W.sub.R, and an exaggeration exponent, Y.sub.R. The efficiency weight is indicative of the weight given to each of the probability estimations with respect to each other. In one embodiment, the probability estimations are given different weights. In the case where only two probability estimates are being combined, a single weight value can be used to define which of the probability estimates is to be given a greater weight in the determining what the collaborative probability estimate should be. Note that if more than two probability estimates are being combined, additional weights W may be necessary to indicate the different weights for each probability estimation. In one embodiment, W.sub.R is initially 0.5 and Y.sub.R is initially 1.0 and these parameters are updateable.

The selection of a region R based on the probability estimates results in the values for W.sub.R and Y.sub.R being output by the processing logic (processing block 303). In one embodiment, the 2-D consensus map comprises a lookup table (LUT) that stores W.sub.R and Y.sub.R values for each region.

After classifying the two probability estimates to obtain the W.sub.R and Y.sub.R values, processing logic calculates the collaborative judgment P.sub.NET In one embodiment, P.sub.NET is calculated according to the following equations (processing block 304). For brevity of exposition assume that the coded decision was true. If not, replace each P with (1-P) et cetera:

Note that this combination of probability estimates is based on an arithmetic mean probability, instead of the geometric mean odds.

After calculating the collaborative judgment, processing logic updates parameters W.sub.R, Y.sub.R (processing block 305). The updates may be performed for each new symbol (e.g., decision, etc.) that is coded. Thus, the updates may occur in real-time (e.g., on-line).

In one embodiment, since the cost of encoding a true decision of probability P.sub.NET is -log P.sub.NET, processing logic updates these parameters using gradient descent update rules. The gradient descent update rules may update according to the following equations: ##EQU1## where .delta..sub.w, .delta..sub.y are adaption rates. The value for each adaption rate which is likely to give best compression performance is likely to be near 0.01 in most applications or near 0.05 when data sets are small or highly non-stationary. Thus, the adaption rates may be selected based on desired performance or the data itself. It would be readily apparent to one skilled in the art as to how to experiment with various values to achieve good performance.

If Y.sub.R =1, the former becomes ##EQU2## Thus, the W.sub.R and Y.sub.R values are set adaptively in real-time, region-by-region. Likewise, this implies that the W.sub.R and Y.sub.R values are learned in real-time as well.

Note that the W.sub.R and Y.sub.R values for each region may be set off-line. The offline update of the W.sub.R and Y.sub.R values may be performed by partitioning the region of possible combinations of probability estimates and using one or more test images to identify the combination of W.sub.R and Y.sub.R values that results in the lowest average bit rate. Once that combination of values is identified, those values are assigned to the region.

For coding, the combined probability estimate may be used by a coder to code information. In one embodiment, the coder may include a bit stream generator that receives the collaborative probability estimate along with the determination of whether a binary decision (result) is or is not in its most probable state (i.e., whether the decision corresponds to the most probable symbol (MPS). The bit stream generator receives the probability estimate and the determination of whether or not the binary decision was likely as inputs and produces a compressed data stream, outputting zero or more bits, to represent the original input data. The coder may also generate code state data which may be used to code subsequent data.

During decoding, the same processes of generating a collaborative probability estimate may be used. That is, multiple probability estimates will be generated based on multiple contexts and combined in the same manner as described above. Intone embodiment, a coder such as one with a bit stream generator returns a bit representing whether a binary decision (i.e., the event) is in its most probable state. The combined probability estimate is generated based on the received bit and returns a result to the context model. The context model receives the result and uses it to generate the original data and update the context bin for the next binary decision.

The process of generating the collaborative judgment may be useful in lossless image compression as well other image processing applications, such as lossy compression, restoration, segmentation, and classification. Lossy coding involves coding that results in the loss of information, such that there is no guarantee of perfect reconstruction of the original data. In lossless compression, all the information is retained and the data is compressed in a manner which allows for perfect reconstruction. Other potential areas of application include multi-sensor integration and decision-making based on accumulated evidence.

Note that other techniques to combine multiple probability estimates may use some or all parts of the process described herein.

FIG. 5 is a block diagram of one embodiment of a coding system having a collaborative probability estimate generator. Referring to FIG. 5, a context model 501 generates multiple contexts, context.sub.1 -context.sub.N, convert each input data bit into a decision. As discussed above, the contexts may be two different groups of pixels. Each context is used to address one or more memories (e.g., lookup tables), such as context memories 502.sub.1 -502.sub.N, containing probability estimates for each context bin. In response to the contexts, memories 502.sub.1 -502.sub.N output probability estimates.

In one embodiment, the two probability estimates are conditioned on two different groups of 8 pixels. One of the probability estimates, P.sub.1, indicating the probability that a pixel is in its most probable state, is output from a LUT based on the values of 8 pixels. The other probability estimate, P.sub.2, indicating the probability that a pixel is in its most probable state, is output from the LUT based on the values of 8 different pixels. If the two pixel groups have common pixels, some of the inputs to the LUTs may be the same.

Collaborative probability estimate generator 503 combines the two probability estimates to generate a collaborative probability estimate 506.

In one embodiment, probability estimate generator 503 comprises a region classifier 510 that receives the probability estimates and produces region identifier 511. A 2-D consensus map (e.g., LUT) 512 receives region identifier 511 and outputs the W.sub.R and Y.sub.R values. Note that in other embodiments, the probability estimates may be used as inputs to the 2-D consensus map directly. A probability estimate combiner 513 uses the W.sub.R and Y.sub.R values to combine the probability estimates as described above and outputs the result P.sub.NET. The P.sub.NET value is also sent to update module 506, which updates W.sub.R and Y.sub.R values for the region, such as described above.

Coder 504 receives the P.sub.NET and codes the data using it. The data may be decisions or indications of whether or not the binary decision was likely as inputs. The output of coder is a compressed data stream, outputting output bits, to represent the original input data. The coder may employ any coding technique that is well-known in the art (e.g., statistical coding, etc.). The coder may generate zero or more output bits along with code state data (not shown), which may be used for coding subsequent data.

The blocks (or units) in FIG. 5 may be implemented in software, hardware, or a combination of both as described above.

Whereas many alterations and modifications of the present invention will no doubt become apparent to a person of ordinary skill in the art after having read the foregoing description, it is to be understood that any particular embodiment shown and described by way of illustration is in no way intended to be considered limiting. Therefore, references to details of various embodiments are not intended to limit the scope of the claims which in themselves recite only those features regarded as essential to the invention.

Thus, a method of compression/decompression has been described.