Digital Halftoning With Minimum Visual Modulation Patterns

The present invention relates to the field of digital image processing and more particularly to a method for digital halftoning continuous-tone color images or generating of constant color image regions using a data base of minimum visual modulation bit-map patterns.

Digital halftoning for the purpose of generating color images with multiple levels has been practical since the mid 1960's (see, for example, "Picture Generation with a Standard Line Printer" by Perry and Mendelsohn, Comm. of the ACM, Vol. 7, No. 5, pp. 311-313). The two major techniques that are in current use are dithering and error diffusion See Digital Halftoning by Ulichney, MIT Press., Cambridge, Mass., pages 77-79, 127, 239-240. The prime dithering techniques are random dither, clustered-dot dither, and dispersed-dot dither. Random dither was developed first but it is rarely used because it produces the poorest quality image.

The other two dither techniques are used with clustered-dot being by far the most prevalent. They are both based on a threshold screen pattern that is generally a fixed size, e.g. 8.times.8 image pixels, which is compared with the input digital image values. If the input digital value is greater than the screen pattern number, the output is set `on`, i.e. 255 for an 8 bit input image, and if it is less, the output is set to 0. The difference between the two techniques is that the lower threshold values are centered in the clustered-dot screen pattern whereas they are scattered in the dispersed-dot screen pattern. As such the clustered-dot technique will have a central dot that increases in size as the signal level increases and the dispersed-dot technique will have small scattered dots that increase in number as the signal level increases. In both techniques the number of levels that can be represented is equal to the size in pixels of the screen pattern, e.g. an 8.times.8 screen can produce 64 unique levels.

Larger patterns mean more levels but also a reduction in the effective resolution because the ability to transition among levels is at a coarser pitch. At the medium pixel rate of copiers and laser printers, e.g. 300-500 dots/inch, the pattern artifacts are visible for screen patterns larger than 4.times.4, and since 16 levels are inadequate precision for typical continuous-tone color imagery a suboptimal resolution/level tradeoff is necessary. This resolution density level tradeoff is illustrated in FIG. 1. A second drawback of dithering is that the pattern for a given level is based on adding an `on` to the pattern of the previous level. This highly constrains the ability to design patterns that have minimal perceived binary noise or modulation at all levels. Efforts to do this for dispersed-dot dither (see "An Optimum Method for Two-Level Rendition of Continuous-Tone Pictures" by B. E. Bayer, Proc. IEEE Int. Conf. Comm. Conference Record, pp. (26-11)-(25-15)) have been severely limited by the requirements for a single screen pattern and a small screen size. Such considerations have led to an increased interest in error diffusion for medium resolution applications.

Error diffusion is fundamentally different from dither in that there is no fixed screen pattern. Instead a recursive algorithm is used that attempts to casually correct errors made by representing the continuous input signal by binary values. The two major components are a matrix of fractions that weight past errors and a threshold operator based on the sum of those weighted errors and the current pixel that determines whether to output an `on` or an `off`. The best error diffusion techniques are two-dimensional, meaning that the error is fed back from previous lines as well as previous pixels. The error feedback mechanism is usually linear in that the sum error is a weighted version of past errors, but the thresholding is highly non-linear making the compound process non-linear. Approximating thresholding as signal-dependent gain, it can be shown that for positive error weights that the output binary signal will be high-pass in uniform regions thus introducing `blue noise` into the image (see Ulichney cited above). This `blue noise` spectrum is shown in FIG. 2. As discussed by Ulichney this `blue noise` is a very favorable feature of error diffusion because the perception of this noise will be reduced by the low-pass filtering of the visual system causing a higher perceived signal-to-noise ratio. Unfortunately the error weights are indirectly related to this preferred noise characteristic and therefore provide suboptimal control, and for certain signal levels the causal feedback can become visually unstable generating correlated patterns or `worms` that are highly objectionable. The most common solution is to randomly modulate the weights which reduces the `worms` but also increases the noise. It is the object of the present invention to provide a new digital halftoning technique that improves the image quality over the aforementioned techniques for all classes of images of interest to a human observer including constant signal levels such as those generated by computer graphics.

The objects of the present invention are achieved by designing and storing large, e.g. 32.times.32 pixels, bit-map patterns for each density level in an image, that have minimum visual noise and modularly addressing those patterns with each pixel value in the image. In a preferred mode of practicing the invention, the bit-map patterns are produced using a combinatorial minimization algorithm called stochastic annealing and a human visual system MTF weighting function to weight the discrete Fourier Transform of each pattern to determine the perceived noise. Each pattern corresponding to an input signal level is produced independently and stored as part of a data base. In a process suitable for computer generated graphics having regions of uniform tone, for each pixel value in a digital image signal, a portion of the corresponding binary bit pattern is modularly selected to form the halftone image.

For a digital image produced by scanning a continuous tone image the process includes the following steps. For a given pixel value of the digital image signal, modularly select a future portion of an N.times.N neighborhood of binary values from the corresponding bit pattern. Combine the future neighborhood of selected binary values with the past portion of the N.times.N neighborhood of previously selected binary bits to form a combined neighborhood of binary bits. Apply a visual blur function to the combined neighborhood of binary bits to produce a perceived output value. Repeat the previous steps for a plurality of neighboring pixel values to produce a plurality of perceived values. Compute the different between the respective perceived values and the given pixel value to form difference values. Replace the given pixel value with the binary value from the bit pattern that has the minimum difference value to produce the halftone image.

As used herein, the term modularly select means to address a portion of a bit pattern with the lower significant bits of an x,y pixel address. The effect of modularly addressing the bit patterns is that of tiling the image with each of the bit patterns, and selecting the portion of each bit pattern that overlaps the pixel being processed. For a system with an input pixel spacing that is larger than the output pixel spacing by an integer factor M, the halftoning steps are equivalent except that an M.times.M bit-map pattern is outputted for each input pixel.

FIG. 1 is a graph illustrating the resolution versus number of levels tradeoff of conventional fixed screen dither;

FIG. 2 is a graph showing the "blue noise" spectrum of the high frequency noise that results from conventional error diffusion;

FIG. 3 is a graph showing the human visual system transfer function used to weight the discrete Fourier Transform of the bit-map pattern in the stochastic annealing process;

FIG. 4a is an example of an initial random bit-map pattern for a mean level of 1/8th or 128 `on` dots;

FIG. 4b is an example of a final minimum cost bit-map pattern for a mean level of 1/8th or 128 `on` dots;

FIG. 5 is a graph showing the relative visual error cost at each T value for the annealing process that generates the pattern in 6b;

FIG. 6 is a block diagram showing the halftone image processing technique according to the present invention for constant tone graphics;

FIG. 7 is a diagram showing the addressing of the data base as the input image is processed;

FIG. 8 is a block diagram showing the halftone image processing technique for continuous tone images;

FIG. 9 is a sketch illustrating the combination of past binary values and future bit-map patterns to determine the pattern from which the current binary value is to be selected;

FIG. 10 shows an unnormalized 5.times.5 human visual system blur matrix used for weighting the bit-maps to select the data base that provides minimum local visual error;

FIG. 11 is a flow chart illustrating the logic flow for the halftoning technique according to the present invention;

FIG. 12 is a sketch of the block overlap and average pixel and line regions used in calculating horizontal and vertical slopes for shifting `on` dots to account for mean shifts in input image blocks;

FIG. 13 is a graph showing the one-dimensional linear mapping of dot spacings to shift local output means across image blocks; and

FIG. 14 is a graph showing the one-dimensional quadratic mapping of dot positions to shift local output means across image blocks.

The method of generating a halftone image according to the present invention involves designing minimum visual modulation bit-map patterns for each image signal density level, selecting the pattern corresponding to a pixel density level, and modularly addressing the pattern to select bits to be used in the halftone image. The bit-map pattern design will be described first, followed by a description of the addressing and pattern selection processes.

The goal of the bit-map pattern design is to minimize the visually perceived modulation for a pattern of zeroes and ones that on average represent a particular signal level. This requires the definition of visual modulation or cost for a given pattern and a method for minimizing the cost. For normal viewing distances, i.e. 10 inches, the effective two-dimensional modulation transfer function or MTF of the human visual system is shown in FIG. 3. It shows the low-pass filter characteristic of the vision system and the loss of sensitivity at 45 degrees corresponding to the dip in the MTF surface at that angle.

This function is given by: ##EQU1## where the constants a,b,c, and d can be calculated by regression fits of horizontal and vertical modulation data to be 2.2, 0.192, 0.114, and 1.1 respectively. f.sub.v.sbsb.ij is the radial spatial frequency in cycles per degree of visual subtense, scaled for a given viewing distance, and f.sub.max is the frequency in cycles per degree at which the function V.sub.ij peaks. f.sub.v.sbsb.ij is calculated as:

where ##EQU2## where dis=viewing distance in mm ##EQU3##

To evaluate the cost for a particular 32.times.32 pattern, the modulus of the discrete Fourier Transform (DFT) of the pattern is calculated and multiplied by the visual MTF and summed. That is ##EQU4## where i,j are x,y spatial frequency indices with i,j=32 representing the output binary pixel rate, V.sub.ij is the visual MTF scaled to the output pixel rate, P.sub.ij is the pattern DFT, and * represents complex conjugation. The Fourier domain was chosen for the minimization because the DFT assumes that the input is periodically replicated in a tile fashion over all space which is consistent with the modular addressing of the pattern to be described later. A 32.times.32 region was chosen to allow for enough degrees of freedom in the Fourier domain to shape the noise into a `blue noise` spectrum as in FIG. 2 without the excessive complexity of larger areas, and because it easily accommodates the 256 levels on an 8 bit input signal with each level corresponding to four `on` dots.

The number of ways to turn p dots `on` out of 1024 is given by ##EQU5## which for all but a few high and low values of n is so large that a brute force search for the minimum cost pattern is not feasible. Resort was therefore made to a probabilistic minimization technique called stochastic annealing or stochastic relaxation (see for example, "Stochastic Relaxation, Gibbs Distributions, and Bayesian Restoration of Images" by S. Geman and D. Geman, IEEE Trans. on PAMI, Vol PAMI-6, No. 6, Nov. 1984, pp. 721-741). The key feature of this iterative technique as compared to more aggressive minimization techniques such as steepest descent (see any text on numerical analysis) is that it can escape from a local minimum by accepting cost increases at each iteration with a decreasing but greater than zero probability. The process includes the following steps:

1. Initialize a given density level pattern with random positions for `on` dots, where p is a number that is proportional to the given density;

2. Calculate the cost using equation (4);

3. Randomly switch a pair of the on/off dots and calculate a new cost using equation (4);

4. Calculate a test statistic q for determining if the new pattern is from a lower cost random process given by; ##EQU6## where .DELTA.cost=new cost-previous cost, and T is a normalization parameter set initially such that a large percentage, e.g. 80%, of new patterns will be judged to be from a lower cost random process even if the .DELTA.cost is greater than 0;

5. If q>1, i.e. .DELTA.cost<0, accept the new pattern. If q.ltoreq.1, accept the new pattern if .epsilon..ltoreq.q, where .epsilon. is a uniform random number between 0 and 1, and reject the new pattern and revert to the previous pattern if .epsilon.>q;

6. After many loops of steps 3-5, e.g. 1500, decrease T to kT where k<1, e.g. 0.95, and go to step 3 so that positive .DELTA.cost values are accepted with a geometrically decreasing likelihood as the search proceeds;

7. Stop when the cost at successive decreases in T are no longer changing or when a fixed number of decreases have occurred, e.g. 300.

A computer program, written in the fortran program language, for performing the above steps is included in Appendix A. This program was executed on a CRAYII.TM. super computer to produce a set of 256 minimum visual noise binary bit patterns, corresponding to 256 density levels.

FIGS. 4a and 4b show the `on` pixel positions of the initial random pattern and the final minimum cost pattern for a mean value of 1/8th or 128 `on` dots that resulted from a single run of this process. As the figures show, the final pattern appears random but in fact is more spatially regularized without large gaps or clumps which lead to undesirable low frequency modulation. FIG. 5 shows the progression of cost estimates at each decrease in T with each temporary dip in the curve being a local minimum from which the minimization process escaped. In addition, empirical results indicate that multiple runs produce slightly different final patterns due to the inherent randomness of the search process and that better results can be achieved by selecting the pattern with the lowest final cost from a small number of independent searches, e.g. 10.

Referring now to FIG. 6, the method of generating a halftone image according to one mode of the present invention is illustrated schematically. A digital monochrome image composed of regions of constant tone is generated by an input device 10 such as a personal computer programmed to generate graphics. The digital image is supplied as 8 bit pixel values representing one of 256 density levels. The x,y location of each pixel on a page is identified by two 16-bit words. Two hundred fifty-six 32.times.32 bit halftone dot patterns (12) generated on a super computer as described above are stored in a bit pattern memory 14. As illustrated schematically in FIG. 7, the 8 bit density level for a pixel 16 from the image 18 is employed to select the corresponding halftone dot pattern. The 5 least significant bits of the x and y pixel addresses are employed to modularly address the selected 32.times.32 bit halftone dot pattern and produce a halftone bit pattern (e.g. 1 bit if the resolution of the input image and the output image are the same) as an output. Referring to FIG. 6, the halftone bit pattern is supplied to a bit map page memory 20. The full 16 bit x,y pixel address is employed to address the page memory, and the halftone bit pattern is stored in the page memory 20 at the indicated pixel location.

When the page memory is full, the contents are supplied to a medium resolution (e.g. 300-500 dots/inch) binary marking engine 22, such as a laser or ink jet printer. Alternatively, if the input is synchronized with the marking engine, the output of the bit pattern memory may be supplied directly to the marking engine without the need for a bit map page memory 20.

The halftone dot pattern generation steps of the halftone image generation process described with reference to FIG. 6 is implemented simply by a programmable read only memory in which the halftone bit patterns are stored.

If the resolution of the marking engine 18 is higher than the resolution of the image supplied by the input device 10, multi-bit portions of the halftone bit maps are supplied to the marking engine for each input pixel.

The images produced by the above described technique for computer generated monochrome images at 400 dots per/inch composed of regions of constant tone show a significant improvement in perceived noise compared to dither and error diffusion techniques.

When the above described method is applied to continuous tone digital images, such as those produced by scanning and digitizing a photograph, there is a perceived increase in noise due to the frequent switching between the halftone bit patterns in areas of gradually changing contrast. The noise was reduced for continuous tone images by a technique of monitoring the visibility of the noise introduced in the halftone image due to switching from one bit pattern to another for adjacent pixels. If the noise introduced by switching between halftone patterns is more than that which would be introduced by selecting a pattern corresponding to the density of the pixel, the pattern for the adjacent pixel is chosen. This improved technique for generating a halftone image from a continuous tone digital image will now be described with reference to FIG. 8.

A continuous tone image 30 such as a photographic image is scanned by scanning apparatus 32, such as a video or graphic arts scanner. The scanner samples the image density at each sampling location (pixel) and digitizes the density value to 8 bits, or 256 levels. The x,y position of each pixel is supplied from the scanner as two 16-bit words. The 8 bit density for each pixel is employed to select a particular bit pattern in a halftone bit pattern memory 34 having 256 bit patterns generated as described previously. The 5 least significant bits of the x and y pixel addresses are employed to modularly address a portion of the selected halftone bit pattern. The portion of the bit pattern addressed is the binary value indicated by the address, the next two binary values in the row, and 5 binary values in each of two successive rows. These thirteen binary values represent the "future portion" in a 5.times.5 block of binary values centered on the location indicated by the address. FIG. 9 illustrates a 5.times.5 block (35) of pixels 36, where the current address location is indicated by an X, and the "future portion" 37 is indicated by cross hatching. In addition to the thirteen binary values from the selected pattern corresponding to the present density pixel value, 13 binary values are also selected from the patterns corresponding to the density of the previous pixel in the current line of pixels and the density of the pixel in the line immediately above the current pixel. The modular address employed for selecting these values is the same as the current pixel, but the density values are provided by a 1 line delay 40, and a one pixel delay 42. Each of these three patterns of 13 binary values is combined with 12 binary values representing a "past portion" 38 of the 5.times.5 block of binary values (see FIG. 9) to generate 5.times.5 blocks of binary values 44. The 12 binary values representing the "past portion" of the 5.times.5 blocks are supplied from a bit map page memory 46 (described below). Each of the 5.times.5 blocks of binary values is processed (48) by applying the 5.times.5 human visual system (HVS) blur filter. The coefficients of the HVS blur filter are shown in FIG. 10, and correspond to the values of the HVS function plotted in FIG. 3. To process the 5.times.5 blocks, the coefficients of the blur filter are multiplied by the corresponding binary values in the 5.times.5 blocks, and the results for each block are summed and normalized to the maximum density value. The absolute value of the nomalized sum of the blurred values from each of the three blocks is subtracted from the actual density value of the present pixel to determine the blur error. The minimum blur error is detected, and the binary bit from the pattern producing the minimum blur error is selected 50. This selected bit is the bit that produces the least visual error in the halftone pattern when the density level changes from one pixel to its nearest neighbor directly above or to the left. The selected bit is stored in the bit map page memory 46 at the location indicated by the current x,y address. When the page memory 46 is filled, the halftone dot pattern is sent to the marking engine 52.

The halftone image generation process described in FIG. 8 is implemented by a programmed digital computer such as a microprocessor having a read only memory (ROM) for storing the halftone bit patterns or alternatively by a custom designed digital signal processing circuit for increased speed. The steps for programming a microprocessor to generate a halftone image from either a constant tone image (as described with reference to FIG. 6) or a continuous tone image (as described with reference to FIG. 8) will now be described with reference to FIG. 11.

The following description assumes that the digital input image signal is presented to the halftoning apparatus in a linear raster scan. At the start of scan, the internal counter j for the line number and i for the pixel number are initialized as well as the non-image (border), i.e., (n<1, m<1), input and output boundaries S.sub.nm and R.sub.nm (100, 104). The counters i and j are incremented by 1 (102, 106). The 5 least significant bits 1,k of i,j are isolated by a modulo-32 operation (108) and passed as a position pointer into the data base. A check is made for a constant tone image (110). If yes, the number of data base accesses s is set of 1 (112) and the current pixel value is used to address the appropriate bit-map in the data base (114). If no, the number of data base accesses s is set to 3 (116) and the current pixel as well as its two nearest neighbors are used to address the appropriate bit-maps in the data base (118). For each data base access the binary values in the bit-maps from 1-N/2.fwdarw.1+N/2 and k.fwdarw.k+N/2 excluding 1-N/2.fwdarw.1-1 for line k are extracted from the data base (120). The past outputted binary values for i-N/2.fwdarw.i+N/2 and j.fwdarw.j-N/2 excluding i.fwdarw.i+N/2 for line i are merged with the extracted binary values for each data base access (122). The N.times.N visual blur coefficients V(u) (124) are used to weight the merged binary fields to calculate the error for each data base access (126). The minimum error e.sub.min is calculated (128) and compared with the error for each data base access to determine which bit-map produced the minimum error (130). The binary value at 1,k from that bit-map is outputted at pixel i,j (132). A check is made for the end of line (134) and end of image (136). If not the end, the appropriate counters are reset or incremented and the outputted value is stored (138), else the output image is complete and the halftoning stops (140).

An assumption in the halftoning technique just described for continuous tone scanned image data is that data base switching due to input signal variations do not significantly affect the preferred high frequency character of the bit-maps. Simulated images show that this is only partially true. Input scan noise and small pixel-to-pixel variations in low contrast image areas can cause low frequency gaps or dot-clumps at switching boundaries that were purposely suppressed in each bit-map. An alternative solution is to configure the switching mechanism to take into account local variations in the input slope. This can be accomplished on a block-by-block basis by estimating the horizontal and vertical slopes of an input block with the difference between the average pixel values and average line values at the beginning and end of the block. To avoid the appearance of visible block artifacts due to integer precision it is necessary to overlap successive blocks by one line and one column in this calculation. This block overlap is shown schematically in FIG. 12. The average value of the block is then used as the data base address and the slope of the pixel value in the blocks are used to shift the `on` dots in the data base so that the average distance between dots across the block correspond to the calculated slopes. This is accomplished by a quadratic mapping of integer coordinates that linearly adjust the dot placements. This is illustrated for one dimension in FIGS. 13 and 14 with the appropriate equations given by ##EQU7## where .mu.o and .mu.b are the average values at the beginning and end of a block of length b, .mu. is the average of the entire block, and the input positions x are the centers of each integer bin in the block starting from 0.5. The expected value of .delta.x is 1/.mu. so that the expected value of .delta.x' at X=O and X=b are 1/.mu..sub.o and 1/.mu..sub.b respectively. Since the spatial precision in each direction is one sample on input and output, it is necessary to approximate the above equations by selecting an integer number of dots d to move an integer number of samples. For a shift of one sample spacing this is ##EQU8## where D is the number of `on` dots in each column or line in the block, and Rnd means round to the nearest integer. For example, if the block is 16.times.16 and x' and D for the first column are 0.75 and 7 then d=Rnd((0.75-0.5)7)=Rnd(1.75)=2 meaning that two dots must be moved to the next column to appropriately decrease the mean value at the beginning of the block. The two dots to be moved can be selected randomly or by a rule such as move the dots with the most unfilled space in the direction of movement making sure that one pixel is never occupied by more than one dot. There will also be an error after each move given by E=x'-x-d/D which in the above example is -0.0357 meaning that more than the exact fractional number of dots were moved to the next column. This error can be ignored or propagated to the next column by adding it to the next x'. Since the column and line means are independent the columns can be shifted first followed by the lines. An additional improvement is to have a variable block size. This can be advantageous in regions that have a non-linear transition across a block such as a sharp edge. If the average block value given by .mu. is greater than a small central average by more than a threshold the block can be decomposed into four blocks of one-fourth the size with each sub-block processed independently.

The halftoning technique may be applied to a color image having a plurality of color components, by performing the halftoning process independently on each component.

The halftoning technique of the present invention is useful in systems that generate binary outputs from continuous tone input images or computer generated graphics. The invention has the advantage that it is optimal in terms of perceived noise in constant signal regions and does not have the periodic patterns of dithering or correlated directional noise of error diffusion. The method has implementation advantages over error diffusion because it does not require real multiplications. This is because the choice of data base address is based on a binary convolution which is merely an addition of the coefficients corresponding to `on` pixels, whereas error diffusion involves weighting the differences between input and output values which are not binary. In addition, the halftoning technique of the present invention requires less storage than error diffusion because only the single bit output image is stored as multiple lines, whereas in high-quality error diffusion (see, for example, the Stucki matrix in Digital Halftoning by Ulichney, MIT Press Cambridge, Mass.) multiple lines of the full-precision input image must be stored. ##SPC1##