Method and apparatus for distinguishing between noisy continuous tone document types and other document types to maintain reliable image segmentation

[0002] The present invention relates to the document processing system art. It finds particular application in conjunction with image segmentation, and more particularly, a method and apparatus for distinguishing between noisy continuous tone document types and other document types identified by measures and techniques that have undesirable positive responses to video representing noisy continuous tone images, in order to maintain reliable image segmentation. However, it should be appreciated that the present invention may also find application in conjunction with other types of digital imaging systems and applications that discern how an image was rendered.

[0003] Document processing systems often need to take an image representation with a low level of organizational structure, such as a simple bit or byte per pixel rectangular video map, and infer structure to guide further processing or analysis of the image. For example, in a reprographic system a document is optically scanned and the reflectance of each pixel is converted to a multi-bit (e.g. 8-bit) digital value, forming a matrix of integers representing the original document.

[0004] Various image processing is then performed to enhance the image and convert it to a different resolution and form which can be rendered by a particular marking technology, such as a bi-level Xerographic printer. The purpose of the processing is to produce an output copy which appears to the human eye to be equal to or in some sense better than the original. Unfortunately, the appropriate image processing is different for different types of input documents, typical broad classes being text, continuous tone, and halftones of various techniques and frequencies.

[0005] Furthermore, many user originals contain more than one document type in different areas of the page, for example a magazine article with inset halftoned photographs, or text embedded in a tinted background. This explains the need to classify each document pixel, so that appropriate processing can be performed. In other applications, the pixel classification can be communicated to a subsequent process that analyzes the pixel level classification map further.

[0006] Algorithms for classifying image pixels according to document type (e.g. text, contone, halftone) make their decision based on image context in the vicinity of the pixel being classified. In the case of a halftone document type, the absolute value of the sum of the laplacians in a small area surrounding the pixel being classified (abbreviated Sij), and the halftone peak count within a context surrounding the pixel being classified, are typically used to detect halftones. However, continuous tone documents which are very noisy, or video from very noisy scanners, can fool such algorithms if enough false video peaks are detected and Sij is large enough. This results in the contone image appearing splotchy on the output document. The splotchy appearance is due to the inappropriate switching between rendering schemes based on varying peak density and Sij.

[0007] In the past these defects were not visible because the use of hysteresis or other state-based methods, along with rectangular segmentation blocks over which statistics were accumulated suppressed such errors, as did rendering TRCs designed to hide shortfalls not only in segmentation, but also in other parts of the system as well, such as the scanner.

[0008] However, recent advances in scanner technology and image segmentation, driven by the desire to more faithfully reproduce the original image, have eliminated the state-based methods and the use of rectangular segmentation blocks. TRCs no longer saturate in the dark and light areas to hide flaws in the system, but are designed to faithfully render the original document at all densities. The new segmentation schemes avoid generation of rectangular artifacts by classifying each pixel independently, precluding the possibility of neighborhood classification statistics overriding a given pixel's erroneous classification, thereby hiding errors. Thus, without state-based methods or the use of blocks over which classification statistics can be gathered to suppress errors when aberrations in the classification occur, pixels are incorrectly classified and subsequently rendered.

[0009] Accordingly, it has been considered desirable to develop a new and improved method and apparatus for distinguishing between noisy continuous tone document types and other document types identified by measures and techniques that have undesirable positive responses to video representing noisy continuous tone images, in order to maintain reliable image segmentation, that meets the above-stated needs and overcomes the foregoing difficulties and others while providing better and more advantageous results.

[0010] The present invention consists of adding a range of video requirement to a document type classification process in order to distinguish between many small video variations associated with a noisy continuous tone document type, and a smaller number of large variations associated with other document types such as halftone or text.

[0011] In accordance with one aspect of the present invention, a method for classifying a pixel of image data as one of a plurality of image types is disclosed. The method includes the steps of determining a first image characteristic value for the pixel being classified, determining a provisional classification for the pixel being classified, and assigning an image type classification to the pixel being classified based on the first image characteristic value and the provisional classification.

[0012] In accordance with another aspect of the present invention, a system for classifying a pixel of image data according to one of a plurality of image types is disclosed. The system includes a first mechanism for determining a first image characteristic value of the pixel being classified, a second mechanism for determining a provisional classification of the pixel being classified, a classifier for assigning an image type classification to the pixel being classified based on the first image characteristic value and the provisional classification, and a processor for image processing the pixel based on the image type classification of the pixel.

[0013] One advantage of the present invention is the provision of a method and apparatus for distinguishing between noisy continuous tone document types and other document types.

[0014] Another advantage of the present invention is the provision of a method and apparatus for providing reliable image segmentation.

[0015] Yet another advantage of the present invention is the provision of a method and apparatus that correctly renders noisy continuous tone document types.

[0016] A further advantage of the present invention is the provision of a method and apparatus that correctly identifies subsequent processing, utilization, or identification of higher level image semantics.

[0017] Still further advantages of the present invention will become apparent to those of ordinary skill in the art upon reading and understanding the following detailed description of the preferred embodiment.

[0018] The invention may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating a preferred embodiment(s) and are not to be construed as limiting the invention.

[0019] FIG. 1 is a block diagram illustrating an exemplary image processing system that incorporates the features of the present invention therein;

[0020] FIG. 2 is a histogram of Sij over a dark, noisy contone image; and

[0021] FIG. 3 is a histogram of a Range of Video over the same dark, noisy contone image used to create the histogram of FIG. 2.

[0022] The following will be a detailed description of the drawings illustrated in the present invention. In this description, the terms “image data” or “pixels” in the form of video image signals, which may be either analog or digital voltage representations of an image, indicate a representation of an image provided from a suitable source. For example, the image signals may be obtained through line by line scanning of an image bearing the original by one or more photosensitive elements, such as an array of charge couple devices commonly referred to as CCDs. Line by line scanning of an image is well known and does not form a part of the present invention.

[0023] Image data may also be derived by a computer workstation program. If the original source of the video from which a workstation program is constructing an output image had its origin from a scanner, regardless of how long ago or through how many transformations it came, then the video can have noise in it. The video may also have noise if it is synthetic and has undergone lossy processing, quantization, or has had a low amplitude spatially random variation (e.g. noise) purposely combined with the image data. Thus, video from a workstation may contain noise, either originating in a physical scanner or synthetically generated and merged with the image. In content, the original video image signals may be composed entirely of a single image component such as lines, text, low frequency halftones, high frequency halftones, contones, or any combination thereof.

[0024] The following description also includes references to slowscan and fastscan digital image data when discussing the directionality of two-dimensional filtering architecture. For purposes of clarification, fastscan data is intended to refer to individual pixels located in a succession along a raster of image information, while slowscan data refers to data derived from a common raster position across multiple rasters or scanlines.

[0025] As an example, slowscan data would be used to describe signals captured from a plurality of elements along a linear photosensitive array as array is moved relative to a document. On the other hand, fastscan data would refer to the sequential signals collected along the length of the linear photosensitive array during a single exposure period which is also commonly referred to as a raster of data.

[0026] Moreover, in describing the present invention, it is assumed that the video signal has a value in a range between 0 and 255. However, any range from the video signal can be utilized in conjunction with the present invention. Furthermore, in the following description, the term “grey level” will be used to describe both black and white and color applications.

[0027] Furthermore, in describing the present invention, the term “pixel” will be utilized. This term may refer to an electrical, (or optical, if fiber optics are used), signal which represents the physical measurable optical properties at a physical definable area on a receiving medium. The receiving medium can be any tangible document, photoreceptor, or marking material transfer medium.

[0028] Moreover, the term “pixel” may refer to an electrical, (or optical, if fiber optics are used), signal which represents the physically measurable optical properties at a physically definable area on the display medium. A plurality of the physically definable areas for both situations represent the physically measurable optical properties of an entire physical image to be rendered by either a material marking device, electrically or magnetic marking device, or optical display device.

[0029] Lastly, the term “pixel,” in the scanning environment, may refer to an electrical, (or optical, if fiber optics are used), signal which represents physical optical property data generated from a signal photosensor cell when scanning a physical image so as to convert the physical optical properties of the physical image to an electronic or electrical representation. In other words, in this situation, a pixel is an electrical, (or optical), representation of the physical optical properties of a physical image measured at a physical definable area on a optical sensor. The term “pixel,” in the rendering environment, may refer to an electrical, (or optical, if fiber optics are used), signal which represents the smallest physical area on a recording substrate that can be rendered. In other words, in this situation, a pixel is an electrical, (or optical), representation of the physical spot created by a laser in a laser printed or the physical spot created by the smallest rendered ink droplet.

[0030] Many of the documents produced today are compound documents in that the documents are composed of several different sub-images that are of different image types or image classes. Some of the common types are text, photos (contones), and halftones. One reason for the increased appearance of compound documents is the widespread use of commercially available word processing and desktop publishing software that is able to generate them.

[0031] As is well known, different types of images require different processing in order to provide optimal image quality. Conventionally, to automatically choose the best processing for different areas of an image, each area is classified into one of several pre-defined classes to determine how to render that part of the image. This image type or image class information can then be used to determine the appropriate processing required to obtain a good rendition of the image when printing, to choose a method of image compression, to determine if optical character recognition would be useful, etc.

[0032] An exemplary automatic segmentation circuit 8 is illustrated in FIG. 1. The circuit 8 is made up of three modules. Image data stored in a data buffer 10 is simultaneously directed to an image property classifying section 20, the first module, and an image processing section 30, the second module, via a data bus 15. The image data is in its raw grey format, for example 6-8 bits per pixel, and is transferred in blocks of about 16 pixels at a time at 400 spots per inch (SPI), or 12 pixels at a time at 300 SPI.

[0033] The image data can be derived from any number of sources, including a raster input scanner, a graphics workstation, an electronic memory, or other storage element, etc. The data buffer 10 can be any suitable commercially available serial in/serial out multi-row buffer having a bit storage capacity sufficient to temporarily store lines of image pixels.

[0034] The image property classifying section 20, can include a number of submodules (e.g. halftone detector, line(text) detector, continuous tone detector, etc.) that determine whether a block of image pixels stored in the data buffer 10 is one type of imagery or another, (e.g. halftone, line/text, contone, etc.).

[0035] In parallel with the image property classifying section 20, the image processing section 30 is made up of any number of sub-processing sections, (e.g. high frequency halftone processor 31, low frequency halftone processor 32, line/text processor 33, or contone processor 34), which perform image processing operations on the same block of image pixels as section 20.

[0036] Each image sub-processing section performs image processing operations that are adapted to improve the image quality of a distinct class of imagery. The third module, control section 41, uses the information derived from the image classifying section 20, to control the image processing section 30. In other words, the control section 41 acts like a multiplexer and selects the proper processed image data according to the image classification determined by the image classifying section 20.

[0037] In a conventional image segmentation scheme, image property classifying section 20 classifies image data as one of a number of classes of imagery (e.g. high-frequency halftone, mid-frequency halftone, low-frequency halftone, line/text, contone, etc.). The image data is then processed according to the properties of the class of imagery that was selected (e.g. low-pass filter and re-screening if it's a high-frequency halftone, threshold with a random threshold if it is a low-frequency halftone, etc.).

[0038] Consequently, the control section 41 decides the type of image processing the image data requires depending on the decision made by the classification section 20. Thus, the output of classification section 20 is quantized to one of a number of different document types. The control section 41 selects the output from one of the image sub-processing sections based upon this classification.

[0039] The image property classifying section 20 utilizes a pixel-by-pixel classification process wherein each pixel is individually classified based on a context of surrounding pixels or, more particularly, the pixel values in a neighborhood of pixels surrounding the pixel to be classified. Although it is desirable for the pixel to be centered, it is not necessary. For example, the pixel being classified could be located in the lower right of the context.

[0040] In a pixel-by-pixel implementation a neighborhood or window of pixels moves along the fastscan direction one pixel at a time and along the slowscan direction one scanline at a time. In contrast, a tiled window implementation moves along the fastscan direction to the next adjoining non-overlapping neighborhood or window of pixels and moves along the slowscan direction to the next adjoining non-overlapping window of pixels.

[0041] The image property classifying section 20 measures, calculates, or otherwise determines intrinsic properties (e.g. microclassifiers) of the video extracted through a simple mathematical formula or heuristic. The microclassifiers have different values (e.g. magnitudes) when computed for images of different types. The values of one or more of the microclassifiers are then analyzed to classify the image pixel according to document type. Further, the microclassifiers can have macro heuristic or mathematical algorithms applied to them to infer a higher level of understanding from the data, suppress errors or other undesirable effects, or to reduce the quantity of data without a commensurate reduction in the information contained therein.

[0042] To characterize continuous tone verses halftone document types, a measure of energy or variation of the video is computed. In this implementation, the Laplacian of the video is first computed from Equation 1 as follows:

[0043] L_i,j=(V_{i−1, j−1}+V_{i−1, j}+V_{i−1, j}+V_i,j−1+V_i,j+1+V_i+1,j−1+V_i+1,j+V_{i+1, j+1})/8−V_i,j  (eq. 1),

[0044] where Vij is the video of the j-th pixel in the i-th scanline, and Lij is the Laplacian of the video at the j-th pixel in the i-th scanline.

[0045] More specifically, the Laplacian Lij is simply the difference between the pixel being classified Vij and the average of its eight neighbors. After computing the Laplacian Lij, the sum of the absolute values of the Laplacians Sij over a small neighborhood of N_Xpixels by N_Yscanlines surrounding the pixel of interest is computed. The Sij operator provides a measure of activity, or energy, within its context region. The parameter N_Xis typically chosen to be 7 for a 400 spot per inch image and 11 for a 600 spot per inch image, and the parameter N_Yis typically chosen to be 3 for a 400 spot per inch image and 5 for a 600 spot per inch image. However, the size of the area used to determine Sij can vary such as a 5×9, 7×13, etc. context.

[0046] A pixel can be classified as, for example, a contone document type or a halftone document type depending upon the value of Sij. Generally, a larger Sij value (about 170 or greater) indicates a halftone document type, while a smaller Sij value (less than about 170) indicates a contone document type. However, Sij alone fails to differentiate between a noisy contone document type having many small variations over the entire context, and a document type having a smaller number of large variations within the context (e.g. halftone, text, etc.). In both cases, the resulting Sij value will be large, thereby indicating a halftone document type, correctly in the case of a single large variation in a few pixels within the context, and incorrectly in the case of many small variations over the entire context.

[0047] Further, as disclosed in copending U.S. patent application Ser. No. 09/159,414 now issued as U.S. Pat. No. 6,128,408 and filed on the same date herewith, Sij can be augmented in light and dark regions of an image to facilitate reliable halftone detection. FIG. 2 is a histogram of Sij over a dark, noisy contone input image. Note that Sij is mostly below a typical halftone threshold of 170, but because this video is dark, Sij will be augmented by, for example, 70-80 to ensure reliable halftone detection. This allows the video's Sij to easily meet the halftone detection algorithm's Sij criteria. Further, because the video is noisy, enough false peaks are detected to meet the estimated halftone frequency criteria. Accordingly, the image property classifying section 20 may incorrectly render the pixel being classified as a halftone document type rather than as a contone document type.

[0048] It should be appreciated that microclassifiers other than Sij can have an undesirable response to noisy contone. For example, a peak counter, from which estimated halftone frequency is derived, can have an undesirable response to a noisy contone video. In both cases, to prevent a pixel from being misclassified as something other than a noisy contone document type, a Range of Video microclassifier Rij can be added to the other classification tests. The Range of Video microclassifier Rij separates the many small video variations from a smaller number of large video variations, thereby enabling the noisy contone document type to be differentiated from other document types. The Range of Video microclassifier Rij is defined as the difference between the largest and smallest video value within a context of pixels in the vicinity of the pixel being classified. In a preferred embodiment, Rij can be determined from a 5×5 context of pixels centered on the pixel being classified. Thus, Range of Video Rij can be determined from Equation 2 as follows:

[0049] Rij=V_MAX−V_MIN,  (Eq. 2)

[0050] where Rij is the Range of Video for the j-th pixel in the i-th scanline, V_MAXis the largest video value within a context of pixels in the vicinity of the pixel being classified, and V_MINis the smallest video value within the same context of pixels.

[0051] The magnitude of the Rij microclassifier determined in accordance with Equation 2 can be applied to the following microclassification rule expressed in pseudo code as:

[0052] If (PROVISIONAL CLASSIFICATION indicates other document types (e.g. halftone or text) AND Rij<=R_THRESH)

[0053] then

[0054] Class=CONTONE,

[0055] where the PROVISIONAL CLASSIFICATION is based on applying mathematical and/or heuristic algorithms to one or more microclassifiers derived from the video image data, for example, the absolute value of the sum of Laplacians Sij, estimated halftone frequency, etc.; Rij is the Range of Video within a context of pixels in the vicinity of the pixel being classified; and R_THRESHis the maximum range value required for a pixel to be classified as contone. Thus, the PROVISIONAL CLASSIFICATION of a pixel being classified is simply an initial classification that is conditioned upon the Range of Video value Rij.

[0056] FIG. 3 is a histogram of the magnitude of the Range of Video classifier over the same dark, noisy contone image used to create the histogram of FIG. 2. The Range of Video is less than about 32 for almost all pixels. If the input image was a halftoned document type, the input image would consist of very white and very dark pixels with little in between, producing a Range of Video value of around 200. Even small halftoned dots attenuated by system modulation transfer function (MTF) would be expected to have a larger range of video than 30. For the same reasons, the edges of text produce similar Range values. Empirically, it has been determined that the Range of Video threshold R_THRESHof about 28 is small enough to enable reliable classification of a legitimate halftone or text pixel when used in conjunction with other microclassifiers (e.g. Sij), while forcing the classification to CONTONE in the case of noisy continuous tone video image data. The appropriate threshold level for any given system depends on a number of characteristics of that particular system, such as signal to noise ratio, optical MTF, and electronic video bandwidth.

[0057] Although the present invention has been described in detail above, various modifications can be implemented without departing from the spirit of the present invention. For example, the preferred embodiment of the present invention has been described with respect to a xerographic printing system; however, the present invention is readily implemented in a thermal inkjet system, a display system, or other image processing system.

[0058] Moreover, the image processing system of the present invention can be readily implemented on a general purpose computer, a personal computer or workstation. The image processing system of the present invention can be readily implemented in software or hardware, such as on an ASIC, thereby enabling the placement of this process in a scanner, electronic subsystem, printer, or display device.

[0059] The present invention has been described with respect to a video range of 0 to 255. However, it is contemplated by the present invention that the video range can be any suitable range to describe the grey level of the pixel being processed. Furthermore, the present invention is readily applicable to any image processing system, not necessarily a binary output device.

[0060] Lastly, the present invention is readily applicable with respect to a monochrome or black/white environment, and a color environment. Namely, the processing operations of the present invention can be applied to each color space value, some function of a given pixel's color space components, or even a function of the color space components of a pixel and other pixels in the neighborhood.

[0061] While the invention has been described with reference to a preferred embodiment, it is not confined to the details set forth above, but is intended to cover such modifications or changes as may come within the scope of the attached claims.