Image processing apparatus and image processing method

[0002] 1. Field of the Invention

[0003] The present invention relates to an image processing apparatus and an image processing method.

[0004] 2. Description of the Related Art

[0005] Since digital techniques today are extensively employed in photocopying machines, many of them perform multi-functions including not only a copying function but also other additional functions. Some photocopying machines work as a printer, a facsimile machine and even a filing machine. They may read and print filed data and file an image coming from a facsimile service.

[0006] Such advanced digital photocopying machines are widely accepted because a user can fax or file a document by simply placing the document on an automatic document feeder in the same way as he or she copies the document.

[0007] The digital photocopying machine is improved in its copying rate year by year and today reaches a rate of 20 to 80 sheets/minute. For this speed performance, many users prefer a digital photocopying machine rather than a conventional dedicated photocopying machine with a slow operational speed when many documents need copying, faxing or filing. The user needs to wait on standby in front of a machine to process a large quantity of documents until all documents are read, or the user needs to return to the machine to retrieve the documents when they are all processed.

[0008] When a large quantity of documents are processed, the orientation of all documents is not always aligned. Thus, the user needs to first align the orientation of the documents.

[0009] The operation of aligning the orientation of a plurality of documents before copying or faxing them on the digital photocopying machine is a troublesome job. Especially when the documents are of a sheet size A, handling them is easy if they are all in a portrait format. However, if portrait and landscape (A4R) formats are mixed, the alignment operation is open to two options of rightward alignment or leftward alignment. Although the A4 documents are typically loaded in a portrait alignment in a feeder in the photocopying machine, they are occasionally placed in a landscape alignment. The A4R (landscape aligned) documents are placed with their text aligned in a normal direction or in an inverted direction facing the port of the feeder. When holes are drilled on one margin of each document for binding purposes, the documents may be intentionally placed left side right.

[0010] FIG. 1 shows document orientations 102-109 when standard A4-size original documents are set in an automatic document feeder 101. As shown, a diversity of alignment settings are possible.

[0011] Because of a diversity of document settings, the alignment method of documents is also different from user to user.

[0012] With the documents not aligned, the photocopying machine delivers copy sheets having a diversity of alignments. If the number of copies increases, the user has to carry out a very tedious job for alignment. Some photocopying machines features an automatic stapling function. With the automatic stapling function enabled, the resulting copy sheets aligned in a diversity of directions are stapled as is.

[0013] Today, a reduction layout process today is available in which a plurality of documents are sequentially read, the read images are reduced, and the reduced images are laid out and printed on a single paper sheet.

[0014] When the documents are different in alignment, the alignment of the images of the documents, if subjected to the reduction layout process, are not consistent, and the resulting images are aesthetically unacceptable.

[0015] Document orientation determination techniques have been developed to free the user from an alignment operation of the documents and to rectify the difference in the setting of the documents from user to user. By integrating a sensor technique for sensing the orientation of the documents fed into a digital photocopying machine, the user is freed from the tedious job of aligning the documents.

[0016] Typically available as orientation determination techniques are a method of using a character recognition technique and a method of using a layout recognition technique. These techniques quickly perform the orientation determination process by checking part of the written content in an original document and outputting the result. In the method of using the character recognition technique, a text area of the original document is extracted, and characters in the text area are determined for the correction orientation.

[0017] The document orientation determination technique recognizes the feature of the text in the original document to recognize the orientation of the original document, and its performance level is not very high. The document orientation determination technique is subject to error. Since color documentation finds widespread use today, the document orientation determination technique has to work as well in color documentation.

[0018] Since the document orientation determination technique is conventionally used for black-and-white documents, an image is handled as a binary image of white and black levels in the document orientation determination process.

[0019] Many digital multi-purpose photocopying machines today, even if they are used for a black-and-white copying, incorporate a color CCD to read color images, since the function as a scanner is also important.

[0020] Since the orientation determination function is designed to work in the black-and-white original documents, color information, though available from the color CCD, remains unused in the orientation determination process. For this reason, the orientation determination occasionally fails to function properly on original documents.

[0021] (1) When the image of a two-color or three-color document, which is in widespread use today, is input with its text area blurred, the orientation of the document is not correctly determined, depending on the color, when its text area is blurred.

[0022] (2) The color text area most important in the determination of the orientation occasionally falls outside the area of interest in the orientation determination process.

[0023] (3) When a document with its text in one color and with its background in another color is handled, the orientation determination process fails to work properly.

[0024] It is an object of the present invention to provide an image processing apparatus and an image processing method for performing an orientation determination process of a color image with high accuracy.

[0025] Accordingly, in one aspect, the image processing apparatus of the present invention preferably comprises image input means for inputting the image of an original document as color image data, orientation determining means for determining the orientation of the original document based on the color image data that was input by the image input means, image editing means for performing a predetermined editing to the data corresponding to the color image data in accordance with the orientation determination result that was provided by the orientation determining means, and output means for outputting image data that was edited by the image editing means.

[0026] The orientation determining means preferably comprises color component analyzing means for analyzing the frequency of appearance of each of the color components based on the color image data and character recognition means for recognizing characters of the color image data in accordance with the analysis result provided by the color component analyzing means.

[0027] The orientation determining means preferably performs character recognition of each of the color components of the color image data in the order of the frequency of appearance of the color components and determines the orientation of the original document from the character recognition result.

[0028] The orientation determining means preferably performs character recognition of the image data of a color component having the highest frequency of appearance to determine the orientation of the original document when the number of color components is small.

[0029] The orientation determining means preferably performs character recognition of the color image data with a background color component removed from the original document to determine the orientation of the original document.

[0030] The image editing means preferably performs a reduction layout process of a plurality of original documents.

[0031] According to another aspect of the present invention, the image processing method comprises the input step of inputting the image of an original document as color image data, the orientation determining step of determining the orientation of the original document based on the color image data that was input through the input step, the image editing step of performing a predetermined editing to the color image data in accordance with the orientation determination result that was provided through the orientation determining step, and the output step of outputting image data that was edited through the image editing step.

[0032] The orientation determining step preferably comprises the color component analyzing step of analyzing the frequency of appearance of each of color components based on the color image data and the character recognition step of recognizing characters of the color image data in accordance with the analysis result that was provided through the color component analyzing step.

[0033] The orientation determining step preferably performs character recognition of each of the color components of the color image data in the order of the frequency of appearance of the color components and determines the orientation of the original document from the character recognition result.

[0034] The orientation determining step preferably performs character recognition of the image data of a color component having the highest frequency of appearance to determine the orientation of the original document when the number of color components is small.

[0035] The orientation determining step preferably performs character recognition of the color image data with a background color component removed from the original document to determine the orientation of the original document.

[0036] The image editing step preferably comprises a reduction layout process of a plurality of original documents.

[0037] According to yet another aspect of the present invention, a storage medium for storing computer-readable program codes for image processing preferably stores a code for the image input step of inputting the image of an original document as color image data, a code for the orientation determination step of determining the orientation of the original document based on the color image data input, a code for the image editing step of performing a predetermined editing to the color image data in accordance with the orientation determination result, and a code for the output step of outputting edited image data.

[0038] FIG. 1 shows the orientations 102-109 of an A4 sheet size document when it is set in an automatic document feeder 101;

[0039] FIG. 2 shows the external appearance of a digital multi-purpose photocopying machine of one embodiment of the present invention;

[0040] FIG. 3 is a block diagram showing the construction of the image processing system of the embodiment;

[0041] FIG. 4 shows the external appearance of an operation panel 311 of the embodiment of the present invention;

[0042] FIGS. 5A-5H shows reduction layouts in which a plurality of reduced documents are arranged;

[0043] FIG. 6 shows 4-in-1 both-side reduction layout examples;

[0044] FIG. 7 shows the reduction layouts in the embodiment of the present invention;

[0045] FIG. 8 is a flow diagram showing the reduction layout process in the embodiment of the present invention;

[0046] FIGS. 9A-9D illustrate the process of a color component analyzer 307;

[0047] FIGS. 10A-10B show the determination result of a text area from an original image;

[0048] FIGS. 11A-11cillustrate a character recognition process; and

[0049] FIGS. 12A-12C illustrate a character orientation determination process.

[0050] Referring to the drawings, the embodiments of the present invention are discussed below.

[0051] The present invention will be discussed in connection with a digital multi-purpose photocopying machine. The present invention is not limited to the photocopying machine, and is implemented in a variety of other apparatuses.

[0052] FIG. 2 shows the external appearance of the digital multi-purpose photocopying machine of one embodiment of the present invention. As shown, an image scanner 201 reads an original document and performs a digital signal processing. The image scanner 201 includes an automatic document feeder. The image scanner 201 is also provided with a color sensor for reading color information.

[0053] A printer module 202 prints on a paper sheet an image corresponding to an original image that is read by the image scanner 201. A telephone line 203 is used for facsimile communication with external apparatuses.

[0054] FIG. 3 is a block diagram showing the construction of an image processing system in this embodiment. As shown, a shading corrector 301 corrects variations in an electrical signal, between pixels, read by an unshown RGB color CCD. A color-to-black/white converter 302 converts RGB color image data into a black/white gray-scale image. Although a stricter method in which RGB luminance data is converted into lightness data is available, the use of G (green) data works without any problem. An edge enhancer 303 sharpens the edge of each character. For example, the edge of an image is enhanced by performing differentiation twice within a window of 5×5 pixels.

[0055] A magnifying circuit 304 reducer data for a reduced copying and interpolates data for an enlarged copying. When a both-side copying or a reduction layout copying is performed, image data is stored in a page memory to be described later. The image data is then read to copy the image as instructed from an operation panel while the address from which the image data read is being switched at the same time. A γ converter circuit 305 converts the image data as luminance data into density data which is output to the printer module. This data conversion is performed using a table. A pulse-width modulation (PWM) circuit 306 converts the image data, now in the form of the density data, into a signal indicative of the emission intensity of laser light, and outputs a pulse having a width corresponding to the density of the image data.

[0056] CPU 312 is the core for controlling the image processing. CPU 312 includes a ROM for storing a program and control data for the apparatus, a RAM for storing a work area and a variety of tables that are used when CPU executes the image processing, and related circuits. The operation panel 311 receives instructions from a user, and all instructions are entered through the operation panel 311. An instruction or command received through the operation panel 311 is sent to the CPU 312. A color component analyzer 307 in this embodiment analyzes what quantity of pixels is present for one color relative to those for other colors in the RGB color data.

[0057] An orientation determining block 308 determines the orientation of an original document by referencing an analysis result 313 input by the color component analyzer 307 and by analyzing the image data that was shading-corrected by the shading corrector 301. More specifically, based on the analysis result 313 analyzed by the color component analyzer 307, the orientation determining block 308 detects a background color considering which color is dominant over the other colors and does not use the dominant color component in the determination process.

[0058] A page memory 309 stores the input image data. In this embodiment, the image data is input on a real-time basis. The orientation determining block 308 inputs the image data corresponding to one document page while immediately notifying CPU 312 of the orientation determination result. In response to the orientation determination result, CPU 312 controls the order of reading from the page memory 309. During a copying operation, the image data is read from the page memory 309, and is sent to the γ converter circuit 305. A scanner unit 310 outputs, to an external apparatus, image data read during a scanning operation.

[0059] When the operation panel 311 selects the orientation determination process and the reduction layout process in the above arrangement, CPU 312 sets the execution of the determination process in the orientation determining block 308. The image processing system receives the RGB electrical signal from the CCD and corrects it in the shading corrector 301. The RGB data is converted into black/white data through the color-to-black/white converter 302 when the destination of the data is a black/white printer. This is because no color data is required in a printing in the black/white printer. When a color scanner inputting is instructed, the data passes through the converter 302 without any processing step added on it. The edge enhancer 303 sharpens the edge of the image (only when selected and instructed), and the edge enhanced image is input to the magnifying circuit 304.

[0060] The shading-processed image data is also sent to the color component analyzer 307. The color component analyzer 307 analyzes color components while receiving the image data. The color component analyzer 307 binarizes the RGB multi-level image data at a threshold and sums resulting binary data on a color-by-color basis. More specifically, the color component analyzer 307 sums the binary data in each of the colors of R (red), G (green), B (blue), C (cyan), M (magenta) and Y (yellow) in the entire image area, and sends the results to the orientation determining block 308. The color component analyzer 307 will be discussed more in detail later.

[0061] The orientation determining block 308 sequentially receives the image data and extracts a text area while storing the image data in the text area. Upon completing the text area extraction, the orientation determining block 308 performs the orientation determination process in the text area of a color image in the order of priority based on the analysis result 313 of the color component analyzer 307. The orientation determining block 308 sends the orientation determination results to the CPU 312.

[0062] The image data that was subjected to the reduction process in the magnifying circuit 304 is temporarily stored in the page memory 309. The page memory 309 stores the image data of a number of sheets. Based on the orientation determination results, an image not in a regular direction is subjected to a rotation process (by changing the manner of reading sequence) and the data of one page is then stored in the page memory 309. In a copying operation, the image data is converted into data compatible with printer characteristics through the y converter circuit 305 and is output to a laser printer unit through the PWM circuit 306.

[0063] FIG. 4 shows the external appearance of the operation panel 311 of this embodiment. In the copying operation, an extended function key 401 has a mode selection function to perform a reduction layout, turn on and off the automatic document orientation sensing, and to select a color document. The user selects the reduction layout through the extended function key 401 in the copying operation.

[0064] The reduction layout copying is now discussed. FIGS. 5A, 5C, 5E and 5G show one-side, reduction layout copying, and FIGS. 5B, 5D, 5F and 5H show both-side, reduction layout copying. The both-side copying function is widely used because it saves paper consumption with the resulting volume of documents being less bulky.

[0065] In the reduction layout, any documents are automatically reduced regardless of the content of the documents. FIG. 6 shows examples of 4-in-1 both-side, reduction layout. As shown, a text, if in Japanese, takes a lateral writing or a vertical writing with a document sheet arranged in a portrait or landscape position. More specifically, documents 601-604 of the lateral writing with characters in the upright position are reduced into layouts 605 and 606, documents 607-610 of the vertical writing with characters in the upright position are reduced into layouts 611 and 612, documents 613-616 of the lateral writing with characters in the horizontal position are reduced into layouts 617 and 618, and documents 619-622 of the vertical writing with characters in the horizontal position are reduced into layouts 623 and 624.

[0066] As seen from FIG. 6, the copy of the actual reduction layout is not one satisfactory to the user.

[0067] FIG. 7 shows the reduction layout provided by this embodiment. As shown, the layout is properly set in consideration of the alignment of each text. More specifically, documents 701-704 of the lateral writing with characters in the upright position are reduced into layouts 705 and 706, documents 707-710 of the vertical writing with characters in the upright position are reduced into layouts 711 and 712, documents 713-716 of the lateral writing with characters in the horizontal position are reduced into layouts 717 and 718, and documents 719-722 of the vertical writing with characters in the horizontal position are reduced into layouts 723 and 724.

[0068] To achieve the proper document arrangement in the reduction layout, the document orientation is sensed.

[0069] The operation of the reduction layout is now discussed. FIG. 8 is a flow diagram showing the reduction layout process in this embodiment.

[0070] In step S801, an operation mode (copying/faxing/filing) selection key 402 in the operation panel 311 shown in FIG. 4 is pressed to select a copying operation. In step S802, the extended function key 401 is pressed to select a reduction layout. Specifically, when the extended function key 401 is pressed, a display mode is changed to a reduction layout display screen as shown in FIGS. 5A-5H, and a desired layout is selected from the display screen.

[0071] In step S803, an orientation determination function is set in an initial setting. Whether or not the color component analysis is performed is also selected in this setting. In step S804, the key for selecting the orientation determination function is pressed. The setting of “document orientation sensing ON, color component analysis ON and copying” selects a color component analysis and orientation determination mode. In step S805, a copy start key is pressed to start a copying operation. In step S806, documents are placed on a document platen. The reader mechanism of the photocopying machine moves to start reading an image.

[0072] In step S809, the image processing described above is performed on the read image data, while the read image data is temporarily stored in the page memory 309. In step S807, the color component analyzer 307 analyzes the color pixels in response to the image. The text area is extracted on a color-by-color basis in step S808. In step S810, the analysis result for one page is sent to the orientation determining block 308, which then determines the document orientation.

[0073] In step S811, the orientation determination result is stored page by page, and it is determined if the number of document sheets specified, for example, 4 document sheets in a 4-in-1 layout are fed in the reduction layout. When the image data is fully input, the process goes to step S813, where the image is read in the order of regular reduction layout to be printed. The resulting copy sheet is delivered in step S814, and it is determined in step S815 if all steps are completed. The above steps are repeated until all steps are completed.

[0074] When an image is input to the orientation determination means in the digital photocopying machine or scanner device in this embodiment, the RGB color images, rather than a black/white image in the conventional method, is input. The image RGB data input on a color-by-color basis is subjected to the following processes.

[0075] (1) In each color image, color data is binarized at a threshold. The image data is summed on a color-by-color basis.

[0076] For example, the number of color pixels are counted for each of the colors of red, blue, green, cyan, magenta and yellow. The number of pixels for the respective colors, namely, the frequency of color components, are sent to the orientation determining block in the order of high to low numbers.

[0077] (2) Text area extraction means performing a preprocess prior to the orientation determination process extracts the text areas in the images on a color-by-color basis.

[0078] (3) In response to the color component analysis result, the orientation determining block performs the orientation determination process based on the color text area information. The color component analysis result serves as information that determines the priority of the text area to be sampled. In the handling of the color component information, the most frequently used color image may be raised in priority. Considering that the color image for the background employs a dominantly large amount of pixels, that color may be excluded from the determination process.

[0079] According to this embodiment, the color information is used in the document orientation determination process. The orientation of a color document is determined at a high-accuracy level.

[0080] Each block is now discussed in detail. Several methods are available for the document orientation determination. On the basis that the alignment of characters plays an important role in the determination of the document orientation, the method of determining the orientation of the original document by determining the alignment of the characters is discussed.

[0081] The original document is typically a mixture of a text, produced by a word processor, drawings, tables and pictures. In such a case, the text areas are determined using part of character recognition technique.

[0082] Color Component Analysis Process

[0083] FIGS. 9A-9D illustrate the process taken by the color component analyzer 307. The color component analyzer 307 binarizes the multi-level image data input by the RGB color sensor. For example, when the color component analyzer 307 binarizes the image data from a red color sensor having a value of “130” at a threshold of “50”, a “1” is obtained. The number of “1s” of the image from the red color sensor is summed by line and by page. If the red color appears frequently, the sum increases. The same process is carried out for the green and blue colors.

[0084] FIG. 9B shows histograms of the appearance of the colors. A symbol “▴” represents a threshold. As understood from FIG. 9B, if a pixel value is greater than the threshold, it is considered large. As a method of extracting the cyan color component, the image data of a green sensor and the image data of the blue sensor are averaged and treated as the image of the cyan color. In the same way, the image data of the magenta and yellow colors are obtained. FIG. 9C shows equations to determine CMY color components.

[0085] In this way, the red, blue, cyan, magenta and yellow pixels are summed by line and by page on a color-by-color basis. FIG. 9D shows an example of the numbers of pixels for the colors. In this example, the number of green pixels per page is the largest, followed by the number of yellow pixels. These numbers are important in determination process. The following process steps are taken.

[0086] (1) The orientation of the document is determined referring to the text area of the color pixels having the largest number. In this case, the color pixels are prioritized according to the frequency of their appearance.

[0087] (2) The background color is neglected. If the color pixels for one color has the predominantly largest number with a large number difference from the color pixels of the second largest number, the color of the largest number pixels is treated as the background color. In the subsequent orientation determination process, the text of the color pixels having the second largest number or smaller is used.

[0088] Specifically, the determination result of the color pixel numbers is subsequently used by the orientation determining block 308 as the color analysis result.

[0089] The color component analyzer 307 may perform the processings for the colors in parallel to keep pace with the pipeline process taken by the image processing system. The predetermined threshold may be set on a color-by-color basis. Making this parameter variable, the threshold for binarizing is modified. This allows variations in the sensors to be corrected and a particular color to be intentionally enhanced.

[0090] Character Alignment Determination Preprocess

[0091] A character alignment determination preprocess is performed sequentially or in parallel (concurrently) on a color-by-color basis.

[0092] [Binarizing Process]

[0093] This process is a preprocess prior to the orientation determination process. The image data input here is multi-level data. The purpose of the binarizing process is to extract the text area from the image. A simple binarizing process serves this purpose well. In such a simple method, a medium value of “128” is used as a threshold.

[0094] When the original document has a background color or a relatively light density, a resulting binarized image may not be appropriate for the subsequent processes. In such a case, a binarizing process using a dynamic threshold level is available.

[0095] In this binarizing process, the threshold is determined based on the luminance of the pixel column prior to a pixel of interest, or the average of density data and luminance or density of a pixel of interest, and the image data is binarized using the resulting threshold.

[0096] In another method, the image processing system receives binary data rather than receiving the multilevel image data. Such a method is acceptable as long as the image data suits the subsequent processes.

[0097] [Text Area Extraction Process]

[0098] In the orientation determination process, the text area is detected from the image data. This is because the algorithm in this embodiment uses the characters as means for determining the orientation of the document.

[0099] Color pixels are detected from the image data color by color. When an area satisfies certain conditions, that area is determined as a text area, and the resulting area information is stored.

[0100] The certain conditions are listed as follows:

[0101] (1) The area has a continuous edge constructed of color pixels.

[0102] (2) The area has a constant width constructed of color pixels.

[0103] (3) There exist nearby color pixel blocks in the area.

[0104] An area that satisfies the above conditions is determined as a text area. Although these conditions alone are not all used to perform an accurate determination, further discussion about them is not provided here.

[0105] FIGS. 10A and 10B show the original image and the result of the text area determination. FIG. 10A shows the original image, in which text areas, a picture and a graph are included. FIG. 10B shows text areas extracted from the original image, namely, 1001-1005 extracted from the original image.

[0106] The text area 1002 contains relatively large character blocks, and the text area 1003 is the widest with the largest number of characters contained therewithin. The text block 1005 is vertically written.

[0107] [Character Recognition]

[0108] The extracted text area is subjected to the character recognition process. Available as the character recognition methods are a feature vector extraction method, a comparison method and the like. For example, as shown in FIG. 11A, the text area containing a character is subjected to character recognition, and the characters in this text area are extracted in a first step. In this step, a rectangle for one character is picked up by detecting continuous color pixels. As shown in FIG. 11B, a single character is then extracted as a pixel block of m×n (64×64 pixels) in a second step. Using a window of 2×2 pixels out of the pixel block, the direction of the distribution of the color pixels (direction vector information) is then extracted as shown in FIG. 11C.

[0109] FIG. 11C shows part of the direction vector information. With the 2×2 pixel window shifted, the feature of the character is obtained by acquiring a significant number of pieces of direction vector information. These feature vectors are compared with the content of a character recognition dictionary stored beforehand, and characters are then extracted in the closeness order of the feature vectors. In this case, a first candidate, a second candidate, a third candidate, . . . in the order of feature vector closeness. The feature closeness of these feature vectors are a distance to the observed character, namely, a measure of similarity (match).

[0110] [Orientation Determination Process]

[0111] The measure of similarity in the character recognition is thus obtained. The character alignment determination process based on the measure of similarity is now discussed. FIGS. 12A-12C illustrate the character alignment determination process, and in this example, a Japanese phrase (sales report of beers) is used.

[0112] FIG. 12A shows a string of characters of the phrase in a standard alignment. FIG. 12B shows the character strings rotated by 270° from FIG. 12A. Now focusing on the character , it is subjected to the character recognition from four directions of 0°, 90°, 180°, and 270° as shown in FIG. 12C. The angle of rotation is changed by modifying the manner of reading the feature vectors, and rotating the original document is not necessary.

[0113] The character recognition result is different from rotation angle to rotation angle as shown in FIG. 12C. FIG. 12C also lists the results of character recognition and their measures of similarity for illustration purposes only, and the actual system does not necessarily provide the same results.

[0114] In FIG. 12C, the character recognition is made at a standard direction (0°), the character is correctly recognized with a high measure of similarity of 0.90. When the character recognition is made at a direction 90° shifted, the character is erroneously recognized as with a measure of similarity as low as 0.40. An erroneous recognition takes place with a measure of similarity lowered because the character recognition is performed based on the feature vectors in directions rotated. In the same manner, when the character recognition is performed at angles of rotation of 180° and 270°, erroneous recognitions take place with measures of similarity lowered. The more complex the character, the greater the differences in the measure of similarity with the direction of the character recognition changed.

[0115] Since the measure of similarity is the highest at the standard direction in the results shown in FIG. 12C, it is highly likely that the original document is aligned in the standard direction. To increase the character alignment determination accuracy, a plurality of characters within the same text area are recognized from four directions in the same way as above. When the character alignment is determined in a single block only, a special character string may be picked up with an erroneous result. For this reason, a plurality of text blocks are subjected to the character recognition process.

[0116] The measures of similarity of the characters of interest are averaged in each of the four directions in the blocks, and the averages of the characters of interest in each of the four directions in text areas are averaged. The direction resulting in the highest final average is determined to be the alignment or orientation of characters (document).

[0117] Rather than determining the alignment based on the measure of similarity of a single character, the alignment of the character is determined referring to not only the measure of similarity of a plurality of characters within the same text area but also the measure of similarity of a plurality of characters within another text area. A high-accuracy orientation determination of the characters (document) is thus possible. The orientation determination is also possible by referring to the measures of similarity of a single character in a text area or the measures of similarity of a plurality of characters within the same text area.

[0118] In this way, the document orientation determination is performed. The characters within a document are varied in form. Some characters may be decorated initials. A drawing may be mistaken for a character. For this reason, the orientation determination process is preferably performed in more areas to result in a high-accuracy determination.

[0119] One example of the document orientation recognition has been discussed. Other recognition methods are also available. Any recognition method is acceptable.

[0120] Since the orientation determining block comprises a CPU having dedicated hardware, the processing system may be easily modified by instructions.

[0121] FIGS. 9A-9D show the color component analysis for one page. A color component analysis result for several lines in a particular block in the document may be output and the priority for sampling is changed per several lines.

[0122] The present invention may be implemented in a system constructed of a plurality of apparatuses (including a host computer, interface units, a reader and a printer) or may be implemented in a standalone apparatus (such as a photocopying machine or a facsimile machine).

[0123] The storage medium holding a program code of software for carrying out the functions of the embodiments is loaded in a system or apparatus. The computer (CPU or MPU) of the system or apparatus reads the program code stored in the storage medium to execute the program. The objects of the present invention are thus achieved.

[0124] The program code read from the storage medium performs the novel functions of the present invention, and the storage medium storing the program code constituting the present invention.

[0125] Available as storage media for feeding the program code are a floppy disk, a hard disk, an optical disk, a magneto-optic disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card a, ROM and the like.

[0126] By executing the program code read by the computer, the functions of the embodiments are performed. Furthermore, the OS (operating system) running on the computer partly or entirely performs an actual process according to the instruction of the program code, and the functions of the embodiments are thus performed.

[0127] The program code from the storage medium is read into a memory incorporated in a feature expansion board in the computer or in a feature expansion unit connected to the computer. The CPU mounted feature expansion board or the feature expansion unit partly or entirely performs the actual process in response to the instruction from the program code. The functions of the above embodiment are executed through the process.

[0128] As described above, the orientation of the color image is determined and the editing is performed on the data corresponding to the color image data in accordance with the determination results. When N document sheets are edited in an N-in-1 layout, for example, the user is freed from arranging the documents in consideration of the orientations of the documents. An easy-to-see N-in-1 image is thus formed.

[0129] Since the orientation determination is performed based on the color component analysis result and the character recognition result, an orientation determination works in the editing processing for documents containing characters.

[0130] Since the character recognition is performed in the order of frequency of appearance of the color components, the orientation determination is performed with a priority placed on the largest number of color characters.

[0131] The character recognition and orientation determination are carried out without any effect from the background color.