Template-based image recognition and extraction

[0002] This invention is applicable to electronic equipment such as an OCR (optical character reader), copier, facsimile machine or processor for implementing an electronic database and, more particularly, relates to an image processing apparatus and method for extracting a specific desired area from a document image.

[0003] Two methods of extracting a desired area from a document are available. The first method is such that whenever the operator wishes to extract a desired area, the operator designates this area in an input image each time. This method involves reading the document image using a scanner, displaying the scanned image on a display monitor and having the operator designate the desired area using a mouse or the like.

[0004] The second method involves creating a template for which size and position information representing rectanglar areas has been decided in advance, applying the rectangular areas decided by the template directly to an input image and then extracting these areas from the input image. In this case rectangular areas whose positions and sizes have been decided by the template are extracted from a scanned document image and the operator need no longer perform the laborious task of specifying extraction areas one after another.

[0005] The first method is disadvantageous in that the operator must specify the desired area each time. This method, therefore, is not suited to the processing of a large number of documents. The second method using the template is disadvantages in that if there is a disparity in position or size between an area to be extracted from the input image and the rectangular area decided by the template, the area to be extracted may be omitted in the extraction process.

[0006] The present invention has been devised in view of the foregoing problems and its object is to provide an image processing apparatus and method whereby it is possible to extract a desired area from a document image in reliable fashion.

[0007] Another object of the present invention is to make possible the rapid and reliable extraction of a desired area from a large quantity of document images.

[0008] A further object of the present invention is to provide an image processing apparatus and method whereby it is possible to reliably extract a desired area from an entered document image while employing a template.

[0009] An image processing apparatus according to one mode of the present invention for attaining the foregoing objects comprises: holding means for holding position, size and attribute as template information in regard to one or a plurality of areas in an image; image input means for inputting a document image; first extraction means for extracting block areas from the document image input by the image input means and evaluating attributes of the extracted block areas; and second extraction means for extracting, from block areas that have been extracted by the first extraction means, a block area that at least partially overlaps an area indicated by the template information and whose attribute agrees with the attribute included in the template information.

[0010] An image processing method according to another mode of the present invention for attaining the foregoing objects comprises: a holding step of holding position, size and attribute as template information in regard to one or a plurality of areas in an image; an image input step of inputting a document image; a first extraction step of extracting block areas from the document image input at the image input step and evaluating attribute of the extracted block areas; and a second extraction step of extracting, from block areas that have been extracted at said first extraction step, a block area that at least partially overlaps an area indicated by the template information and whose attribute agrees with the attribute included in the template information.

[0011] Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0013] FIG. 1 is a block diagram illustrating the construction of an image processing apparatus according to a first embodiment of the present invention;

[0014] FIG. 2 is a flowchart for describing the procedure of template save processing according to the first embodiment;

[0015] FIG. 3 is a flowchart for describing the procedure of area extraction processing according to the first embodiment;

[0016] FIG. 4 is a diagram showing an example of a document (document A) read in for the purpose of generating template data;

[0017] FIG. 5 is a diagram showing an example of the display of a screen for setting areas in template save processing according to the first embodiment;

[0018] FIGS. 6A, 6B are diagrams useful in describing the data structure of template data generated by designation of areas and setting of attributes;

[0019] FIG. 7 is a diagram for describing the registered state of template data;

[0020] FIG. 8 is a diagram illustrating the manner in which a “Document A Template” is registered in the template data;

[0021] FIG. 9 is a diagram showing a document B, which is an example of a document to be processed;

[0022] FIG. 10 is a diagram showing an example of results obtained by executing area partitioning processing in regard to the document B of FIG. 9;

[0023] FIG. 11 is a diagram useful in describing the manner in which a template and the blocks of a document are compared;

[0024] FIG. 12 is a flowchart illustrating the procedure of template save processing according to a second embodiment of the present invention;

[0025] FIG. 13 is a diagram showing an example of results of area partitioning processing in this case;

[0026] FIG. 14 is a flowchart illustrating the procedure of template save processing according to a third embodiment of the present invention;

[0027] FIG. 15 is a diagram showing a state in which a block 3 and a block 5 have been selected in template save processing according to the third embodiment;

[0028] FIG. 16 is a diagram showing the data structure of the document A template in the third embodiment;

[0029] FIG. 17 is a diagram useful in describing results of comparing blocks in a case where a desired area is extracted from the document B using the document A template of FIG. 16;

[0030] FIG. 18 is a flowchart illustrating the procedure of template save processing according to a fourth embodiment of the present invention; and

[0031] FIG. 19 is a flowchart for describing the procedure of area extraction processing according to a fifth embodiment of the present invention.

[0032] Preferred embodiments of the present invention will now be described in detail in accordance with the accompanying drawings.

[0033] [First Embodiment]

[0034] FIG. 1 is a block diagram illustrating the construction of an image processing apparatus according to a first embodiment of the present invention. The apparatus includes a scanner 101 for irradiating a document having an image, reading the reflected light and converting the reflected light to an electric signal; a scanner interface 102 for converting the electric signal obtained by the scanner 101 to a binary digital signal and sending this signal to other components; a pointing device 103 (since a mouse is used in this embodiment, the device will be referred to as a mouse below) for entering desired coordinates on the window of a display; an interface circuit 104 for receiving a signal from the mouse 103 and transmitting the signal to other components; a CPU 105 for executing overall control of the apparatus and processing such as area partitioning; a ROM 106 storing programs, which are executed by the CPU 105, for various control operations and for various processing, as well as font data; a RAM 107 used as a working area for expanding a document image and for area partitioning processing; a display 108 for displaying input images and the like, wherein an image displayed is stored in a VRAM area created by a prescribed address area in the RAM 107; an external storage device 110 such as a hard disk in which data and the like is stored; and an interface 111 for the external storage device 110. These components are interconnected by a bus 112.

[0035] The flow of processing according to the first embodiment will now be described in accordance with the flowcharts of FIGS. 2 and 3. FIG. 2 is a flowchart useful in describing the procedure of processing for saving a template according to the first embodiment, and FIG. 3 is a flowchart useful in describing the procedure of processing for extracting an area according to the first embodiment.

[0036] Processing for saving a template used in area extraction will be described first with reference to FIG. 2. A document A of the kind shown in FIG. 4 having a format desired to be saved is read and converted to binary image data by a scanner 101 at step S201. Next, at step S201, small areas (referred to simply as “areas” or “blocks” below) having attributes such as “text”, “table” and “figure” are set on the input image obtained.

[0037] According to this embodiment, the document A of FIG. 4 is read by the scanner and the read image is displayed on the display 108. FIG. 5 is a diagram showing the display of a screen for setting areas in template save processing according to the first embodiment. An attribute menu 51 is displayed together with the image of document A on the display 108. The operator uses the mouse 103 to select a desired attribute from the attribute menu 51 and to designate frames indicative of rectangular areas. By thus causing the frames of selected attributes to be displayed at desired positions, attributes are set in regard to each rectangular area. When a desired attribute is selected from the attribute menu 51, the color of the border of the rectangular frame displayed by operating the mouse 103 is set to a color that is associated with the selected attribute. In FIG. 5, the border color of a rectangular frame 501 is black, which indicates that the attribute of this area is “text”. The border color of a rectangular frame 502 is red, which indicates that the attribute of this area is “table”. The border color of a rectangular frame 504 is yellow, which indicates that the attribute of this area is “figure”.

[0038] FIGS. 6A, 6B are diagrams useful in describing the data structure of template data generated by designation of areas and setting of attributes. Area data is stored on an area-by-area (block-by-block) basis in the manner shown in FIG. 6A. The area data is obtained by registering starting-point coordinates X, Y, width and height of the area (namely position information indicative of the rectangular frame displayed by operating the mouse), and includes an “attribute” field in which an identification number corresponding to the attribute set for the above-mentioned rectangular frame is set. Each attribute and its identification number are as shown in FIG. 6B. It should be noted that the X, Y coordinates of the starting point represent the upper left corner of the particular area.

[0039] Next, at step S203, the assemblage of area data of each of the blocks set in the manner described above is registered and saved as a template. For example, FIG. 7 shows an example in which a template is registered anew under the title “Document A Template” as the fourth of a group of already existing templates.

[0040] FIG. 8 is a diagram illustrating the manner in which a “Document A Template” is registered in the template data. Since the areas indicated by the rectangular frames 501-505 have been set on the image of document A in the manner shown in FIG. 5, five blocks are registered in the document A template. Which rectangular frame corresponds to which block of the blocks 1-5 may be decided, by way of example, in accordance with the order in which the rectangular frames were designated at step S202.

[0041] Processing for extracting desired areas from a newly read image using a template registered in the manner described above will now be explained with reference to the flowchart of FIG. 3.

[0042] First, a template necessary to extract desired areas is selected at step S301. In this embodiment, the template names of templates registered as shown in FIG. 7 are displayed on the display 108 and the operator is allowed to select the desired template using the mouse 103. Next, at step S302, a document to undergo processing is read by the scanner 101 and converted to binary image data in order that the area partitioning processing, described later, may be executed. This example will be described on the assumption that the document A template set at shown in FIG. 5 has been selected and that a document B, shown in FIG. 9, has been read in as the document to be processed.

[0043] This is followed by step S303, at which the input image obtained is subjected to area partitioning processing known to those skilled in the art, blocks are extracted and the attribute of each block is evaluated. FIG. 10 is a diagram showing an example of results obtained by executing area partitioning processing in regard to the document B. Each block of the blocks A-E is stored as extracted area information, with the data structure of the stored blocks being the same as that of the area information shown in FIGS. 6A, 6B. In other words, information representing the position, size and attribute of each extracted block is stored.

[0044] Next, at step S304, area data of blocks that have been extracted from document B are compared with the area data of blocks that have been saved in a selected template (the document A template). This is followed at step S305 by the extraction of a block whose area at least partially overlaps the area of a block in the template and has the same attribute as that of the area it overlaps. If such a block is extracted at step S305, then this block is deemed to be a block identical with the desired block and the image contained in the area of this block is output at step S306.

[0045] FIG. 11 is a diagram useful in describing the manner in which a template and the blocks of a document are compared. Blocks that have been recorded as the document A template are indicated by dashed lines in FIG. 11, and blocks that have been extracted from the image of document B are indicated by broken lines. Though blocks A and B that have been extracted from document B are somewhat displaced from the block positions of the document A template, the blocks A and B have portions that overlap the block positions of the template. The attribute of both of these areas is “text”. Blocks A and B, therefore, are extracted at step S305. The end result is that the area data of all blocks A-E of document B obtained in FIG. 10 is output.

[0046] Thus, in accordance with the first embodiment, as described above, even if the position and size of an area set in a template differ slightly from the position and size of an area to be extracted from a document image that has actually been read, a desired area can be extracted from the document image reliably.

[0047] At step S305 described above, a block extracted at least partially overlaps an area set in the template and has the same attribute as that of this area. However, whether or not a block is to be extracted may be decided upon taking into account the degree of overlap between the two blocks. For example, it may be so arranged that a block to be selected must overlap a block in the template by 70% or more and must have the same attribute. Furthermore, an arrangement may be adopted in which this ratio can be set for each block of the template.

[0048] [Second Embodiment]

[0049] According to the first embodiment described above, the setting of areas saved in a template is performed manually using the mouse 103. However, area setting processing for saving a template can be automated using area partitioning processing of the kind executed at step S303.

[0050] The flow of such processing will be described in accordance with the flowchart of FIG. 12. FIG. 12 is a flowchart illustrating the procedure of template save processing according to the second embodiment of the present invention.

[0051] The document A (see FIG. 4) having the format desired to be saved is read in and converted to binary image data, which is for area partitioning processing described below, by the scanner at step S1201. The input image obtained is then subjected to area partitioning processing at step S1202 to extract various areas (blocks) such as a text area, figure area, table area and title area. It should be noted that the area partitioning processing used at step S1202 may employ a technique will known to those skilled in the art. The result of this area partitioning processing is shown in FIG. 13. Each block is thus extracted and identification numbers corresponding to the various attributes as well as the position information are obtained as area partitioning data on a per-block basis. The structure of the area data regarding this document in this case can be assumed to be the same as that shown in FIG. 6. The area data of each of the extracted blocks is then registered and saved as the “Document A Template” in the manner illustrated in FIG. 8.

[0052] It should be noted that the processing for extracting areas from the image of an input document using the template registered as set forth above is as described earlier in conjunction with the flowchart of FIG. 3.

[0053] Thus, in accordance with the arrangement described above, a document to serve as a template need only be read by the scanner 101 to generate the template automatically. This enhances operability.

[0054] [Third Embodiment]

[0055] In the second embodiment, all areas extracted by area partitioning processing in template save processing are saved as a template. However, it can be so arranged that only a desired area among the extracted areas is selected and saved as a template. In the third embodiment, specific blocks among blocks extracted in area partitioning processing are designated by the mouse 3 and only the area partitioning data of these blocks are saved as a template.

[0056] FIG. 14 is a flowchart illustrating the procedure of processing for saving a template according to the third embodiment. The flow of processing of the third embodiment will now be described in accordance with the flowchart of FIG. 14.

[0057] The document A (see FIG. 4) having the format desired to be saved is read in and converted to binary image data by the scanner 101 at step S1401. The input image obtained is then subjected to area partitioning processing at step S1402 to extract various areas (blocks) such as a text area, figure area, table area and title area. The result of this extraction is as described above in connection with the second embodiment (FIG. 13). By way of example, area partitioning processing indicated at step S303 in FIG. 3 can be used as the area partitioning processing at step S1402.

[0058] This is followed by step S1403, at which specific blocks are selected from the extracted blocks using the mouse 103. FIG. 15 is a diagram showing a state in which a block 3 and a block 5 have been selected. In the case of the example shown in FIG. 15, the selected blocks are indicated by hatching so as to be distinguished from other blocks.

[0059] Next, at step S1404, only the area partitioning data (attributes and position information, etc.) of the blocks selected at step S1403 is saved as a template. In this example, only the area partitioning data of blocks 3 and 5 is saved as the document A template in the manner shown in FIG. 16.

[0060] A case where document B shown in FIG. 9 is processed using the document A template saved in the manner explained above will now be described. FIG. 17 is a diagram useful in describing results of comparing the template and each block of a document in a case where desired areas are extracted from the document B using the template composed solely of the blocks selected in FIG. 15. If the area extraction processing of FIG. 3 is executed in the case of this example, only the area partitioning data of blocks C and E is output. The blocks C and E among the blocks (indicated by the broken lines) of document B are judged to be areas that at least partially overlap the blocks (indicated by the dashed lines) recorded in the template and that have the same attributes as those of the template blocks.

[0061] Thus, in accordance with the third embodiment, as described above, desired areas can be selected from automatically extracted area data and the selected areas can be saved as a template.

[0062] [Fourth Embodiment]

[0063] In the third embodiment, areas to be saved as a template are designated by the operator. However, it goes without saying that areas that are not to be saved as a template may be designated by the operator. In the fourth embodiment, a desired area among blocks that have been extracted by area partitioning processing is designated by a mouse or the like and area data of blocks other than the designated block are saved as a template.

[0064] FIG. 18 is a flowchart for describing the procedure of processing for saving a template according to the fourth embodiment. The document A (see FIG. 4) having the format desired to be saved is read in and converted to binary image data by the scanner 101 at step S1801. The input image obtained is then subjected to area partitioning processing at step S1802 to extract various areas (blocks) such as a text area, figure area, table area and title area. This extraction operation provides results already described above in connection with the second embodiment (FIG. 13).

[0065] Next, at step S1803, desired blocks are selected from among the extracted blocks by using the mouse 103. For example, in FIG. 15 described above, blocks 3 and 5 are illustrated as being in the selected state. As shown in FIG. 15, the selected blocks are indicated by hatching so as to be distinguished from other blocks. Whereas these selected blocks were registered as a template in the third embodiment, the area data of these selected blocks is deleted in accordance with the fourth embodiment. Though the interior of a rectangular area of a selected and deleted block is hatched in the description given above, the method of representing a selected area is not limited to this expedient. For example, an arrangement may be adopted in which the frame border indicating the rectangle of the block is erased along with the deletion of the area data.

[0066] Next, at step S1804, only the area partitioning data (attributes and position information, etc.) of the blocks selected at step S1803 is saved as a template. In this example, only the area partitioning data of blocks 1, 2 and 4 is saved as the document A template as the result of the selections shown in FIG. 15.

[0067] The extraction of areas from an input document image using a template thus obtained is as already described above with reference to the flowchart of FIG. 3 in the first embodiment.

[0068] [Fifth Embodiment]

[0069] In each of the foregoing embodiments, documents to be processed are placed in the scanner 101 and read in one by one. However, automatic extraction by template in a case where a large number of documents have been entered by the scanner 101 using an ADF (Automatic Document Feeder) also is possible. FIG. 19 is a flowchart for describing the procedure of area extraction processing according to a fifth embodiment of the present invention. The flow of processing will now be described in accordance with the flowchart of FIG. 19. It should be noted that the processing of steps S1901, S1903-S1907 is the same as that of steps S301-S306 in the first embodiment.

[0070] The method of template registration may employ the technique of any of the first through fourth embodiments described above.

[0071] A desired template to be used to extract desired areas is selected at step S1901. Next, it is determined at step S1902 whether there is a document to be input, i.e., whether there is a document in the ADF. If the decision rendered is “YES”, control proceeds to step S1903, where the document is read by the scanner and converted to binary image data by the scanner 101. If the decision at step S1902 is “NO”, however, then this processing is terminated.

[0072] Next, the input image obtained is subjected to area partitioning processing at step S1904 to extract blocks. For example, if document B shown in FIG. 9 was read in at step S1903, then results of the kind shown in FIG. 10 are obtained by the area partitioning processing executed at step S1904.

[0073] Next, at step S1905, area data of a blocks that have been extracted by the area partitioning processing of step S1904 are compared with area data of blocks that have been saved in the template elected at step Sl901. This is followed by step S1906, namely by the extraction of a block whose area at least partially overlaps the area of a block in the template and has the same attribute as that of the area it overlaps. Here the extracted block is construed to be a block identical with the desired block defined in the template and the area partitioning data of this block is output (step S1907).

[0074] For example, if the document A template obtained based upon document A shown in FIG. 5 is selected and the document B shown in FIG. 9 is read in by the scanner 101 and processed, then areas overlap as shown in FIG. 11. (Blocks that have been recorded as the document A template are indicated by dashed lines, and blocks that have been extracted from the image of document B are indicated by broken lines.) Since the attributes of the blocks whose areas overlap each other are the same (see FIGS. 5 and 10), the data of the areas of all blocks in the document B obtained in FIG. 10 is output. Control then returns to step S1902 and processing continues.

[0075] Thus, in accordance with each of the embodiments as described above, the following advantages are obtained when desired areas are extracted from a document image by a template: (1) Operator intervention is reduced; (2) the accuracy of desired area extraction is improved; (3) large quantities of documents can be processed automatically; (4) operability is enhanced; and (5) overall processing time is shortened.

[0076] Though the block areas are set as rectangular areas in each of the foregoing embodiments, the areas may have any shape, such as circular or elliptical, as long as they are closed areas.

[0077] The present invention can be applied to a system constituted by a plurality of devices (e.g., a host computer, interface, reader, printer, etc.) or to an apparatus comprising a single device (e.g., a copier or facsimile machine, etc.).

[0078] Furthermore, it goes without saying that the invention is applicable also to a case where the object of the invention is attained by supplying a storage medium storing the program codes of the software for performing the functions of the foregoing embodiments to a system or an apparatus, reading the program codes with a computer (e.g., a CPU or MPU) of the system or apparatus from the storage medium, and then executing the program codes.

[0079] In this case, the program codes read from the storage medium implement the novel functions of the invention, and the storage medium storing the program codes constitutes the invention.

[0080] Further, the storage medium, such as a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, non-volatile type memory card or ROM can be used to provide the program codes.

[0081] Furthermore, besides the case where the aforesaid functions according to the embodiments are implemented by executing the program codes read by a computer, it goes without saying that the present invention covers a case where an operating system or the like running on the computer performs a part of or the entire process in accordance with the designation of program codes and implements the functions according to the embodiments.

[0082] It goes without saying that the present invention further covers a case where, after the program codes read from the storage medium are written in a function extension board inserted into the computer or in a memory provided in a function extension unit connected to the computer, a CPU or the like contained in the function extension board or function extension unit performs a part of or the entire process in accordance with the designation of program codes and implements the function of the above embodiment.

[0083] Thus, in accordance with the present invention as described above, it is possible to reliably extract desired areas from an entered document image while employing a template.

[0084] As many apparently widely different embodiments of the present invention can be made without departing from the spirit and scope thereof, it is to be understood that the invention is not limited to the specific embodiments thereof except as defined in the appended claims.