Method for inset detection in document layout analysis

[0002] The following related application is hereby incorporated by reference for its teachings:

[0003] “A SYSTEM FOR DOCUMENT LAYOUT ANALYSIS,” Robert S. Cooperman, application Ser. No. 08/652,766, filed May 23, 1996 U.S. Pat. No. 5,784,487 Patel, Jay.

[0004] This invention relates generally to the processing of information during a document layout analysis in an optical character recognition (OCR) system, and more particularly to a method for detecting insets in the structure of a document page so as to further complement the document layout and textual information provided in an optical character recognition system.

[0005] A portion of the disclosure of this patent document contains material which is subject to copyright protection, Copyright© 1992-1997 Xerox Corporation. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0006] The present invention is a system for providing information on the structure of a document page so as to complement the document layout and textual information provided in an optical character recognition system. As described by A. Rahgozar and R. Cooperman in application Ser. No. 08/585,142, filed Jan. 11, 1996 U.S. Pat. No. 5,841,900, it is known to employ OCR systems to divide or segment a digitized document based upon the boundaries of the text, graphic or pictorial regions therein. Document layout analysis is a process by which the information regarding the organization of the document content, i.e. its structure, is extracted from the document image. The structure identifies document entity types (e.g. paragraphs, figures and tables), their properties (e.g. the number of columns in a table), and their interrelations (e.g., a figure is above a caption). Although OCR is an inherent part of this process, it is not intended as an aspect of the present invention. Document layout analysis includes identifying the sections of a page, identifying regions of the document that represent captions to images or equivalent sections on a document page, the identification of column boundaries and techniques for employing such layout information to fit pages of differing sizes into a common document format. Indeed, such information is important to state-of-the-art OCR systems that attempt to produce textual output in “reading order” (e.g., appropriately reflecting the flow of text in columns, headers, footers, captions and now insets). It will be further appreciated that the methodology presented here is equivalently applicable to any of a number of document types so long as they may be represented as a digital image divided into segments or regions according to content.

[0007] Heretofore, a number of commercially available OCR products have disclosed techniques that make use of page segmentation, and then preserve only the size and location of the text regions. For example, a product marketed under the mark Omnipage™ outputs text regions as frames. Other OCR products are known to be limited to identification of a single section per page of a document. The typical system does not preserve section structure, and text flow. While, some systems may preserve sections under specific circumstances, for example, where the columns in a section did not overlap one another, none of the systems finds insets (regions not part of the text flow) within a document page.

[0008] In accordance with the present invention, there is provided a document layout analysis method for determining document structure data from input data including the content and characteristics of regions of at least one page forming the document, the method comprising the steps of: segmenting the regions within the page to identify regions characterized as text, graphics, and rulings; within the text regions, identifying those text regions representing headers footers and captions and for the remaining text regions analyzing text regions to identify and characterize certain text regions as insets; recomposing the text regions of the document; combining the text regions into columns; and determining boundaries of at least one column on the page.

[0009] In accordance with another aspect of the present invention, there is provided a document layout analysis method for determining document structure data from input data including the content and characteristics of regions of at least one page forming the document, the method comprising the steps of: receiving page data; segmenting the regions within the page to identify regions characterized as text, graphics, and rulings; within the text regions, identifying those text regions representing headers footers and captions and for the remaining text regions identifying those remaining text regions that are frame and credit insets; recomposing the text regions of the document; identifying, within the recomposed text, any center, column and stray insets; and recalculating the sections of the document so as to produce output data indicative of the reading order of text regions therein.

[0010] One aspect of the invention deals with a basic problem in optical character recognition systems—that of preserving detail of the input document and in particular the flow of text (reading order) within the document. In particular, the present system is directed to a layout analysis system including inset detection that can be used to extend the capability of an OCR package to more accurately recreate the document being processed. Such a system produces output data for a word processor or a reading assistance device by preserving the reading order of the document to facilitate editability and a close approximation of the original appearance of the document.

[0011] This aspect is further based on the discovery of a technique that alleviates the text flow problems of traditional OCR systems. The technique provides information to supplement that generated by OCR processes. The technique employs methods to identify and characterize insets that may be found in a document image.

[0012] The technique described above is advantageous because it can be adapted to provide or supplement document layout data for any document processed by a system employing the techniques. The techniques of the invention are advantageous because they extend the capability of an OCR system—providing the ability to closely approximate a document in a word processing or speech synthesis environment, and enabling editability and readability while preserving the document's original appearance.

[0013] FIG. 1 is a general diagram of a personal computer or data processing system that provides an embodiment for various aspects of the present invention;

[0014] FIG. 2 is a data flow diagram depicting the flow of information in accordance with an aspect of the present invention;

[0015] FIGS. 3-5 are flowcharts illustrating various aspects of the operation of the document analysis system in accordance with the present invention; and

[0016] FIGS. 6, 7 and 8 are illustrations of various examples of the insets detected by the methods described herein.

[0017] The present invention will be described in connection with a preferred embodiment, however, it will be understood that there is no intent to limit the invention to the embodiment described. On the contrary, the intent is to cover all alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims.

[0018] For a general understanding of the present invention, reference is made to the drawings. In the drawings, like reference numerals have been used throughout to designate identical elements. In describing the present invention, the following term(s) have been used in the description.

[0019] The term “data” refers herein to physical signals that indicate or include information. When an item of data can indicate one of a number of possible alternatives, the item of data has one of a number of “values.” For example, a binary item of data, also referred to as a “bit,” has one of two values, interchangeably referred to as “1” and “0” or “ON” and “OFF” or “high” and “low.” A bit is an “inverse” of another bit if the two bits have different values. An N-bit item of data has one of 2N values.

[0020] The term “data” includes data existing in any physical form, and includes data that are transitory or are being stored or transmitted. For example, data could exist as electromagnetic or other transmitted signals or as signals stored in electronic, magnetic, or other form.

[0021] “Memory circuitry” or “memory” is any circuitry that can store data, and may include local and remote memory and input/output devices. Examples include semiconductor ROMs, RAMs, and storage medium access devices with data storage media that they can access. A “memory cell” is memory circuitry that can store a single unit of data, such as a bit or other n-ary digit or an analog value.

[0022] A “data processing system” is a physical system that processes data. A “data processor” or “processor” is any component or system that can process data, and may include one or more central processing units or other processing components. A processor performs an operation or a function “automatically” when it performs the operation or function independent of concurrent human control.

[0023] “Character” means a discrete element that appears in a written or printed form of a language. Characters in the English language can thus include not only alphabetic and numerical elements, but also punctuation marks, diacritical marks, mathematical and logical symbols, and other elements used in written or printed English. More generally, characters can include, in addition to alphanumeric elements, phonetic, ideographic, or pictographic elements.

[0024] A “character type” is a category of which a character may be an instance, such as the letter “a” or the number “3”.

[0025] A “character recognition system” is a system that can operate on an item of data defining characteristics of a character other than its character type to obtain an item of data indicating its character type.

[0026] A “word” is a set of one or more characters that is treated as a semantic unit in a language. “Text” is an arrangement of one or more lines of characters; the characters of a text may form words.

[0027] A “page” is a portion of a document, wherein the portion may be properly characterized as printed or printable information that may be rendered on a single sheet of the output medium. As used herein, the term “page” is not only applied to hard copy substrate sheets, but to corresponding electronic representations of such sheets. The electronic representations may be produced by one or more image input terminals, for example, a digital scanner. A “document” consists of one or more “pages.”

[0028] A “region” is a portion of a page identified by a boundary be it visible or invisible. The boundary may have a regular rectangular shape or an irregular shape. The boundaries of two regions may overlap, meaning that the bounding boxes about regions may share common borders, but that the pixels within a bounding box are uniquely associated with a region defined by a bounding box. A “section” is a portion or region of a page that contains a particular number of columns, of particular widths and distances from the left margin. The flow of text does not continue from one section, to another, on the same page. For example, adding more text to a section will generally increase the height of that section in a word processor. This definition is believed to be consistent with the capabilities of certain word processors.

[0029] As used herein, “textual” regions of a page image are those regions containing alphanumeric or similar symbols, wherein the symbols form units of semantic understanding. Conversely, “non-text” or “picture” regions are those regions of a page image having little or no textual content, typically content that is pictorial or graphic in nature. Non-text regions include, for example, halftoned images, continuous tone images, tables, line drawings, clip-art.

[0030] A “table” as used herein is intended to refer to textual data arranged in a regular structure so as to generally have a plurality of rows and columns. A “caption” is a textual region associated with a non-text region such as a picture, wherein the text generally includes descriptive information associated with the picture. An “inset”, as the term is used herein, is a region of text within a document that is not part of the flow of text for the larger region or section in which it appears (e.g., a sidebar), nor is it a header, footer or caption. Insets may be further classified by type including, for example, frame insets, credit insets, center insets, column insets and stray insets.

[0031] An “image input device” or “image input terminal” (IIT) is a device that can receive an image and provide an item of data defining a version of the image. A “scanner” is an image input device that receives an image by a scanning operation, such as by scanning a hard copy document.

[0032] An “image output device” of “image output terminal” (IOT) is a device that can receive an item of data defining an image and provide the image as output. A “display” is an image output device that provides the output image in human viewable form. The visible pattern presented by a display is a “displayed image” or simply “image.”

[0033] Turning now to FIG. 1, the present invention finds particular use in image scanning systems that have the capability for optical character recognition. Although directed to aspects of an optical character recognition engine, a preferred embodiment for the present invention is a desktop data processing system having the capability for receiving digitized input. As shown in FIG. 1, such a system could include a computer workstation 30, including a keyboard 32, a system unit 34 and a display (monitor) 36. The workstation preferably includes a Windows™-compatible personal computer equipped with an lntel™ or compatible 80486 or Pentium™ microprocessor, a VGA, SVGA or Multisync color monitor, eight megabytes of RAM, eight to sixteen megabytes of permanent virtual memory, Microsoft Windows™ 95 or Windows NT operating system, and a minimum of nine megabytes free hard disk space. In addition, the data processing system preferably includes an image input terminal 40 for producing a digitized representation of a hard copy document. Alternatively, the digitized image source may be a disk or similar storage device for recording encoded image files, including fax-modem systems. Data processing system 30 may further include a printer 42 to render printed output of data produced, stored or otherwise made available via the data processing system. It will be appreciated by those skilled in the art that various aspects of data processing system 30 may be remotely located and that the functionality of such a system may be enabled via a network or similar communication links between various components of the system.

[0034] Considering FIG. 2 in conjunction with FIG. 1, the general operation of an optical character recognition system will be described. It is to be understood that a preferred embodiment for the present invention is a computer workstation or data processing system similar to that illustrated in FIG. 1. Accordingly, various aspects of the present invention are implemented in software that controls the operation of a data processing system. FIG. 2 represents the data flow from a hard copy document 60 to a word processing document 62.

[0035] Initially, hard copy document 60 is placed or otherwise presented to image input terminal 40, where it is scanned or digitized as represented by operation 64 of FIG. 2. The output of the digitization process is rasterized data, reference numeral 66. The rasterized data is preferably in the form of an array of image data, for example a bitmap such as a Tag Image File Format (TIFF) formatted file or an equivalent graphics file format After digitization, the raster data 66 is passed first to a structure determination operation 68 and then to an optical character recognition (OCR) engine 72, where it is processed in accordance with OCR techniques described herein. In a preferred embodiment, the OCR engine is the engine found in the Xerox TextBridge® Pro 98 or TextBridge® Pro 3.03 OCR software. In general, the OCR engine operates to identify characters and characteristics for each of a plurality of words identified within the raster data. Output of the OCR engine comprises a number of files, including files 76a, 76band 76c. File 76apreferably provides a means by which the document may be viewed on display 36 (FIG. 1) or in conjunction with a data searching tool. File 76ccomprises data including descriptions of characters, fonts, and locations on the page and may be in a format readable by the native word processing system 80 or speech synthesizer 82.

[0036] The system described herein for document layout analysis uses information, produced by an OCR engine as inputs. In a preferred embodiment, the structure determination operation 68 would divide or segment each page into regions, each region preferably consisting essentially of elements of a common type or context (e.g., text or graphic/non-text (drawing, pictorial)). Further details of the manner in which such systems may segment a document page are described by Rahgozar et al. in U.S. patent application Ser. No. 08/585,142 or by Mahoney in U.S. patent application Ser. Nos. 08/563,479 and 08/565,181, filed Nov. 28, 1995, the relevant portions of which are hereby incorporated by reference. The resulting text regions can be as large as a column, or as small as a paragraph. The underlying requirement being that a correct reading order for a page, and likewise a determination of headers, footers, captions, and insets can be accomplished by arranging the text regions identified. Accordingly, in the input provided to the OCR recognition process, each non-text region should correspond to one graphic (picture) region. The document structure determination operation described herein may work adequately, even with some errors introduced by a page segmentation system.

[0037] For each region or segment identified within a particular page of the document, the following information is preferably provided by a segmentation system (not shown):

[0038] 1) A description of each region as either text, graphic (e.g., picture, drawing, etc.), ruling, timestamp, or table cell;

[0039] 2) The coordinates of the region's bounding box in absolute distance from either the scanned edge or the hard copy document's edge; and

[0040] 3) For text regions, font information comprising: (a) the most common font, (b) the font size, (c) the font style (e.g., regular, bold, italic, fixed pitch, etc.), and (d) the number of lines of text within the region.

[0041] In response to the above inputs and in accordance with the present invention, a document layout analysis system with inset detection capability preferably provides the following outputs:

[0042] 1) An identification of insets within the document;

[0043] 2) A classification of the inset type (e.g., frame, credit, center, column or stray; examples of frame, center and column depicted in FIGS. 6-8);

[0044] 3) The reading order of text regions;

[0045] 4) A grouping of text regions into sections.

[0046] Having generally described the inputs and outputs of a document layout analysis system including inset detection capability in accordance with the present invention, attention is now turned to the operation of such a system.

[0047] The overall program flow of the document layout analysis system is depicted beginning with FIG. 3. In particular, FIG. 3 is a flow chart generally illustrating various upper level steps executed to produce the output of the document layout analysis system—information sufficient to recreate a close approximation of an original document so that the information may be employed to produce a formatted file (e.g., word processor format 76cin FIG. 2) that is editable and bears a close approximation to the appearance of the original document.

[0048] Referring to FIG. 3, the process begins at step 100 where the document layout analysis system is invoked or otherwise started, preferably by a call to execute instructions as are generally described hereafter. At step 200, the document layout analysis system, operating on the data processing system characterized above, receives page data. Subsequently, at step 300, the page data is segmented by characterizing the text, graphics, and rulings regions of the image, before identifying headers, footers, and captions at step 400 as described in “A SYSTEM FOR DOCUMENT LAYOUT ANALYSIS,” application Ser. No. 08/652,766, previously incorporated by reference. Step 500 identifies, within the segmented page, those segments corresponding to frame insets and credit insets as will be described in further detail.

[0049] Once the segments have undergone a primary categorization the document segments may be “recomposed” so as to place the segments into an appropriate output order, step 600. Again, the various details of the recomposition operation, including identification and merging of related segments can be found in application Ser. No. 08/652,766. Once the document page is recomposed, and the sections identified therein include information characterizing the sections and the segments making up the section, aspects of the present invention are again employed to identify center insets, column insets, and stray insets within the document page, step 700 described below.

[0050] After the final inset detection operation is completed, there is preferably some further processing (including recalculation of section information and vertical ruling analysis) in order to determine the output or reading order of the various segments within the sections, step 800. For example, if a section includes a picture and a caption, the output order will indicate where the caption is to be placed so that it appears in the same position when the page data is later accessed in a word-processing environment. At step 900, the data for a particular page is returned or otherwise temporarily stored in memory operatively connected to or inserted in a memory device that forms part of the data processing system.

[0051] Step 1000 tests to determine if all of a plural number of pages forming the document have been analyzed in accordance with steps 200-800 and the data stored as described in step 900. If not, processing continues at step 200 for subsequent pages of the document. Otherwise, processing continues at step 1100, where page margins are characterized based upon the individual page information previously obtained and stored. Once accomplished, various regions or pages within the document may be resized so as to provide a document with consistently appearing pages. It will be appreciated that the advantages arising from the resizing operation of step 1200 are wide ranging—allowing a user to avoid having to reformat, or more specifically resize, various features of the document within a word processing environment. Once the optional resizing operation is accomplished, the document layout analysis operations are completed and control is returned to the aspects of the system that invoked the document layout analysis, preferably to continue with OCR operations and further format analysis on the image data in accordance with the document layout information.

[0052] It will be appreciated by those skilled in the art of optical character recognition and digital image processing that the order of the above-described steps may be altered in accordance with the information required by the receiving operation. For example, the inset detection and recomposition steps, while being illustrated as occurring before OCR for certain product requirements, may also be run after the OCR operation is applied to image data that has simply been segmented. Moreover, the results achieved with the recomposition and inset detection operations may actually be slightly better if the operations followed an OCR operation.

[0053] Having described the general operation of the document layout analysis system, attention is now turned to the following description, including representative source code, where further details of various inset detection operations (steps 500 and 700) are described in detail. Identification of frame and credit insets is accomplished in accordance with the steps depicted in FIG. 4. In particular, the process depicted in FIG. 4 starts by analyzing rulings within the image, step 510. The analyze ruling step further includes the subprocesses of analyzing full frames, step 512 and finding pictures and text in the full frames, step 514. After locating the pictures and text in the full frames, the frame insets are found, step 520, and then the credit insets are found, step 530.

[0054] The step of analyzing rulings (510) includes the operation of analyzing full frames. In particular, for each horizontal ruling within the frame, the following code is executed:

[0055] The measurement unit called RECOMP_UNITS, is preferably on the order of 0.1 mm so, “const RECOMP_UNITS MAX_FRAME_JOINT_GAP=70;”, refers to a distance of approximately 7 millimeters. Alternate units of, perhaps resolution based measurement (e.g., pixels or a relative scale based upon the original document size), may be employed as well for dimensions indicated herein. The preceding C++ function will call MakeARuleFrame, which makes Rule Frames (a list of the four rulings that make a frame), and the coordinates on the page of the resulting box.

[0056] Another specific type of box that is considered is a text region in reverse video (region where foreground brightness is brighter than background—requiring prior recognition of the reverse video region), with three rulings forming a box below the reverse video. In such cases, the box is assumed to have a top at the top of the area of reverse video background, and is otherwise treated as other boxes. Thus, for each text that is reverse video that also is not a header, footer, or caption: the same C++ function is called and if that function returns true, the text region is classified as a frame inset. The Rule frame extends to the top of that text frame.

[0057] Subsequent to completing the analysis of full frames, the step of finding pictures and text in full frames, step 514, is executed where for each frame earlier created MakeARuleFrame operation is carried out to determine which text regions are within a frame:

[0058] where GLeft( ), GRight( ), GTop( ), GBot( ), get the coordinates of the frame. The input arguments, iTop, iLeft, iBot, iRight are the coordinates of the smallest box that encloses all the text in the region.

[0059] For each text region that is neither a header, footer, nor a caption the find frame insets operation, step 520, below is executed:

[0060] The above code fragment finds the smallest possible frame for each eligible text region, and declares it to be an inset of that frame, using the AddInsetWithin function. Each frame now includes a list of any insets that are a part of it.

[0061] If the frame also contains a picture, the text might actually be a caption, and not an inset, but for the purposes of the present embodiment captions and insets are treated the same, and this distinction does not have to be made. It will be apppreciated that a caption can often be associated with a figure, and that the format of the caption may have different requirements than the format of an inset. A further description of the methods employed to find captions can be found in U.S. patent application Ser. No. 08/652,766.

[0062] There is one remaining condition that could cause the text within a frame to not be called an inset. In pseudo-code:

[0063] For each frame:

[0064] For each text region initially considered an inset of that frame:

[0065] For each other text region also initially considered an inset of that frame:

[0066] if both the text regions are each wider than 19 mm and region A is to the side of region B,

[0067] then no text regions in that frame are classified as Frame Insets.

[0068] where the state of “being to the side of”, is determined using the following code:

[0069] Subsequently, step 530, described below is executed to find credit insets. Credit Insets are typically author credits, found at the bottom of an article. For the purposes of this system, only text region insets within the spacing Parameter-D of the bottom of the page, and within the spacing Parameter-E of the left of the page are considered as possible credit insets—where “parameter-#” denotes a predetermined or programmable variable. Those are the credit insets that would be output in a fixed location. Other article credits might flow with the end of the article, and therefore would not be considered an inset.

[0070] Credit insets have less than Parameter-F lines of text, and have a ruling above them that overlaps the X-direction coordinates of the bounding box of the text region, by at least Parameter-G percent. In addition the rulings of credit insets is within Parameter-H distance of the top of the text region, its width is below Parameter-I, and the height of the text region is typically less than Parameter-J. The following code accomplishes the various calculations and analyses of the find credit insets operation using parameter values that were established empirically:

[0071] Having described in detail the operations associated with step 500, attention is now turned to the detailed explanation of the operations accomplished in step 700 for identification of center insets, column insets, and stray (or non-column) insets. In a preferred embodiment, these functions are called after recomposition operation (step 600) has identified sections, reading order, columns, and column widths (e.g., as described in U.S. patent application Ser. No. 08/652,766). However, after the inset classification functions are complete, the recomposition functions that identify reading order, columns, and column widths, are preferably rerun to treat each newly found inset in a similar manner as the recomposition function treats captions. The identification of those insets, improves the results of reading order, column determination, and column width determination.

[0072] Referring to FIG. 5, in a preferred embodiment, one or more functions may be called before the inset classification steps. In particular, the function bInsetLikeFont( ), step 710, tests to determine if the text region has font and/or reverse video characteristics that are common for insets. A text region is considered to have an inset-like font if the font information of the most commonly used font in the region, has at least one of the following attributes; bold, italic, reverse video, or at least MIN_TIMES_AV_PTSIZE_FOR_INSET_SIZE=1.5, times the average font height on the page. A further discussion of font attributes can be found in U.S. Pat. No. 5,668,891, issued to Fan et al. on Sep. 16, 1997.

[0073] A second function that is preferably invoked before inset classification determines if the text region has main text to the left or right of it, step 720. The pseudocode for such a function is as follows:

[0074] for each text region on the page (called “TextA”)

[0075] if the text region is wider than 19 mm.

[0076] MAX_TEXT_NOT_TO_SIDE_OVERLAP=1.5 mm

[0077] EffectiveTop=TextA's top+MAX_TEXT_NOT_TO_SIDE_OVERLAP;

[0078] EffectiveBot=MAX (TextA's bottom—MAX_TEXT_NOT_TO_SIDE_OVERLAP, Effective Top);

[0079] for each other text region on the page (called “TextB”)

[0080] if the TextB's top is less than or equal to EffectiveBot

[0081] and TextB's bottom is greater to or equal to EffectiveTop

[0082] and TextB is wider than 19 mm

[0083] then

[0084] if TextB's left is left of TextA's left

[0085] then TextA is considered to have main text to its left

[0086] if TextB's right is right of TextA's righ

[0087] then TextA is considered to have main text to its right

[0088] If a text region meeting the above conditions were found both to the left and the right of this text region, it is marked as such, so that a call to “there-is-main-text-to-the-left-and-right” for this text region would return true.

[0089] Center insets (FIG. 6) are found within columns of main text flow in step 730. In a preferred embodiment, only center insets that span more than one column are found. Although insets in the middle of a single column are often falsely identified, it is possible to find center insets entirely within a column. To be classified as a center inset a text region that is found must have the following characteristics:

[0090] span at least two columns;

[0091] have an inset-like font (as previously described); or

[0092] for each text on the page that is not a header, footer, caption, or previously identified inset, if the text region is wider than 19 mm, and the text has an inset-like-font, and there is main text to the left and right, and This-Text-In-Section-Can-Be-A-Center-insect (code follows) requirements are true.

[0093] And another type of center inset, with somewhat different conditions may also be found, where for each text region on the page:

[0094] where bColumnTextAboveAndBelow (pText) does the following:

[0095] Since the inset for a multi column case, even if just the 1-1 col type, has to be wider than the columns above and below it:

[0096] the function pTextUpperRight->GIsTextContinues( ) checks whether a particular text region is likely to have ended in a hard return (in which case it returns FALSE), or whether it likely continues somewhere else on the page (in which case it returns TRUE)

[0097] In further identifying the center insets, the following operations may be employed (and may also be used in the recomposition operations previously described. Additional data is collected as follows:

[0098] Rights is an array of the distance from the beginning of the page to the end of each line in the text region;

[0099] Lefts is an array of the distance from the beginning of the page to the beginning of each line in the text region;

[0100] NumLines is the number of text lines in the text region;

[0101] AvLeft is the mean of all the Lefts for this text region; and

[0102] AvRight is the mean of all the Rights for this text region.

[0103] It will be appreciated that additional “rules” may be employed to improve or alter the robustness of the identification algorithm presented herein. For example, the step of finding center insets (730) may also include additional steps such as: (i) determining whether the text region 730rhas columnar text regions 730a, 730bor 730cpositioned to at least one side thereof; (ii) determining whether the text region 730rspans at least two columns 730a-730c; (iii) determining whether a width dimension (730w) of the text region is greater than a predefined width (I); (iv) identifying a left-hand column 730awhere a left edge of the text region is located, then determining the portion of the left-hand column's width (732) that the text region comprises and determining that the text region is a center inset whenever the portion is greater than a predetermined value (A) and less than a second predetermined value (B); (v) identifying a right-hand column where a right edge of the text region is located, then determining the portion of the right-hand column's width (734) that the text region comprises and determining that the text region is a center inset whenever the portion is greater than the predetermined value (A) and less than the predetermined value (B); (vi) determining that a column portion difference value, representing the difference between the portion of the left-hand column's width and the portion of the right-hand column's width, is less than a predetermined value (C); determining whether the text region has a number of text lines at least equal to a predetermined value (F); (vii) determining whether there is another text region above and vertically aligned with said text region, and having a width greater than the predefined width (I); (viii) determining whether there is another text region below and vertically aligned with said text region, and having a width greater than the predefined width (I); (ix) determining whether there is a first column having a left edge, and whether a left edge of said text region is located within a predetermined distance (D₁) of the left edge of the first column; (x) determining whether there is a second column having a right edge, and whether a right edge of said text region is located within a predetermined distance (D₂) of the right edge of the second column; (xi) determining whether there is columnar text above and below the text region, wherein the columnar text regions above and below the text region are each lined up with the text region, and where each columnar text region includes at least a predetermined number of lines (G), and each columnar text region has a width that is less than a predetermined percentage (H) of the width of the text region, and wherein each columnar text region is closer than a predetermined value vertical distance (I) from the text region; or (xii) determining that a text region immediately above the text region, is likely to continue on the same page.

[0104] The process of determining if a columnar text region interrupted by a center inset (e.g., region 730b) is likely to continue on the same page would preferably include the steps of (a) determining the width of the last line of the text region above (736), (b) determining the width of the average line of that text region (also equal to 736 in FIG. 7, but not necessarily so), and determining that it is unlikely to continue when the width of the last line is less than a predetermined percentage (K) of the width of the average line of that text region.

[0105] After finding center insets, the find column insets process is initiated at step 740, where columns off to one side of the main text flow (e.g., inset 742 in FIG. 8) are identified. In a preferred embodiment, only the leftmost, and rightmost columns within a section are considered to be possible column insets as consideration of other columns would likely lead to a higher incidence of false identification as a column inset. Thus, there must be at least two, and preferably at least three columns in order for a section of the image to have a column inset. Leftmost and rightmost columns are considered to be column insets if they meet column width requirements. For purposes of explanation tests will be characterized for either a leftmost or rightmost column, but one skilled in the art of page layout analysis will appreciate that the tests may be applied to either the leftmost and/or rightmost columns of a page section. In one test, if the leftmost or rightmost columns are less than Parameter-O percent the width of the average of the other columns, and have less than Parameter-P percent of the number of lines that the average of the other columns do it may be a column inset. Furthermore, Parameter-O and Parameter-P are preferably different if the column has an inset-like font.

[0106] Where const INT32 MIN_COLS_FOR_COL_INSET=3; for each section on the page, the following operations are executed:

[0107] as specifically represented by the function blsColInset below:

[0108] Next, blsColInset is called again for the last column, with the following values, to determine any insets are present within the column:

[0109] ThisColWid=LastColWid

[0110] NextColWid=PenultimateColWid

[0111] OppositeColWid=FirstColWid

[0112] AvColWid=AvLeftColWidths

[0113] ThisColNumLines=LastColNumLines

[0114] NextColNumLines=PenultimateColNumLines

[0115] AvColNumLines=AvLeftColNumLines

[0116] bThisBoldltRev =bLastColBoldltRev

[0117] It will be appreciated by those skilled in the art of document analysis that various modifications and alterations to the above-described identification techniques may be employed based upon empirical knowledge about the documents. For example, as with any of the parameter-based tests, it is possible to utilize a programmable variable for the parameter so that one could simply alter the parameter value to obtain improved performance. It is also contemplated that the programmable may be adjusted automatically by the system after an analysis of a plurality of “test” (exemplary) documents so as to determine typical parameter values.

[0118] Lastly, stray or non-column insets are detected at step 750. It may be preferable to perform a page recomposition operation before initiating the stray inset detection operation, because stray insets are other bits of text, not properly part of the text flow and not previously identified as an inset. If the rules that determined columns, did not include a column where the stray inset texts are, they are considered to be stray insets. Furthermore, stray insets must be Parameter-Q distance outside of the outermost column if they do not have an inset-like font, or their midpoint can be within Parameter-R (which may be a positive or negative value) of the column end if they do have an inset-Like font. The width of a stray inset, multiplied by Parameter-S, must be less than the width of the columns of the closest section. To identify stray insets, for each section on the page:

[0119] for each text region in this section that meets this condition

[0120] where pText->GFullColWidth( ) is true if pText is wider than 19 mm, and pText->GNumLines( ) is the number of lines of text in pText.

[0121] Practically, the “rules” employed in the stray inset determination operations may include one or more of the following: (a) less than a predetermined number of lines (e.g., 5 lines); (b) determining a distance from a left edge of a leftmost column in a text region to a right edge of a rightmost column in the text region, wherein the text region is a part of document having columnar regions, and identifying as a stray inset a the text region having a width less than a predetermined percentage (S) of the distance; (c) determining whether the width of the text region is narrower than a predefined width (I); (d) determining whether a portion of the text region lies outside of a columnar space defined by the outermost edges of outermost columns within a section of the document; (e) whenever a portion of the text region lies within the columnar space, determining if the text region has less than a predetermined number of lines (T), and if so, characterizing the text region as a stray inset; or (f) whenever a portion of the text region lies within the columnar space, determining that the width of the text region is less than a predetermined fraction of the width of the columnar space.

[0122] It is, therefore, apparent that there has been provided, in accordance with the present invention, a method and apparatus for page layout analysis, including a method for detecting and classifying insets within various portions of a document page. It will be appreciated by those skilled in the art that various implementations and modifications to the above-described embodiment may be accomplished to meet the requirements of alternative embodiments. Such natural extensions or modifications of the method described herein for inset identification are included within the spirit and scope of the of the invention as defined by the appended claims.