Text structure analysis method and text structure analysis device

[0002] 1. Field of Invention

[0003] The present invention relates to a text structure analysis method and text structure analysis apparatus applied to analytical processing of text. More specifically, the present invention relates to an apparatus and method for taking differing parts of a plurality of texts and extracting a portion of the content of a given text.

[0004] 2. Description of Related Art

[0005] In known methods of text analysis, when performing processing on two given documents by determining and extracting the differing portions of the given documents, it is common to process them by treating one line or one sentence as a unit and performing structural analysis based on their connections/relationships. For example, there is known a method of processing text by examining the connections/relationships of sentences and creating a tree or graph of the text based on these connections/relationships. Another known method of performing text analysis creates a paragraph having joined sentences from the connections/relationships of the sentences.

[0006] Japanese Laid-Open Patent Applications No. 4-23765 (JP 4-23765), No. 6-35960 (JP 6-35960), No. 7-200589 (JP 7-200589), and No. 8-6945 (JP 8-6945) disclose examples of the tree/graph method of text analysis. Japanese Laid-Open Patent Applications No. 4-306768 (JP 4-306768) and No. 5-324708 (JP 5-324708) disclose examples of the method employing paragraphs of joined sentences.

[0007] The method according to JP 4-23765 performs syntactic analysis regarding each of two texts and tries to detect differing parts in these texts using syntax trees.

[0008] In the apparatus according to JP 6-35960, a document structure detection unit that uses surface-level vocabulary and a document structure detection unit that uses grammatical subjects are utilized to perform text analysis. The apparatus performs detailed structural analysis of documents, not only by using vocabulary information appearing on the surface level of the sentences, but also by using grammatical subjects detected from each sentence, including subjects not clearly indicated in the sentences.

[0009] The method of JP 7-200589 extracts text having qualifying relationships between sentences in the form of a tree structure. The method arranges and displays a portion of text using the extracted tree structure.

[0010] The method of JP 8-6945 generates nodes based on rules governing the assembly of attributes of neighboring lines, connects the nodes with links, and applies costs to the nodes and links. The method interprets the logical structure of the sentences by traversing text graphs.

[0011] The method of JP 4-306768 joins and performs structural analysis of sentences based on connections/relationships between the sentences in the documents.

[0012] The method of JP 5-324708 restores paragraph information according to connections/relationships and segmentation rules for each sentence, and performs structural analysis by considering that paragraph information.

[0013] All of these known methods and apparatus perform processing that examines the connections/relationships of texts with a sentence or line as being the smallest unit treated. As a result, computational volume is great and large amounts of computational time are required for processing.

[0014] Additionally, all of these known methods and apparatus merely perform processing according to predefined rules (rules regarding connections/relationships) The user cannot change the method of analysis in accordance with a particular sentence being processed. Furthermore, after having performed some sort of processing using the results of structural analysis, when outputting the results, various problems arise, such as having to perform analysis again using the results of the structural analysis and having to reconstruct the text for output.

[0015] Also, when taking the differing part between two sentences or lines of text, processing in the known methods is performed so as to indicate the fact that a changed portion exists in that line by outputting only the line having the change or by outputting the entire text and assigning a mark to the start of the line having the change.

[0016] For example, consider a portion of text representing a three-day weather forecast, such as shown in FIG. 2, having a change in a part of its content. For this example, the weather forecast is changed from that of FIG. 2 to the forecast shown in FIG. 4. Comparing the contents of FIG. 2 and FIG. 4, the probability of precipitation on the 2nd of the month was changed from 40% to 20% and the lowest temperature on the 3rd of the month was changed from 6° C. to 8° C.

[0017] In the known methods, when attempting to output the content of the differing parts of two portions of text or line units, only the changed part is displayed, as shown in FIG. 8A, or the entire text is displayed with a mark (for example, an asterisk) assigned to the line having the change, as shown in FIG. 8B.

[0018] In the example shown in FIG. 8A, “9<” indicates the content of the 9th line before change, that is, the 9th line in FIG. 2 (Probability of Precipitation 40%) and “9>” indicates the content of the 9th line after change, that is, the 9th line in FIG. 4 (Probability of Precipitation 20%). In the same manner, “15<” indicates the content of the 15th line before change, that is, the 15th line in FIG. 2 (Lowest Temperature 6° C.) and “15>” indicates the content of the 15th line after change, that is, the 15th line in FIG. 4 (Lowest Temperature 8° C.).

[0019] In FIG. 8A, because only the line having the change is displayed, the surrounding context cannot be grasped. Similarly, in FIG. 8B, all of the text is displayed, but the context cannot be grasped because too much text is displayed.

[0020] Thus, the aim of the present invention is to provide a text structure analysis method and text structure analysis device capable of processing units or blocks of textual content. The processing is performed by extracting content for each collection of content of the text, for example, when taking the differing part of two texts and extracting a part of the textual content.

[0021] In a first aspect of the present invention, the text structure analysis method comprises detecting a content boundary pattern for each collection of textual content from an input text indicating a boundary of that collection, establishing a content boundary for the input text based on the result of that detection, and treating the input text in units or blocks of textual content for each collection of content based on the established content boundary. In another aspect of the present invention, the content boundary pattern is a lexical unit or code occurring repeatedly within the text. In a further aspect of the present invention, the content boundary pattern is a control code occurring repeatedly within the text when the input text is coded with a text coding language.

[0022] In another aspect of the invention, the content boundary pattern is a pattern respectively representing a starting point and an ending point of a given collection of content, and the content contained between these patterns is treated as one unit of textual content.

[0023] In a further aspect of the present invention, a text structure analysis device comprises a content boundary pattern storage means for storing a content boundary pattern for each collection of textual content from an input text indicating a boundary of that collection and a text analysis means for detecting a boundary section present in an input text based on the content stored in this content boundary pattern storage means. The text analysis means establishes a content boundary for the detected boundary section.

[0024] Thus, the present invention is capable of extracting portions of text in units or blocks of textual content based on established content boundaries. The invention performs this extraction by detecting a content boundary pattern indicating a boundary of a collection of textual content from an input text. As a result, processing based on a structural analysis from connections/relationships of the portions of text, and the like, becomes unnecessary and processing can be greatly simplified.

[0025] As stated, the lexical units and codes appearing repeatedly in the text may be established as content boundary patterns. These content boundary patterns can be indicated by the user in accordance with the text that is the object of the processing (hereinafter referred to as the object text). They can also be discovered automatically by the system. That is, by detecting the lexical units and codes appearing repeatedly in the input text, the system can store those detected lexical units and codes as content boundary patterns and can establish content boundaries in the input text based on those stored lexical units and codes. Accordingly, it becomes possible to establish the content boundaries in accordance with the object text.

[0026] As also stated, when the input text is coded in a text coding language, the control codes appearing repeatedly in the text may be established as content boundary patterns. In this case as well, the content boundary patterns can be indicated by the user in accordance with the object. They can also be discovered automatically by the system. That is, by detecting the control codes appearing repeatedly in the input text, the system can store those detected control codes as content boundary patterns, and can establish content boundaries in the input text based on those stored control codes. Accordingly, it becomes possible to establish the content boundaries in accordance with the object text.

[0027] By using a pattern representing the starting point and a pattern representing the ending point of the textual content of a given collection, the system can indicate both the starting point and the ending point of a given collection and can clearly mark the scope of text for extraction.

[0028] These and other features and advantages of this invention are described in or are apparent from the following detailed description of the preferred embodiments.

[0029] The preferred embodiments of this invention will be described in detail, with reference to the following figures, wherein:

[0030] FIG. 1 is a block diagram showing the configuration of a preferred embodiment of the text structure analysis system according to the present invention;

[0031] FIG. 2 is a drawing showing an example of unchanged object text;

[0032] FIG. 3 is a drawing showing a data structure of the unchanged object text of FIG. 2 according to a preferred embodiment of the present invention;

[0033] FIG. 4 is a drawing showing the object text of FIG. 2 after modification;

[0034] FIG. 5 is a drawing showing the result of processing the differing part of the portions of text shown in FIG. 2 and FIG. 4;

[0035] FIG. 6 is a drawing showing an example of textual content coded using HTML;

[0036] FIG. 7 is a drawing showing the text created using the text coding language in FIG. 6;

[0037] FIG. 8A is a drawing showing a prior art method in which only the different lines are displayed; and

[0038] FIG. 8B is a drawing showing another prior art method where the full text is displayed and the different parts are marked.

[0039] FIG. 1 is a drawing showing the configuration of the text structure analysis apparatus of the present invention. The apparatus comprises a content boundary pattern storage unit 1, a text analyzer 2, and an input text 3.

[0040] Content boundary pattern storage unit 1 stores a pattern (patterns) representing a boundary of textual content representing one collection of textual content. This boundary will be explained with reference to an example of a weather forecast, as shown in FIG. 2. As shown in FIG. 2, information representing the date and day of the week is recorded as “1st (Mon.)” in the paragraph below the heading “Weather Forecast.” In the paragraph below that, contents such as “Weather Partly Cloudy”, “Probability of Precipitation 10%”, “Lowest Temperature 3° C.” and “Highest Temperature 12° C.” are recorded as the contents of the weather forecast on that day. Information representing the date and day of the week recorded as “2nd (Tues.)” and the contents of the weather forecast on that day are recorded next, followed by information representing the date and day of the week recorded as “3rd (Wed.)” and the contents of the weather forecast on that day.

[0041] In such text, the parts enclosed in the frames indicated by the dashed lines in the drawing can be treated respectively as collections of content x1, x2, and x3.

[0042] Here, the lexical unit “number+number suffix” representing each day is used as a boundary pattern representing the boundary of each of these collections of content x1, x2, and x3. That is, the boundary pattern p1 in collection x1 is “1st”, the boundary pattern p2 in collection x2 is “2nd”, and the boundary pattern p3 in collection x3 is “3rd.” In regard to such boundary patterns, as described previously, “number+number suffix” may be automatically extracted by the system from the text and may be established by the user as “number+number suffix”. The boundary pattern “number +number suffix” established by the system or by the user is stored in the content boundary pattern storage unit 1 as a content boundary pattern.

[0043] When the object text is input, text analyzer 2 establishes content boundaries of the object text based on the content boundary patterns stored in content boundary pattern storage unit 1. The processing for establishing the content boundaries in regard to the input object text is explained with reference to FIG. 3.

[0044] FIG. 3 shows the data structure of the text of the weather forecast shown in FIG. 2. The boundary pattern stored in the content boundary pattern storage unit 1 is “number+number suffix”. Accordingly, when a text, such as in FIG. 2, is input in text analyzer 2, the data structure of the input text (shown in FIG. 3) is checked from the start of the text. The part matching the boundary pattern stored in content boundary pattern storage unit 1 is detected and the content boundary is established from the start of the matching part.

[0045] In this example, the parts matching “number+number suffix” are “1st”, “2nd”, and “3rd.” Boundaries #1, #2, and #3 are established, respectively referring to the starting points of these boundaries, namely, the numbers “1”, “2”, and “3”. The addresses of “1”, “2”, and “3” are established, thus establishing the content boundaries of the object text.

[0046] Also, the text from the start of each content boundary to immediately before the next content boundary is defined as one unit of content. For example, the text from content boundary #1 to immediately before content boundary #2 is defined as collection x1, the text from content boundary #2 to immediately before content boundary #3 is defined as collection x2, and the text following content boundary #3 is defined as collection x3.

[0047] When the weather forecast shown in FIG. 2 is input into the text analyzer 2, the content boundary patterns of the input text established by the user or by the system are stored in the content boundary pattern storage unit 1. Accordingly, in the document analyzer 2, the content boundaries are established in regard to the input text based on the boundary patterns stored in the content boundary pattern storage unit 1, and the contents x1, x2, and x3 are extracted based on the first input text.

[0048] When the text of the weather forecast is changed (as shown in FIG. 4) and new information is input into the text analyzer 2, in the same manner as before, the content boundaries are established in regard to the input text based on the boundary patterns stored in the content boundary pattern storage unit 1. The collections x11, x12, and x13 are extracted, based on the second input text, as shown in FIG. 4.

[0049] By comparing the differing parts for each of the collections (x1 and x11, and x2 and x12, x3 and x13), the difference between the first and second input text can be determined and the corresponding collections containing the next textual content are displayed. In these examples, the probability of precipitation was changed from 40% to 20% within the contents of collection x2 and collection x12 and the lowest temperature was changed from 6° C. to 8° C. within the contents of collections x3 and x13.

[0050] The results of the above processing are shown in FIG. 5. As is clear from FIG. 5, the results are displayed in units or blocks of textual content. Therefore, when compared with the prior art display of only the changed parts (shown in FIG. 8A), because the surrounding contents related to the differing parts are also shown, discrimination of the contents becomes easier for the viewer. Also, comparing the above method with the prior art method of displaying the fill text (shown in FIG. 8B), because the display is in units of textual content in this invention, the problem of having too much content displayed is avoided.

[0051] Additionally, by displaying the changed parts in a different color from the rest of the text (for instance “Probability of Precipitation 20%” on the 2nd and “Lowest Temperature 8° C.” on the 3rd), it is easier for the viewer to visually recognize the changed parts.

[0052] Furthermore, when the input text is coded in a text coding language, the control codes may also be established as the content boundary patterns. For example, in HTML (Hyper Text Markup Language), “<table>”, “</table>”, “<tr>”, “<th>”, and the like, are used as control codes. For example, when the contents coded as shown in FIG. 6 are displayed, they are displayed as a table as shown in FIG. 7.

[0053] Here, “<table>”indicates the start of the first table, and “</table>”indicates the end of that table. Accordingly, by treating the text between the “<table>”and “</table>” codes as one group, they can also be established as one unit or block of textual content. In this case, “<table>” is used as the content boundary pattern indicating the start of the first unit or block of textual content and “</table>” is used as the content boundary pattern indicating the end of the first unit or block of textual content.

[0054] Because the start and end of a unit of textual content can be specified by treating control codes as content boundary patterns, information not directly related to the content, such as “The weather forecast is updated every 6 hours” as shown in FIG. 6, can be excluded when extracting the textual content as a unit or block of content. Accordingly, the unit of content can be limited to only that which is necessary.

[0055] Even when the content of one text is, for example, text crossing two lines, and the content of the other text itself is not changed, but crosses three lines, having a line break and a space, accurate processing and comparison of the two texts can be accomplished with the present invention. This is done by extracting the entirety of the texts as one unit or block of content and comparing the texts without treating the line breaks and spaces as differences. In short, in regard to that which does not pertain to the content of the text itself and is changed only in form and has no significance, because those can be ignored, the differing parts of the text itself can be taken.

[0056] Additionally, because there is no need to examine the connections/relationships of the units of text, the computational volume can be greatly reduced and the processing can be made far easier. That is, because the contents can be extracted by only checking the input text according to the content boundary patterns stored in the content boundary storage unit 1, there is no need to perform structural analysis while considering connections/relationships as in the prior art, and therefore, the processing can be made easier. Furthermore, in the present invention, because the contents can be output as they are, the processing volume can be greatly reduced.

[0057] Also, content boundary patterns, for example, patterns of lexical units such as “number+number suffix” or control codes of text coding languages such as “<table>” and “</table>” can be established by the user as content boundary patterns. However, when these patterns of lexical units or control codes appear repeatedly at fixed intervals within the text, it also becomes possible to extract the contents automatically on the system side by automatically detecting these patterns. Thus, because it becomes possible to establish the boundary patterns in accordance with the input text, it becomes possible to extract the matching contents in the text.

[0058] The embodiment explained above is an example of a preferred embodiment of the present invention. However, the invention is not limited to this and various modified embodiments are possible within a range not deviating from the main principle of the present invention. For example, in the embodiment described previously, “number+number suffix” was used as a content boundary pattern, but the invention is not limited to this particular embodiment and any possible identifiers of content boundaries are acceptable if they can be used to detect a collection of textual content.

[0059] Furthermore, a processing program for performing the processing of the present invention can be stored on a storage medium such as a floppy disk, optical disk, hard disk, and the like. Additionally, it can also be in a format obtainable via a network.