System and method for generating a taxonomy from a plurality of documents

[0001] This application is a continuation application of and claims priority to U.S. application Ser. No. 09/289,174, filed on Apr. 9, 1999. The disclosure of the prior application is considered part of and is incorporated herein.

[0002] This invention relates generally to a system and method for processing a document and in particular to a system and method for generating a list of classifications (a taxonomy) from a plurality of phrases which are extracted from a set of documents.

[0003] Various factors have contributed to the storage of vast amounts of textual data information in computer systems and computer databases. The dramatic increase in the storage capacity of computer storage devices, such as hard disks, tape drives and the like and a decrease in the cost of these higher capacity computer hard drives are factors. Other factors include an increase in the transmission speed of computer communications, an increase in the processing speed of personal computers and an expansion of various computer communication networks, such as a bulletin board or the Internet. People therefore currently have access to the large amounts of textual data stored in these databases. However, although the current technology facilities the storage of and the access to this textual data, there are new problems that have been created by the vast amount of textual data that is now available.

[0004] In particular, a person trying to access the textual data in these computer databases needs a system for analyzing and processing the data in order to retrieve the desired information quickly and efficiently without retrieving extraneous information. In addition, the person trying to access the information needs an efficient system for condensing each large document into a plurality of phrases (one or more words) which characterize the document so that the person can browse the phrases and understand the document without actually viewing the entire document. It is also desirable to be able to automatically generate a classification system based on the extracted phrases stored in the computer database so that a person may use the classification system to browse through the database and focus in on the most relevant documents or pieces of textual data in the computer database. The classification system may be known as a taxonomy which consists of one or more subject matter headings with sub-headings reflecting the phrases extracted from the documents in the database.

[0005] To generate a typical subject matter classification system, a person must manually generate a classification system into which the one or more pieces of textual data in the database may then be manually or automatically classified. Thus, the typical classification system is manually generated and does not use the extracted phrases from the documents in the database to create the classification system. Therefore, the typical classifications are generally very broad reflecting the inability to more accurately classify the documents since the exact contents of the documents are not known. Thus, the typical classification may permit the user to select only a broad category which probably then contains too many documents to easily review. In addition, for each different database, a new classification system must be manually generated which is slow and time consuming.

[0006] Thus, it is desirable to provide a taxonomy generation system and method which solves the above problems and limitations with conventional classification systems and it is to this end that the present invention is directed.

[0007] A system and method for generating a taxonomy in accordance with the invention is provided in which the phrases extracted from the plurality of documents or pieces of textual data stored in the database may be used to automatically generate a taxonomy for the database. The generated taxonomy is unique to the particular database (since it is based on the phrases extracted from the database) and permits a searcher in the database to rapidly find documents within the database since the taxonomy has more detail than a typical classification system. Thus, a search through the taxonomy may reduce the number of documents which the user must review. In addition, the taxonomy generation system may automatically generate a taxonomy for any database provided that key phrases from documents within the database are available.

[0008] In more detail, a broad topical library is manually generated and used to provide a structure for the taxonomy. The topical library may be of the same broad level of detail as a typical classification system. Next, the phrases extracted from the documents within the database may be clustered and mapped to form maps illustrating the relationships between the phrases within the documents. The phrases in the maps which are connected to the most other phrases in the map, known as leaders, may be used to form the first level of the taxonomy underneath the topical index. Next, phrases which are related to the leaders based on the clustering form the next level of the taxonomy and so on until the taxonomy has been automatically generated. In this manner, the taxonomy generated reflects the actual phrases contained in the database so that it is more accurate than typical classifications.

[0009] Thus, in accordance with the invention, a system for generating a taxonomy for a database is provided including a database having a plurality of pieces of text and a plurality of phrases extracted from the pieces of text wherein the phrases are associated with one or more other phrases. A leader phrase from the phrases in the database is identified wherein the leader phrase is associated with a predetermined number of other phrases in the database. A first level of a taxonomy is generated based on the identified leader phrases wherein the leader phrases form a first level of headings in a hierarchical topical outline. A second level of the taxonomy is generated based on phrases in the database associated with the leader phrases wherein the phrases are sub-headings underneath the leader phrases with which they are associated. The generated taxonomy reflects the phrases extracted from the pieces of text in the database so that a user searches through the database using the final taxonomy.

[0010] FIG. 1 is a block diagram of a text processing system;

[0011] FIG. 2 is a block diagram of a taxonomy generation system in accordance with the invention;

[0012] FIG. 3 is a flowchart illustrating a method for generating a taxonomy in accordance with the invention;

[0013] FIG. 4 is a diagram illustrating a map containing clustered phrases;

[0014] FIG. 5 illustrates an example of a topical library;

[0015] FIG. 6 illustrates an example of a taxonomy generated in accordance with the invention;

[0016] FIG. 7 illustrates an example of a final taxonomy incorporating the taxonomy of FIG. 6 and the topical library of FIG. 4 in accordance with the invention;

[0017] FIG. 8 illustrates an example of a user interface for searching a top level of the taxonomy in accordance with the invention; and

[0018] FIG. 9 illustrates an example of a user interface for searching a lower level of the taxonomy in accordance with the invention and for displaying the documents relating to the selected topics.

[0019] The invention is particularly applicable to a system for generating a taxonomy from English language documents and it is in this context that the invention will be described. It will be appreciated, however, that the system and method in accordance with the invention has greater utility, such as to document in other languages and to any other types of pieces of text that may be stored in a database. To better understand the invention, a text processing system will now be described.

[0020] FIG. 1 is a block diagram of a text processing system 10. The text processing system 10 may include a parser 12, a clusterizer 14, a map generator 16, a database (DB) 18, a taxonomy generator 20 and a central processing unit (CPU) 22. The parser, the clusterizer, the map generator, and the taxonomy generator may all be separate software applications stored in the memory (not shown) of the text processing system 10 which are executed by the CPU 22. The text processing system may receive one or more pieces of text, such as stories, press releases or documents, and may generate a searchable taxonomy which may be searched by one or more client computers (Client 1, Client 2 and Client N). To generate the searchable taxonomy, each piece of text may be received by the parser 12 which processes each piece of incoming text and generates one or more key phrases for each piece of text which characterizes the piece of text. The key phrases may be stored in the database 18. Once the key phrases are extracted from each piece of text, the clusterizer 14 may generate one or more clusters of the key phrases based on the relationships between the phrases. The clusters generated may also be stored in the database 18. The map generator 16 may use the generated clusters in the database in order to generate a graphical map showing the relationships of the key phrases within the various pieces of text to each other so that a user of the system may easily search through the database of pieces of text by viewing the key phrases of the pieces of text. The clusters may also be used to generate the taxonomy in accordance with the invention as described below with reference to FIGS. 2-9. The database 18 may also store a topical library, as described below with reference to FIG. 5, which also may be used to generate the taxonomy. Additionally details about the clusterizer and the map generator are disclosed in co-pending U.S. patent application Ser. No. 08/801,970 which is owned by the assignee of the present invention and which is incorporated herein by reference. The text processing system may be implemented in a variety of manners including a client/server type computer system in which the client computers access the server via a public computer network, such as the Internet.

[0021] In operation, a client computer may access the text processing system 10 via a computer network, such as the Internet, using a browser application. The text processing system may generate a user interface showing the top level of the taxonomy previously generated. The user may then select a particular subject matter from within the taxonomy and another level of the taxonomy will be provided to the user. As some point, the user interface will also provide the user with the ability to look at documents which satisfy a given taxonomy classification. The interaction with the user will be described in more detail with reference to FIGS. 8 and 9. Now, the taxonomy generator 20 in accordance with the invention will be described.

[0022] FIG. 2 is a block diagram of the taxonomy generator 20 in accordance with the invention. The taxonomy generator 20 may include a taxonomy creator 30, a combiner 32 and the database 18 from the text processing system. The taxonomy generator 20 may receive the extracted phrases from the parser, the clusters from the clusterizer (both of which are stored in the database 18) and a topical library/index which may also be stored in the database 18. The topical library may be a high level classification of subject matter within a particular area such as might be available in a typical text processing system for searching through the document in the database. The extracted phrases and the clusters may be input to the taxonomy creator 30 which generates a taxonomy from the extracted phrases and the clusters. An example of the taxonomy generated will be described below with the reference to FIG. 6. The taxonomy from the taxonomy creator is output to the combiner 32 which then combines the taxonomy with the topical library to generate a final taxonomy. The final taxonomy may be searched by a user accessing the text processing system. Since the final taxonomy has been created from the extracted phrases and the clusters, the final taxonomy reflects the documents in the database more accurately than a typical classification system which has no association with the documents in the database. Now, a method for generating a taxonomy in accordance with the invention will be described.

[0023] FIG. 3 is a flowchart illustrating a method 40 for generating a taxonomy in accordance with the invention. The method begins at step 42 in which the clusters received by the taxonomy system are analyzed to identify the leader clusters. The leader clusters are phrases in a map, as described below, which have associations (as shown by the lines in the map) with a large number of other phrases. The leader clusters tend to be broader phrases than the less connected other clusters so that the leader clusters form the best broad subject matter classifiers for the documents in the database. The leader clusters may then be used to form the first level of the taxonomy in step 44. In particular, the leader phrases are arranged in a hierarchical outline format as shown in FIG. 6. Next, the phrases associated with the leader phrases (those phrases with links to the leader phrases) are identified in step 46 and the next level of the taxonomy is generated from these clusters in step 48. In particular, the phrases related to each leader phrase are arranged underneath the leader phrase in a hierarchical relationship to form more specific sub-headings. Next, the taxonomy generator determines if there are any more phrases which are going to be added into the taxonomy based on the complexity and depth of the map. If there are more clusters, the method returns to step 48 and generates the next level of the taxonomy. If there are no more clusters to process, the taxonomy generated is combined with a topical library in step 52 to generate a final taxonomy in accordance with the invention. Now, an example of generating a final taxonomy from a map showing one or more clusters of phrases and a topical library will be described.

[0024] FIG. 4 is a diagram illustrating a map 60 containing a plurality of clustered phrases. The map 60 may be generated as described in the above incorporated co-pending patent application. As shown, the map contains one or more phrases 62 whose relationships to other phrases are shown by a line 64 connecting the two phrases. As described above, the leader phrase clusters may be used to generate a first level of the taxonomy. In this example, the phrase fracture intervention trial may be a leader phrase since it has a relationship with a plurality of other phrases. In contrast, the phrase mission bay is not a leader phrase. In this example, a taxonomy is generated using only the phrases shown in FIG. 4 will be described. It will be appreciated, however, that the taxonomy normally includes a plurality of phrases. To generate the taxonomy, a topical library 70 as shown in FIG. 5 may be used to define the broad subject matter classifications of the topics. In this example, the sub-classifications underneath the fracture heading will be generated in accordance with the invention based on the phrases shown in FIG. 4.

[0025] FIG. 6 illustrates an example of a taxonomy 80 generated in accordance with the invention based on the leader phrases shown in FIG. 4 and in particular the fracture intervention trial phrase. In particular, the phrase is located within the taxonomy and includes three sub-categories 82 in this example. In particular, the sub-categories may include the phrase called alendranate, the phrase hip fracture and the phrase participant received alendronate. As shown in FIG. 4, these phrases are related to the leader phrase and therefore appear as sub-headings underneath the leader phrase. Similarly, the leader phrase hip fracture is identified from FIG. 4 as a leader phrase and there are one or more phrases underneath it in the taxonomy. The symbols prior to the phrase in the diagram indicate that the phrase is contained elsewhere in the taxonomy. Referring back to FIG. 4, each of the phrases underneath a leader phrase has a relationship with the leader phrase so that those phrases become sub-categories underneath the leader phrase.

[0026] FIG. 7 illustrating a portion of a final taxonomy 90 generated when the generated taxonomy shown in FIG. 6 is combined with the topical library shown in FIG. 5. As shown, the final taxonomy generated in accordance with the invention contains more detail than the typical topical library. In addition, the headings (leader phrases) and the sub-headings of the taxonomy are generated from the pieces of text in the database so that the headings and sub-headings more accurately reflect the pieces of text in the database. Thus, it is easier to locate documents within the database using the automatically generated taxonomy in accordance with the invention. Now, several examples of the user interfaces for searching through the taxonomy in accordance with the invention will be described.

[0027] FIG. 8 illustrates an example of a user interface screen 100 for searching a top level of the taxonomy in accordance with the invention. The user interface may include a keyword search portion 102 and a classification search section 104. The keyword search portion permits the user to enter keywords which will be compared against the taxonomy to generate a list of documents which match the keywords. The classification search portion 104 permits the user to browse through the high level classifications and click on any classification of interest. In this example, assume that the user clicks on the Accidents & Injuries sub-category underneath the Disease Areas heading. The user is then shown another user interface page which will now be described.

[0028] FIG. 9 illustrates an example of a user interface screen 10 for searching a lower level of the taxonomy in accordance with the invention. As shown, this screen may include a keyword search section 112 for keyword searching as above, a current search status line 114 which may list, for example, the current category within the taxonomy being searched and the path to get to that category, and a sub-classification section 116 for listing the sub-classifications and selecting them. In this example, the taxonomy does not have more specific classifications. Therefore, the selection of one or more sub-classification headings (a phrase associated with a leader phrase in this example) and the selection of a document button 118 by the user causes those documents within the selected one or more sub-classifications to be shown to the user of the system. In this manner, the use may rapidly narrow down the number of documents to be reviewed by using the more comprehensive taxonomy in accordance with the invention.

[0029] While the foregoing has been with reference to a particular embodiment of the invention, it will be appreciated by those skilled in the art that changes in this embodiment may be made without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims.