Method for automatic deduction of rules for matching content to categories

[0001] 1. Technical Field

[0002] This invention relates generally to computerized document management systems. It relates more specifically to methods for finding the correct place to store a new document in a strange taxonomy.

[0003] 2. Background

[0004] Computer storage of soft documents is ubiquitous in modern technological life. Documents are filed in the computer according to a filing scheme, or taxonomy, which may organize documents by a variety of criteria. A taxonomy may be for a complex relational database with millions of records or for a web site with hundreds of pages. New documents are frequently created which must be added to the existing taxonomy. For a user of the computer to file the document, the user conventionally must have knowledge of the taxonomy in which the new document is to be stored. The user then selects a category, or node, within the taxonomy based on the user's knowledge of that taxonomy and stores the document in the selected category.

[0005] Access to strange taxonomies carries with it a costly learning curve for each user, because computer storage taxonomies can be incredibly complex. Furthermore, both those who would store and those who would retrieve documents from a taxonomy must have knowledge of it. As documents are added to an existing taxonomy, new categories may be formed, requiring frequent updating of the user's knowledge.

[0006] Human users vary in their perceptions and so each individual may categorize documents differently. As a consequence, one user's classification may be confusing to the next user's searching method, making document retrieval inefficient or impossible.

[0007] Accordingly, what is needed is a way to store documents that does not require the users who store and retrieve documents to know the taxonomy in which the document is stored. What is also needed is a uniform and predictable storage method that makes searching for documents in strange taxonomies more reliable and, possibly, faster.

[0008] One aspect of the invention is a method of classifying document content within a taxonomy, the taxonomy comprising a plurality of categories in a computer document storage organizational scheme and a plurality of first documents tagged according to the taxonomy, each first document within the plurality of first documents classified within one category of the plurality of categories according to the taxonomy, the method comprising the steps of a) spidering both the taxonomy and the plurality of first documents tagged according to the taxonomy to generate pairings of each first document within the plurality of first documents with one category within the plurality of categories within the taxonomy; b) creating a rule generation document representing each of the pairings of each first document within the plurality of first documents with one category within the taxonomy; c) parsing a second document according to the rule generation document; and d) classifying the parsed second document into one category within the plurality of categories within the taxonomy.

[0009] Another aspect of the invention is a method for categorizing the content of a new document within a strange taxonomy, the strange taxonomy comprising a plurality of categories and a plurality of documents within at least one of the categories within the plurality of categories, wherein a root node for the taxonomy has been provided, the method comprising the steps of a) automatically spidering the taxonomy to identify each category among the plurality of categories and each document among the plurality of documents classified within each respective category; b) automatically forming pairs for each of the documents, the pair comprising one of the documents and the category within which the one of the documents is classified; c) automatically extracting keywords from each of the documents in each of the categories; d) automatically associating the extracted keywords from each of the documents within each of the categories with the category in which the documents are classified; e) automatically generating rules, each rule mapping at least one of keywords and patterns of keywords to the category in which the documents containing the at least one of keywords and patterns of keywords are classified; f) automatically parsing an unclassified document to determine new keywords therein; and g) automatically classifying the unclassified document into at least one of a new category and a category having documents containing at least one of keywords and patterns of keywords similar to the new keywords.

[0010] A third aspect of the invention is an apparatus comprising: a) at least one processor; b) a memory coupled to the at least one processor; b) computer-readable data storage media coupled to the at least one processor; c) a plurality of documents tagged according to a taxonomy, the documents residing in the computer readable data storage media, the documents comprising content, the content comprising keywords; and d) a rule-deducing content classification mechanism residing in memory.

[0011] A fourth aspect of the invention is a program product comprising: a) a rule-deducing classification mechanism residing in memory, the rule-deducing classification mechanism operable to spider the taxonomy and the tagged documents classified therein, to deduce rules for classifying documents within the taxonomy, and to classify a new document according to the taxonomy; and b) computer-readable signal bearing media bearing the rule-deducing classification mechanism.

[0012] A fifth aspect of the invention is a method of storing a new document according to a strange taxonomy, the method comprising a step of providing the new document and a starting point in the strange taxonomy to a rule-deducing document classification and storage computer program, the program automatically spidering the strange taxonomy and tagged documents classified therein, automatically deducing rules for classification of the new document, automatically classifying the new document according to the rules deduced, and automatically storing the new document according to the classification of the new document.

[0013] A sixth aspect of the invention is a method of classifying document content within at least one taxonomy, the at least one taxonomy comprising a plurality of categories in a computer document storage taxonomy and at least one first document tagged according to the taxonomy, the at least one first document classified within at least one category within the plurality of categories of the at least one taxonomy, the method comprising the steps of a) at least one of spidering and crawling both the at least one taxonomy and the at least one first document tagged according to the taxonomy to generate at least one pairing of at least one first document with the at least one category in which the at least one first document is classified within the at least one taxonomy; b) creating a rule generation document representing the at least one pairing of at least one first document with the at least one category within the at least one taxonomy; c) parsing a second document according to the rule generation document; and d) classifying the parsed second document into at least one category in the at least one taxonomy.

[0014] A seventh aspect of the invention is a method of finding documents in a computerized document management system, wherein the lost documents are lost to search engines because of incorrect filing, the method comprising the steps of a) retrieving each document; and b) saving each document under a new taxonomical root using a rule-deducing classification mechanism.

[0024] Referring now to the figures, FIG. 1 shows a diagram of a process 100 for an exemplary method for automatic deduction of rules for matching content. The process 100 begins when a new document is presented in step 102 to be stored in an existing document management system or other data management system having data content tagged according to a taxonomy. The existing document management system normally comprises at least one tagged document classified in at least one category. “Document” is used broadly herein to describe a discrete data structure having tagged content, or data. “Metadata” conventionally refers generally to information about the data, or information about the document content. “Metadata” also refers more specifically to information about the data, or information about the document content, that is contained or isolated separately from the content. Herein, the specific usage of “metadata” will be assumed unless otherwise described.

[0025] “Tags” refer to information about the content that is contained within and distributed throughout the content, and so represent a subclass of metadata in the general sense of the term. Tags are normally invisible to the document user but may be searched for or parsed. For example, a text document may have each keyword followed by or preceded by a tag that identifies a keyword, making it convenient to search for keywords in a document. Rather than search for each keyword in a long sequence of keywords, a search can be done for keyword tags, thereby finding all keywords in a single search pass. Tags rely on conventions for their effectiveness. Conventions for tagging documents is the target of significant standardization efforts, but tagging conventions still proliferate. Tags may comprise links. A particular tagging convention may be part of a taxonomy for a document management system.

[0026] A “taxonomy” is a systematic scheme for organizing discrete but related individual items, such as plant species or documents. For example, a taxonomy for documents may comprise a hierarchy of related categories, wherein each lower level of the hierarchy comprises subsets of the categories of the next higher level. A document is tagged according to a taxonomy when some tags contain information associated with a category in the taxonomy. For example, the tag could identify a keyword and give a category in which a document with such a keyword belongs. Such a system would usually result in oversimplification: in most document management systems, the taxonomy is quite complex and reference to more than one keyword is required for proper classification. For example, in even a simple hierarchal taxonomy, at least one keyword would be needed to make a choice at each level of the hierarchy. Hierarchies can be dozens or even hundreds of layers deep. In another example, the tagging convention could provide for a dedicated category tag or tags which could be quickly searched.

[0027] A “strange” taxonomy is a novel taxonomy wherein the user has no view of the internal organizational scheme. Thus, a strange taxonomy is a black box into which a user stores documents and from which a user retrieves documents. Any taxonomy may be made strange by the use of the present invention. An advantage of a strange taxonomy is that documents are filed according to predictable computer algorithms, and open taxonomies are filed according to human perception. Predictable filing enables predictable searching.

[0028] “Keyword,” as defined and used herein, refers to individual keywords and to key phrases. “At least one keyword” may be an individual keyword, a keyword phrase, or a pattern of keywords. Patterns of keywords are collections of keywords associated with a particular document or a particular category.

[0029] Step 102 presents a new document for classification to a document management system having documents tagged according to the taxonomy of the document management system. In response to the presentation step 102, step 104 automatically spiders the existing taxonomy to determine its categories and to identify documents within each category. A document may be identified within a category by storage location, metadata, or tagging. Spidering, and a functional variant thereof, crawling, are known in the art. The spider is a program that searches along a sequence of links between documents to map a taxonomy. Different types of taxonomies, such as web sites and databases, require spiders specifically adapted to the particular type of taxonomy. Spiders can provide a wide variety of information. The spider of step 104 returns pairings of each document with its category or categories. Conventionally, each document is classified in only one category, but not in all cases. In a hierarchal taxonomy, a document may be classified in all of the categories between its immediate category and the topmost, or root, category. If the spider finds no documents in step 104, the keywords of the new document may be identified by grammatical, lexicographical, syntactical, or similar analysis, tagged, and a new category may be created for the new document.

[0030] Next, step 106 creates a rule generation document, which be discussed in more detail infra under FIG. 3. The document-category pairs found by the spider are used as a basis for making classification rules. Each document in a category is analyzed by an analysis technique appropriate to the application to extract keywords and key phrases. Mappings from keywords to categories are thereby formed, and the rule generation document records the rules that represent or are derived from these mappings.

[0031] The next step 108 in process 100 (FIG. 1) is to parse the new document according to the rule generation document 402 (FIG. 4). It may be wasteful to search the new document for every possible combination of keywords when only particular keywords or particular combinations of keywords will uniquely identify a category. The new document may not be tagged according to the taxonomy. Accordingly, the new document is first parsed in step 108 to find only those keywords that contribute to uniquely identifying a category: the keywords in the rule generation document. If such a pattern of keywords is found, then a category may be uniquely identified and step 108 is complete. If relevant keywords are found, but several categories are still possible, more steps are required. In alternate embodiments, the new document may be searched for all keywords initially or after a failure to find keywords from the rule generation document in the new document. All of the keywords may be used in forming a new category.

[0032] In step 110, the parsed new document may be automatically classified according to the strange taxonomy. Classification may include automatically tagging the document according to the taxonomy. Automatic tagging may involve inserting keyword tags associated with the new document's keywords which were used for classification within the taxonomy. In some embodiments, it may further include one or more category tags. In some embodiments, classification may further involve tagging additional keywords found by grammatical, lexicographical, syntactical, or similar analysis of the document. In some embodiments, automatic classification step 110 may be accomplished by a pre-existing classification engine. In some embodiments, classification may include storing the document in storage location associated with the category. In other embodiments, classification may be accomplished by additions to the metadata of the document. At the end of step 110, the process starts again at step 102 with the presentation of the next new document.

[0033] FIG. 2 depicts a second exemplary embodiment 200 of the method for automatic deduction of rules for matching content which employs a dual spider approach. In step 212, a first spider, specifically adapted for a document management application, spiders the taxonomy and records the taxonomy, including the categories and the documents in those categories, in a map document. The map document may be an XML document. XML provides a common format for the output of various specifically-adapted spiders. A new map document may be made in step 212, perhaps by editing the existing map document, after each new document is added. Each new classified document has the potential to change a decision about the classification of the next new document. In a document management system having only one source of new documents, automatically editing the map file with data on the new document is acceptable. In a multiuser system, updates should be made after each new document is added. For small multiuser systems, a single map document may be used and edited. For large multiuser systems, multiple map documents may be used. An advantage of the dual-spider approach is that the map document may be used by software searching for a stored document. At the completion of step 212, the process 200 loops back to receive the next new document in step 102. Step 102 may be the same as step 102 in process 100.

[0034] In step 204, a second spider, adapted to spidering the map document created in step 212, spiders the map document in response to being presented with a new document to classify from step 102. Spidering the map document in step 204 may be extremely fast, allowing classification decisions to be made rapidly. In an embodiment, the map document may be spidered to support a search for a classified document. Steps 106, 108, and 110 may be the same as in process 100.

[0035] FIG. 3 shows exemplary steps within step 106, wherein a rule generation document is created based on document-category pairings returned 302 by the spider. The rule generation document holds rules which map keywords or patterns of keywords to categories. The next step 304 extracts keywords from the documents returned by the spider. The spider may include links to the documents in the document-category pairings received in step 302. The extraction of keywords may comprise reading keywords from metadata, reading tags and their associated content, and/or parsing the document for keywords based upon grammatical, lexicographical, syntactical, or similar rules. Step 306 associates the extracted keywords with the categories of the document-category pairings.

[0036] Step 308 simplifies these potentially complex results into a series of one-to-one associations. For example, if a keyword is associated with ten categories, ten one-to-one associations may be formed. From these one-to-one associations, patterns of keywords may be discerned and mapped to categories. The identification of patterns of keywords uniquely identifying categories may comprise part of rule generation 310. Identification of unique keywords that uniquely identify categories may comprise another part of rule generation 310. The rule may be simply: “Keywords A, B, and not C means category 21,” expressed in a markup or similar language or structure. Any rule that maps from one or more keywords to a category is contemplated by this invention. For example, a rule may incorporate more than the mere existence of a keyword by weighting a keyword in a document title more than a keyword in a document abstract which, in turn, is weighted more than a keyword in the document body which, in another turn, is weighted more than a keyword in a footnote.

[0037] The rule generation table represents the document-category pairings as document-keyword-patterns-to-unique-category pairings.

[0038] FIG. 4 shows exemplary steps for the case when a unique category identification cannot be made from the keywords occurring in the new document and the rule generation document. Steps 402 and 403 are the basic steps of parsing the new document 402 and associating categories with the keywords found in the new document 404. Step 405 inquires if step 404 results in a unique classification and, if so, ends step 108. If not, the categories associated with even one keyword are scored according to a predetermined scoring criteria in step 408. The predetermined criteria may comprise similarity to at least one pattern or part of a pattern of keywords associated with a category, frequency of keywords in a category, commonality of keywords among documents in a category, absence of particular keywords among documents in a category, uniqueness of documents in a category, and the like.

[0039] If a category with a unique maxima score (or unique minima score, depending on the scoring scheme) emerges, that category may be determined in step 408. If a category is the only category above (or, depending on the scoring scheme, below) a threshold score, that category may be determined in step 408. If the scoring in step 406 leaves more than one category as a possibility, the document may be classified in each remaining category or none in step 408. Alternatively, the first-in-time acceptable score may be chosen. If it is determined that no categories score well enough to form a basis for classification or if multiple categories are indicated but multiple classifications are ruled out, or if no keywords are matched (step 718, FIG. 7) then a new category may be formed in step 728 (FIG. 7). The category may be based on keywords found in the new document which do not match those in the rule generation document.

[0040] FIG. 5 depicts a third exemplary embodiment 500 of the method for automatic deduction of rules for matching content. Embodiment 500 comprises a method for automatically deducing rules for classifying documents within a plurality of taxonomies that are not necessarily arranged in a hierarchy. Step 504 initiates in response to the presentation of a new document for classification in step 102 and spiders all the taxonomies available. Step 102 may be the same as step 102 in process 100. In step 506, a single rule generation document is created which includes rules for the categories in all of the available taxonomies. Step 108, parsing the new document, may be the same as step 108 in embodiment 100. When the rules are applied in step 510 to classify the new document, the process 500 may inherently choose the best taxonomy for the document, wherein the best taxonomy may be the one with the best-fitting category. Using embodiment 500, a user presenting an unread document can learn a great deal about its content. In an alternate embodiment, the step 510 may return a notice to the user as to which taxonomy was selected, either for user approval or open loop. At the end of step 510, the process loops back to step 102 to receive the next new document.

[0041] FIG. 6 depicts a fourth exemplary embodiment 600 of the method for automatic deduction of rules for matching content. Embodiment 600 comprises a method for automatically deducing rules for classifying documents within a plurality of taxonomies that are arranged in a hierarchy. The approach is a level-by-level approach. Responsive to the presentation of a new document for classification in step 102, step 604 spiders down one level in the hierarchy. Initially, this is from the root to the first level. Any level may have both additional taxonomies and categories containing documents. Based on document-taxonomy and document-category pairings acquired by the spider, a rule generation document is created in step 606 for selecting either a category or a taxonomy which may be on the lower, spidered level. Step 108 in embodiment 600 may be the same as step 108 in embodiment 100. In step 610, the new document, parsed in step 108, is classified into either a taxonomy or into a category. Step 611 enquires as to whether the classification effort has reached the lowest level needed. If a suitable category has been found, the answer is YES, and the new document is classified into the suitable category in step 612. If the level is not the lowest level, and no suitable category has been found, the answer is NO, and the next level is spidered in step 604. On any level that has been reached, where no suitable category or taxonomy has been found, the failed-search steps of embodiment 100, discussed under FIG. 4, above, or the failed search steps 718–728 of embodiment 700, discussed below under FIG. 7 below, may be incorporated. After classification of the new document in step 612, the process 600 loops back to step 102 to receive the next new document.

[0042] FIG. 7 depicts a fifth exemplary embodiment 700 of the method for automatic deduction of rules for matching content. Steps 102, 104, 106, and 108 may be the same as the like-numbered steps in embodiment 100. Embodiment 700 comprises steps 718 and 728 for responding to a failed search for keywords in the new document, which new keywords match keywords in the rule generation document. Step 718 tests for failure, and, if no keyword matches are found, directs the generation of a new category in step 728. Other failure criteria may be added or substituted for the one illustrated in step 718. For example, failure to achieve an acceptable score in step 406 (FIG. 4) of embodiment 100 (FIG. 1). If the reason for the search failure is that there are not yet any documents classified within the taxonomy, creating a new category is still the correct response to the failure. The keywords in the very first document classified into the taxonomy must be found by grammatical, lexicographical, syntactical, or similar textual or data analysis or must be pre-tagged. Thus, a new taxonomy can begin from a blank root node.

[0043] FIG. 8 depicts an apparatus 800 for implementing an embodiment of the method for automatic deduction of rules for matching content. Apparatus 800 comprises a central processing unit, or processor, 802 coupled to a memory 804 by bus 860. Also coupled to the processor 802 by bus 860 and storage interface 806 is a computer-readable data storage device, or Direct Access Data Storage Device (DASD) 890. Conventionally, the documents classified within the taxonomy are physically located on the DASD 890. In some embodiments, the classified documents may be physically located on a removable data storage medium such as disk 895. In other embodiments, the document and taxonomy information may be available from computer-readable signal-bearing media through an interface to bus 860.

[0044] Program 824, resident in memory, may have been loaded into memory from DASD or from computer-readable signal-bearing media through an interface to the bus 860. Program 824, the rule-deducing document classification mechanism, is a software instantiation of the method for automatic deduction of rules for matching content. Program 824 responds to the presentation of a new document in step 102 (FIG. 1) by classifying it within a strange taxonomy known to the user only by a root node or alias for the root node. Presentation of the new document may be by execution of a conventional or dedicated “save” command. The program 824 may be part of a larger program operative to create and save documents. Alternatively, the rule-deducing document classification mechanism 824 may incorporate automatic saving after document classification is complete.

[0045] Using apparatus 800, a document author, database administrator, document management system administrator, or other user merely identifies a taxonomy or a group of taxonomies and saves the document. All details of location and classification within the taxonomy may be hidden from the user. New users do not have to learn the taxonomy in order to classify or retrieve documents. The classification is uniform and therefore tractable to search engines. Documents do not become unretrievable by search engines because of mis-classification.

[0046] A legacy document management system, comprising many mis-classified documents, may be reformed or refurbished by retrieving each document and then saving each legacy document (through program 824), in turn, under a new root node. In an embodiment, retrieval of the documents may be based upon storage location rather than conventional search criteria, in order to obtain the documents that conventional search engines miss. One or more faux documents may be seeded under the root node to provide an initial structure, if desired. If grammatical, lexicographical, syntactical, or similar analysis of documents is used to identify keywords, even untagged legacy documents will become properly tagged and classified according to the taxonomy. Documents lost by mis-filing in a document management system may thus be found.

[0047] The foregoing description has described selected embodiments of a method for automatic deduction of rules for matching content.

[0048] While the invention has been particularly shown and described with reference to selected embodiments thereof, it will be readily understood by one of ordinary skill in the art that, as limited only by the appended claims, various changes in form and details may be made therein without departing from the spirit and scope of the invention. For example, many of the special features of each exemplary embodiment may be incorporated in other embodiments.