Multiparameter indexing and searching for documents

[0001] This application claims priority from U.S. Provisional Patent Application No. 60/449,227, filed on Feb. 21, 2003, the contents of which are incorporated by reference to the extent necessary for proper understanding of this disclosure.

[0002] It is well-known to search through databases of documents using content-based, text searching. Many Internet-based search engines, such as Google, enable content-based searching using proprietary searching techniques and algorithms. There are also several products focused on the legal space that employ content-based search techniques, including products with trade names such as Lexis and Westlaw).

[0003] Another common technique for searching through databases of documents is to use content-based text searching in conjunction with pre-defined categories. Examples are document management systems, including those with trade names Documentum, iManage or DocsOpen. Those systems include databases with profile information about documents, which enable users to search for documents using a combination of category and text based searching. These existing systems, however, typically only include metadata about documents that is either (i) pre-set properties (such as who created the document based upon system login information) or (ii) information that is user-supplied.

[0004] The present technique teaches a multiparameter document categorization and search technique. According to aspects of this system, the information to be searched, herein called documents, are specially indexed using an abstract creation engine running on an abstract creation computer, that may employ a series of rules-based components to populate a database automatically with information about such documents. The engine categorizes documents according to both objective and subjective criteria according to a set of rules. The engine also employs content-based document abstracting, to enable searching through a combination of full-text, content-based information and detailed abstract information. This application also discloses project-based organization and retrieval of procedural information.

[0005] These and other aspects will now be described in detail with reference to the accompanying drawings, wherein:

[0006] FIG. 1 shows a block diagram of the abstract creation engine and computer;

[0007] FIG. 2 shows a diagram of the searching using the specially created abstracts in combination with content-based, text searching and incorporated workflow content; and

[0008] FIG. 3 shows a process flow for a specific rule set.

[0009] The embodiment describes a document indexing and searching system. According to the present system, documents are analyzed according to a set of rules, and abstract files are created relating to contents and categories of such documents. The abstract files may be searchable files relating to contents of the documents. Searches can be carried out among the categorized documents. The search may therefore produce more pinpointed results. In an embodiment, the abstract files may be in markup language, e.g., XML, or Xtensable Markup Language, HTML, or any other markup language.

[0010] As described above, the term document(s) is used to refer to any source of information. The documents may be actual documents created by users, or published documents such as books, magazine articles, treatises, or publicly available information sources. In one aspect, the system is optimized for use by legal professionals, and therefore the documents may be legal documents, collections of statutes and rules, legal treatises, and other similar legal documents. However, the system is not limited to being used with legal documents, and in an alternative embodiment, the system is used to abstract documents which are not necessarily legal in nature.

[0011] A block diagram of the basic document indexing system is shown in FIG. 1. Multiple types of documents, shown as 102, 104, 106, are input into the Abstract Creation Computer 110. The Abstract Creation computer 110 may include an operator interface with a number of operator controls shown as 112, and may automatically create abstracts of the input documents.

[0012] Initially, an input sorter shown as 120 collects the different kinds of documents, which documents can be in any of a number of different formats. The input sorter may include an interface to a scanner, and also a port for receiving other kinds of documents. The sorter may accept documents in multiple different formats, such as Microsoft Word documents, documents in XML or HTML, imaged documents (e.g., pdf, TIFF), or other formats. The input sorter investigates the format of the incoming information, and converts it to an acceptable format. For example, if the input format is in an image format, then the sorter 120 may optically character recognize certain text within the image, and create an XML document based on the optically recognized image. The converted document, available at 122, is input to the abstract creation components running within the abstract creation computer computer 110.

[0013] This abstract creation computer 110 may be formed in any kind of computer, preferably a server running Windows 2000 Server

[0014] The abstract creation components analyze the documents, categorize the documents, and publish information about the documents. An abstract about each document is created in a searchable format. In an embodiment, the abstract is in XML format. The abstract is created in a memory module 120 that is associated with the computer 110.

[0015] A number of interconnected programs and program modules capture and interpret data about each document. The components are discussed below in further detail.

[0016] Prior to processing the documents, the presort module 130 may sort the documents into high-level categories depending on configurable criteria. The presorting may operate according the flowchart of FIG. 3. This module may also segregate documents into particular groups depending upon file size and number of characters based upon configurable criteria, or business rules Business rules is a generic term for domain-specific rule sets. For example, if a title includes the word Complaint, the document may be of type COMPLAINT. The system can then use these rules, in conjunction with rules to determine the document's legal type category. As an example, the rule can read IF FIND COMPLAINT, AND ALSO FIND ANSWER, THEN ANSWER OVERRULES) to categorize information.

[0017] At 300, the module acquires documents. As discussed above, this may include obtaining the document in either electronic or image form, from any source. At 302, the documents are filtered based on size. Any document less than a few lines could be assumed to have minimal useful information, for example.

[0018] The documents are then initially sorted, based on title or the like at 304. For example, in this embodiment, the documents can be initially sorted according to whether they relate to deals or other general categories (DealBank), to litigation (LitBank), or are letters/memos (MemoBank) Documents should be further sorted into document types, if known. In an embodiment, the high-level categories may include documents created by lawyers, local rules, state rules, federal rules, publicly available information sources, treatises and other publications, and other similar document categories. The user can select any one of these multiple categories.

[0019] The documents are then further filtered according to custom criteria at 306. File naming conventions and other metadata available in document management or file storage systems are evaluated to identify documents that might not be included in further processing. For example, documents might have a file name of junk or do not use.

[0020] Known metadata about the document is saved to a file related to the document known as a Document Abstract Specification (DAS) file at 308. A query of an existing document management system, for example, can produce a report of the metadata that the system stores about the document. This information, such as title, author, and client matter number can be associated with the document through its DAS file.

[0021] This is followed by the documents being converted to a common format, e.g., XML or text form, at 310. The system may alternatively convert the documents to one or more of HTML, DOC, XML, or TXT. This allows the same tool to be used in the conversion of SmartRules and SmartRules Citations.

[0022] The documents are again filtered at 312 to create classes of documents that are based on the total amount of text. Some documents may pass the minimum file size threshold at 302 due to objects such as charts, logos, and graphs within the file. Nevertheless, these files may not contain sufficient useful text to be used as part of the system. For example a letter with a logo in the header could say simply Attached please find a copy of your Employment Agreement. Such a document might not be desirable in a searchable document collection, and may be segregated by this component depending upon configuration settings. 312 may be optional, and an alternative could use the original size filter at 302 by checking the character count on the Properties Sheet within the file itself, to determine file size threshold.

[0023] The documents are sorted into folders at 314. For example if two folders of agreements that have been converted are to be merged, the txt and junk subfolders should be merged below the newly created folder. Finally, the documents are submitted for further processing at 316. Folders that have been converted and cleaned may optionally be submitted to the creation computer recursively. For example the tool can be instructed to process a folder called Deal and to process all of its sub directories.

[0024] As described above, the documents are processed to recognize and extract both objective data and subjective data. 140 represents the objective data extraction engine. This may be based on both system wide categories and also on user selected categories. For example, for a lawyer-created document, objective information may include lawyers listed on the document, a court of filing, and other information which can be determined from the document.

[0025] Lists of different allowable categories may be maintained to determine this information. For example, in order to determine the lawyer associated with a document, a list of possible lawyers could be maintained. Objective data abstractor 140 compares the contents of the document with all the possible lawyer names. If any of those lawyer names are found, then the document is categorized with that lawyer name. This avoids obtaining names that are not actually lawyer names, such as plaintiff/defendant names, typists' names, and the like. Alternate ways of determining lawyer names may look for certain lawyer-indicating terms, such as Esq, or LLP, and add the names with a specified relationship to those terms to the database of lawyer names used in the searching.

[0026] Similarly, objective data abstractor 140 may maintain a list of all possible court names. The user can select other categories and add or remove names as necessary. This may be used to determine the court name within the document.

[0027] More generally, the objective data abstractor determines objective information from the document, that is, a specific type of information such as a specified type of name. The objective data abstractor also rejects other information based on context within the document.

[0028] The subjective data abstractor 145 includes software that recognizes, analyzes and extracts subjective data from the file, again based on input characteristics and business rules. Subjective data may include information such as a legal task associated with the document; e.g., is it a complaint; a motion for a preliminary injunction; a patent application; or the like. This is done using rules that analyze the content and layout of a document based on specified criteria. For example, a document maybe categorized as a complaint based on its layout and contents. This is interpreted by a component that applies a series of rules to interpret the layout and contents of the document, and identify the applicable categories that apply to the document.

[0029] Another category of subjective information may be the document's objective, i.e., what is the document designed to achieve, or other subtype classification. Again, as above, this is defined in terms of rules which query document characteristics to determine the document's objective or subtype. One objective item may be whether a specific point of law is being urged. Another item of objective information may be substantive principles that are addressed in the document.

[0030] More generally, therefore, the subjective abstracter determines information categories within the document, rather than specific information of a specified type.

[0031] Module 150 refers to the iterative processing unit, which is a series of software instructions that analyze documents and compare data extracted from a document to known values in a database, in order to draw conclusions about the document being processed. For example, the document may be associated with a group of other documents, and information about those other documents may be known. Additional data about such document may thus be derived based on the data relating to other documents in the database. The system can automatically reprocess the documents that have already been processed, if specified required data fields have not been extracted. For example, additional information about documents obtained after the document has been processed may enable a previously-unidentifiable category to be determined. The reprocessing mechanism typically will not change any assigned category. If the document has not initially been categorized with a document type, then the document may later be re-categorized when it is determined that the document looks like a complaint, based upon what the system has concluded about other documents that were complaints. Analogously, once an attempt to extract all of the objective and subjective data has been made, the iterative processor re-processes the once-categorized document, to see if these additional rules enable improved interpretation of the data.

[0032] 155 represents a domain specific ruleset, which may be used to provide rules which are specific to a particular application of the Abstract Creation Computer (e.g., the legal industry as one example). A rules composer 160 may allow the user to create, view or modify rules for interpreting the data points that have been extracted or analyzed by the system.

[0033] 165 represents a component extractor, that segregates the documents into distinct sub-parts according to a configurable rule set. For example, this may parse a document into its individual clauses, which are separately saved to the database. Multiple sets and subsets may be created for each application.

[0034] 170 relates to a full-text indexer, which indexes the documents to allow content-based, full text searching. This may use any existing tool known in the art.

[0035] 175 creates hypertext links within the documents. This may include a rule set that recognizes internal references to various data according to specified formats and automatically generates hypertext or other links to data that resides inside or outside the system. For example, this may recognize cites to various statutes, and create a link to either an Internet site hosting the statute, or to a document which includes the statute rules within the database.

[0036] The operator controls 112 may enable the operator to create, modify or view business rules, and adjust rules and thresholds. The operator can also view the processing results and edit them, publish and take other actions in accordance with the system and permissions, set and adjust privileges and permissions for users on the system, as well as monitor usage and create and manage the user groups.

[0037] The preferred output from the system is in XML format. The XML abstracts may include merged results from all the extractions, as well as metadata that has been created from the extractions. The XML abstracts are stored in storage 180 along with the original and converted versions of the document.

[0038] An important feature of this system is the ability to create a detailed abstract file about each document in a database. In use, the system might be used within a law firm, and applied to documents within the law firm's database. The Abstract Creation Computer 110 creates this abstract file (Document Abstract Specification file), which is formed of known metadata extracted from the file properties, the document management or file store, and metadata generated by its own component processing. This metadata information can then be searched. These categories may include Tasks to which document relates (generally, a document's high-level Type, the objective of the document, authors, parties, substantive areas, legal topics and concepts, jurisdiction, court, judge, dates, governing law, contents of clause titles or body, unique identifier in document storage systems, associated client numbers, as well as content-based full-text.

[0039] The categorized documents can be searched according to the searching engine shown in FIG. 2. Importantly, the system uses a multiple data point searching tool, shown as 200. The users can search according to any criteria or combination of criteria that has been discussed and extracted, stored or generated according to any of the Abstract Creation Components 100 noted above. The user interface may allow the user to select one or many of these documents, based on one or many criteria.

[0040] Once the search characteristics are selected, 210 enables processing the search criteria by interpreting the criteria and conducting numerous searches across the multiple databases for relevant results. This component searches for documents matching search criteria, and may incorporate in search results other information that may be related to the user's likely task, including project-based procedural guides.

[0041] The processing obtains not only the exact results as requested, called herein explicitly requested results, but also uses its own internal rule set to obtain documents which may be relevant according to the rules even if not explicitly requested. One aspect of the internal rule set is a built-in legal thesaurus, which automatically searches for synonyms for a specified word in its context. The rule set-determined-results may use domain specific taxonomies that are based on project related concepts, for example document type and objective.

[0042] The results are displayed on a user interface 220 which shows viewing, sorting and manipulating search results. This interface integrates the results of the searches across the various databases. According to an aspect of this user interface, the search results are created and displayed in a way that allows a user to peer within parts of the document. For example, the search results may be displayed showing an abstract of the document, including the reasons why the processing engine 210 determined that the document was relevant. This tool is labeled the document abstract tool, and enables the users to obtain increasingly detailed descriptions of the search results prior to opening the individual result. The initial part enables viewing information about the document, example title, jurisdiction, parties, other relevant information. Clicking on the document brings up a window showing other relevant information about the document, for example substantive legal areas, (example trademark, copyright) with each substantive legal area allowing a drill down to create more information about that legal area.

[0043] For example, clicking on TRADEMARK may bring up the different sub categories within trademark which are discussed, such as dilution, or registration.

[0044] Another aspect of this system includes a special-purpose application 230. One such special-purpose application is the Smart Rules application which is a tool that organizes, compiles and presents legal research in a project specific approach. This goes against the usual technique of organizing the information by source, in favor of a new technique that favors organization according to its relevance to a users' anticipated project.

[0045] For example, a user may specify a specific type of legal activity or document, and in return receive rules, codes, laws and editorial information that would be relevant to that type of document or project, regardless of the original source of that material, in a single search. The search results may also include narrative information about the rules, codes and laws, as well as hypertext links to the specific sources either inside or outside the database system.

[0046] The management and publishing of the SmartRules system may be facilitated by the Abstract Creation Engine running on the Abstract Creation Computer. The Abstract Creation Engine may create hypertext links in editorial content to link that content to information in other parts of the database or on the internet. This can be done manually by creating abstracts for each of a plurality of anticipated topics. Alternately, this may use the Abstract Creation Computer on each of a number of different sources of information to automatically create this information.

[0047] The user performs a single search describing the activity and the court, and this delivers relevant rule parts, and also checklists and other information. The SmartRules can be pre-compiled, for each of multiple documents, courts, and jurisdictions based on the Abstract Creation Engine.

[0048] Using an example of the SmartRules system, a user may input criteria indicating a project concerning a Complaint for the United States District Court for the Central District of California. The SmartRules system returns a collection of information including those things which are necessary to comply with procedural and court rules, as well as editorial content and practice information, in a single search. The returned information may include state rules and local rules referenced in the editorial content, links to underlying rules and statutes or other sources, and may include information from external sources such as treatises, about the subject. The returned information may also include court specific rules, judge specific rules, and state or federal regulations or rules and related information. This compares with existing search systems which are organized and used according to the source of information, not by user task.

[0049] The information which is returned is categorized. The categorized information includes categories such as timing of the complaint, specific rules about the complaint such as page limits, fonts and the like, form and format of the complaint, information about how to introduce things into evidence, and other such information related to that activity. Also, users may do a content-based search in SmartRules, so that a user may obtain all results that address a certain statute, or other text based criteria.

[0050] Each section may include links to the actual rules and statutes, so that the user can click on a link and view the actual rule and/or statute within a separate window.

[0051] Another special-purpose information that forms a part of the user interface 210 is a document component search tool, which searches for common documents components across the individual documents or files that is enabled by the components extractor 165. This enables users to search for individual sub-parts of documents or files, that have been identified in advance by the component extractor.

[0052] The end user interaction tool 240 allows the end-users to obtain more information about the search results, and also allows users to designate part or all of the search results for classification in user-defined classification systems called Folios.

[0053] As described above, extraction of each of a plurality of fields occurs according to rules that are written to extract the data from those fields. Certain rules and their functions are described herein in further detail, to illustrate the concepts. However, it should be understood that these rules merely illustrate the concepts of using rules; and that other rules may be and are used. In each of these examples, information about the document is found by looking for clues within the document, and extracting the information from the document itself. The determination of document types may cause execution of different rules and rule sets are used for the different high-level document types. For example, a document which is categorized as a litigation document may have title, counsel name, and parties extracted in a different way than a document that is classified as a deal document

[0054] Counsel (For a Deal Document)

[0055] For extraction of counsel, a database of counsel names may be maintained. This information may also be obtained from text-based indicators in the documents (such as term LLP, or obtained from document management system or storage systems.

FOR EACH RULE IN THE RULES FILE REPEAT THE
FOLLOWING:
FOR EVERY MATCH IN THE DOCUMENT DO
RETRIEVE THE STRING THAT MATCHED THE FIRST
SUB-EXPRESSION S1(;
RETRIEVE THE STRING THAT MATCHED SECOND
SUBEXPRESSION S2;
COUNSEL S1 S2;
STORE THE COUNSEL IN THE LIST AND CONTINUE WITH
NEXT MATCH;

[0056] Example with a copy to:

[0057] Shook, Hardy & Bacon L.L.P.

[0058] Rule:

[0059] withs*as*copys*tos*:(.*)(LLPP.0,1C.0,1L.L.P. P.A.)

[0060] In the example above, the regular expression matches this string. The first subexpression matched is Shook, Hardy & Bacon and the second sub-expression matched is L.L.P. Either one will allow a match. In this case, the regular expression has 2 subexpressions.

[0061] Note that the same or different rules can be used to extract counsel from a non-deal document. Since different documents look different, a rule may be specially written to deal with the different place that the information might be.

[0062] Date

[0063] The data rule operates as follows:

[0064] Extract first few lines in the document to limit the date search.

[0065] For each rule in the DateRules File, repeat the following steps until a match is found or rules are exhausted.

	IF MORE THAN ONE EXPRESSION MATCHES RETURN
	ERROR.

[0066] If a match is obtained, extract the date until the string ending with 4-digit year using regular expression.

	CLEANSE THE DATE EXTRACTED BY REMOVING
	LEADING AND TRAILING SPACES,
	NEW LINES ETC. ELIMINATE UNWANTED WORDS AND
	CHARACTERS FROM DATE STRING.

[0067] e.g.: AGREEMENT AND PLAN OF MERGER (this AGREEMENT), dated as of Jan. 22, 2001, by and among Corning Incorporated, . . .

[0068] Matching Rule: (Dateds*n*ass*n*ofs*n*(the))

[0069] The above rule gets matched for the given example and the matched string will be dated as of, so that the date is after the string. To extract the date, another rule can be applied such that everything after the matched string until the four digit number, providing: Jan. 22, 2001.

IF NO MATCHES, NEXT RULE:
FOR EACH RULE IN THE DATECLAUSE RULES
FILE REPEAT THE FOLLOWING STEPS UNTIL A MATCH IS
FOUND OR RULES ARE EXHAUSTED.
IF A MATCH IS OBTAINED, EXTRACT THE DATE
UNTIL THE STRING ENDING WITH 4-DIGIT YEAR USING
REGULAR EXPRESSION.
CLEANSE THE DATE EXTRACTED BY REMOVING
LEADING AND TRAILING SPACES, NEW LINES ETC. ELIMINATE
UNWANTED WORDS AND CHARACTERS FROM THE DATE
STRING.

[0070] e.g.: PLAN EFFECTIVE DATE AND SHAREHOLDER APPROVAL. The Plan has been adopted by the Board effective Jan. 8, 1997, subject to approval by the . . .

[0071] Matching Rule:

[0072] (PLANs*n*EFFECTIVEs*n*DATEs*n*ANDs*n*SHAREHOLDERs* n*APPROVAL)(.*)effectives

[0073] HERE THE EXPRESSION MATCHES UNTIL . . . BOARD EFFECTIVE AND THEN THE SAME DATE RULE WILL BE APPLIED AS IN THE ABOVE CASE TO EXTRACT THE DATE PART.

[0074] Title

[0075] Title extraction may use multiple different rules. The basic approach is:

SKIP ALL EMPTY AND BLANK LINES.
EXTRACT FIRST FEW LINES IN THE DOCUMENT TO LIMIT SEARCH.
SKIP ANY TITLE HEADER IN THE DOCUMENT USING THE RULES DEFINED IN
TITLEHEADERLIST.TXT
FOR EACH RULE IN THE TITLERULES FILE, REPEAT THE FOLLOWING STEPS UNTIL A
MATCH IS FOUND OR RULES ARE EXHAUSTED.
IF THERE WAS A MATCH EXTRACT THE MATCHED STRING.
CLEANSE THE STRING AND CHECK FOR NOISE WORDS USING RULES DEFINED IN
TITLENOISEWORDS.TXT
IF TITLE EXTRACTED MATCHED NOISE WORDS SKIP AND CONTINUE TO SEARCH.
ELSE CLEANSE THE EXTRACTED STRING BY REMOVING UNWANTED NEW LINE AND
WHITE SPACES.

[0076] e.g.: INCENTIVE COMPENSATION PLAN

[0077] 1. Purpose. The purpose of this Incentive Compensation Plan (the Plan)is to assist Lincoln National Corporation, an Indiana corporation . . .

[0078] In the example above the first title rule matches INCENTIVE COMPENSATION PLAN which is all in caps.

[0079] Another rule can simply look for words in all CAPS in the beginning of the document.

[0080] DocType/SubType for Deal Bank Documents, Titles are Extracted Primarily Through Comparison of Known Titles to a Doctype/Subtype Matrix.

[0081] This makes use of DocTypeRules.txt rules file. The format of the rules file is as follows:

[0082] TITLE_RULE<TAB>TEXT_RULE<TAB>CHAR_COUNT<TAB>DOC_TYPE<T AB>DOC_SUBTYPE

[0083] TITLE_RULE will be empty if there is no title rule.

[0084] Approach

FOR EACH ENTRY IN THE DOCTYPERULES FILE REPEAT THE FOLLOWING STEPS.
FIRST SEE IF TITLE RULE IS AVAILABLE, IF SO APPLY THE RULE ON THE TITLE EXTRACTED.
IF SUCCEEDED GET THE CORROSPONDING DT/ST.
IF THE DT/ST ARE ALREADY IN THE LIST SKIP IT ELSE SAVE THE DT/ST IN THE LIST.
IF FAILED TO EXTRACT FROM THE TITLE RULE OR NO TITLE RULE WAS AVAILABLE
APPLY TEXT RULE ON FIRST N CHARS OF THE DOCUMENT.
IF SUCCEEDED SAVE CORRO. DT/ST IF NOT ALREADY IN THE LIST.

[0085] Parties

[0086] Parties information can be found in the beginning of the document, in the signature block or/and in the title of the document itself. Each of these may use a different set of rules.

[0087] Approach:

EXTRACT FIRST FEW LINES IN THE DOCUMENT.
REMOVE ANY BLANK LINES.
FOR EACH RULE IN THE PARTYRULE FILE REPEAT THE FOLLOWING STEPS.
IF A MATCH, EXTRACT THE MATCHED STRING
IF THE EXTRACTED STRING IS SAME AS TITLE IGNORE THE STRING.
IF THE MATCHED STRING HAS ANY NOISE WORDS SKIP IT.
ELSE STORE THE PARTY IN THE LIST.
REPEAT THIS RULE ON THE REST OF THE BUFFER FOR MORE PARTIES UNTIL THE END OF
THE BUFFER.
IF NO PARTIES EXTRACTED:
FROM THE TITLE STRING OF THE DOCUMENT EXTRACT EACH LINE
AND CHECK FOR INC., CORPORATION, INCORPORATED, CORP, AND COMPANY.
IF FOUND, THAT LINE OF TEXT WILL BE TREATED AS THE PARTY.
IF NO PARTIES EXTRACTED IN ABOVE 2 STEPS
SEARCH FOR STRING IN WITNESS WHEREOF IN THE DOCUMENT
IF MATCH FOUND REPEAT THE FOLLOWING STEPS UNTIL ALL THE PARTIES HAVE
BEEN EXTRACTED OR END OF FILE HAS BEEN REACHED:
LOOK FOR BY OR BY_OR BY:
EXTRACT ALL THE LINES OF TEXT PRECEDING BY OR BY_OR BY:
LOOK FOR A LINE, IN ALL CAPS, THAT IS CLOSEST TO BY_OR BY: OR BY WHICH
WILL BE TREATED AS ONE OF THE PARTIES AND ADDED TO THE PARTY LIST.

[0088] Governing Law.

[0089] For extraction of Governing Law, StateRules.txt is used, which includes rules related to Governing Law. Another file called StateList.txt is used for looking up all the State /Province Information.

FOR EACH RULE IN THE RULES FILE REPEAT THE FOLLOWING STEPS:
RUN THE RULE ON THE DOCUMENT TEXT.
IF THE RULE MATCHED, EXTRACT THE STATE, IF ANY, FOLLOWING THE RULE
MATCH. TAKE FOR INSTANCE IN ACCORDANCE WITH THE LAWS OF THE STATE OF DELAWARE. IN
THIS CASE THE RULE WOULD MATCH THE PHRASE IN ACCORDANCE WITH THE LAWS OF THE STATE
OF. SO WE'LL LOOK FOR THE STATE TO FOLLOW THIS.
IF STATE IS FOUND BREAK OUT OF THE LOOP.

[0090] As noted above, other rules, having analogous parameters, may be used.

[0091] Many of the rules given above were for Deal documents. Litigation documents may also have abstract fields. Due to the presence of a substantially consistent caption on the first page of litigation documents, different techniques may be used to capture the data.

[0092] Some DocTypes are dependent on other Doc Types. For example

[0093] eg: see document 0080002.01

[0094] NOTICE OF HEARING ON DEMURRERS AND DEMURRERS OF DEFENDANTS KAUFMAN AND BROAD HOME CORPORATION, KAUFMAN AND BROAD OF SOUTHERN CALIFORNIA, INC., AND KAUFMAN AND BROAD HOME SALES, INC. TO THE ALLEGED THIRD, SIXTH AND SEVENTH CAUSES OF ACTION OF THE COMPLAINT

[0095] (Memorandum of Points and Authorities In Support Thereof Attached Hereto; Motion To Strike Portions Of Complaint Filed Concurrently Herewith)

[0096] There are 4 matches here:

[0097] Notice

[0098] Demurrers

[0099] Memorandum of Points and Authorities

[0100] Motion To Strike

[0101] The Abstract Creation Engine uses rules to make subjective conclusions about document types. For example, if the rules uncovered terms Answer and Complaint, the rules can determine that the Document Type is an Answer only. This is achieved by the rules which consider the relationships between document types and pre-set desired outcomes for all conditions.

[0102] Demurrers and Notice are related/dependent.

[0103] Notice dominates Demurrers and its located before Demurrers

[0104] Also the presence of to next to Notice helps.

[0105] Back tracking (AI technique)

[0106] General:

[0107] Given a document, first look for Abstract already in the database.

[0108] Certain fields like Jurisdiction, Judge Name, Firm name will repeat.

[0109] Assumption:

[0110] One document will not have more than one Judge Name, or Case number.

[0111] There are instances of finding more then one Court names in one document. In those cases, hierarchy rules are applied.

[0112] As the table in the database fills, a continuously improving strike rate is obtained. However, at all times the search can be limited to the first page.

[0113] Case Number:

[0114] Case number is generally found next to Case No: Docket No etc. If a case number is easily found, then a lookup can be done in Existing published and queued documents to get known Abstract fields associated with that case, including:

[0115] Abstract field

[0116] DocType And Doc Title

[0117] DocType And Doc Title:

[0118] The Abstract Creation Engine uses the rules to make subjective conclusions about document types. For example, if the rules uncovered terms Answer and Complaint, then the rules determine that the Document Type is an Answer only. This is achieved by a list of relationships between document types and pre-set desired outcomes for all conditions.

[0119] Approach:

OPEN A DOCUMENT
LIMIT SEARCH TO FIRST OR SECOND PAGE (E.G., 52-60 LINES)
TRAVERSE THROUGH EACH POSSIBLE DOCTYPE LIST
FIND THE DOCTYPE KEWORD/PHRASE IN THE FIRST PAGE
IF FOUND
GET THE SENTENCE IN WHICH THIS WORD OCCURS.
THIS BECOMES THE DOCUMENT TITLE.
IF THIS DOCTYPE IS DEPENDENT ON ANOTHER DOC TYPE
GET THE ORDERING TO DETERMINE DOMINANT DOCTYPE
VERIFY USING TRAITS (FOLLOWING WORD) TO GET
DOCTYPE

[0120] Firm/Counsel Name

[0121] Firm name is generally found at start of the document.

[0122] Firm name can be found followed by LLP or LLC. It can be found in Above or Below line of Lawyer Name. Lawyer Name may be followed by Bar . . . No.

[0123] Judge Name/Dept

[0124] Judge name may be found next to Judge Name, Magistrate, Dept:, Dept No:. It is generally found near to document Title.

[0125] State/Jurisdiction

[0126] Jurisdiction Processing Logic is done as a Four Step Process. Take an Jurisdiction Title as example.

[0127] In The District Court Of

[0128] Harris County, Texas

[0129] 281st Judicial District

[0130] The Jurisdiction Header can be extracted first. This should contain enough information to allow obtaining State Name, Court Type and Court Name. In the above example, this allows extracting The District Court Of Harris County, Texas. This is done by the Stepped Jurisdiction Rules.

[0131] Each line in this Rules list corresponds to a Rule. Each Rule contains up to three Sub Rules separated by a tab. To extract the above string, one of the rules as IN THE (DISTRICTJUSTICE) COURT

[0132] ({circumflex over ()}w*s*) 0,1w*sCOUNTY,s*TEXAS d*w*sJUDICIALs*DISTRICT is found.

[0133] Incidentally, this Rule extracts all three lines of the above Jurisdiction Title, even though two lines would have been sufficient. The Sub Rule IN THE (DISTRICTJUSTICE) COURT extracts In The District Court, while the Sub Rule ({circumflex over ()}w*s*) 0,1w*sCOUNTY,s*TEXAS will extract Harris County, Texas and the Non-Mandatory Sub Rule d*w*sJUDICIALs*DISTRICT will extract 281st Judicial District.

[0134] Subsequent to the extraction, the above strings are concatenated and the Jurisdiction Header is thus constructed. This Header is then used for the further three steps.

[0135] Next, extract the Court Type from the Jurisdiction Header obtained above. This is done using the litCourtList Rules. The Court Type extracted in the above example is DISTRICT.

[0136] Third Step: All the Court Types are mapped to a default Court Type Mapping based on the California system. If the Court Type of any State differs from that of the default, then it is mapped to the default in the litCourtNameAlias Rules. In the above case, the District court in Texas is mapped to Superior court in California. One of the rules in this list is TEXAS DISTRICT SUPERIOR (JUDICIALCOUNTY). Herein there are four Sub Rules separated by a tab. The first Sub Rule identifies the State (Texas in this case), the second Sub Rule identifies the Name (District), the third gives the mapped Court Type (Superior herein), while the fourth Non-Mandatory Sub Rule provides the supporting string which helps in Positive identification. If there is either JUDICIAL or COUNTY in the Jurisdiction Header, that when this Court Type gets mapped to Superior Court, otherwise it will be a District Court of Texas (for ex, take another Jurisdiction Title IN THE UNITED STATES DISTRICT COURT FOR THE WESTERN DISTRICT OF TEXAS EL PASO DIVISIONThis is a TexasW.D. Court). Thus, the Court Type is mapped to SUPERIOR in the present case.

[0137] Finally, the mapped Court Name is obtained from litCourtNames Rules list. Herein, the Court Name strings likely to be encountered form the basis for creating the respective Rule. Each Rule is composed of three Sub Rules like TEXAS (COUNTYs*OFs*HARRIS)(HARRISs*COUNTY) Harris , each separated by a tab. The first Sub Rule is the State Name (TEXAS in this case ), the second is the Name-Expression ((COUNTYs*OFs*HARRIS)(HARRISs*COUNTY) herein) to map the name in the Jurisdiction Header, while the third Sub Rule is the actual Court Name (Harris to name here) in the DB. Accordingly, Harris gets extracted here.

[0138] With the State, Court Type and Court Name, the Business Layer checks with the database values and if a match is made, then the CourtID is extracted which is what is stored in the abstract for this document. Anytime, a request/Search is made for this document, the CourtID is used to get the STATE and COURTNAME for display.

[0139] The above represents the rules for extracting State based Courts. Before this process is done, the extraction of Jurisdiction Header is done using the litJurisdictionList. This extraction has Rules to extract Federal and ADR Agencies Courts. If one of these Rules match, then the stepped Jurisdiction Rules parsing is not done and hence no State gets extracted. If no State is extracted, then Parse for the Federal Courts using the litFedCourtNames Rules. If this fails, then push these through litTribunalInfo to get Tribunal Information.

[0140] An application provides full text search support on Litigation and Deal documents, SmartRules and Clause Heading of Deal documents. Clause Headings will be stored as VARCHAR in a column and the documents will be stored on the FileServer.

[0141] The Indexing service provides:

[0142] 1. Property search. This search is more of statistical information and more of metadata like Author, Subject type, Word count, Last written etc.

[0143] 2. Full text search.

[0144] Proximity search (proximity term: near)

[0145] Inflectional (generation term)

[0146] Weighted search (weighted term: queries that match a list of words and phrases, each optionally given its own weighting)

[0147] Free text

[0148] Simple terms: Single word or phrase

[0149] Prefix terms: They are extension of simple terms where they can have the form of wildcards like agree*.

[0150] Contains search conditions: AND, AND NOT, OR

[0151] The same feature set extends at the TSQL table level as well (i.e these predicates are available in a little different syntax if the query is performed against a database table/column instead of external files).

[0152] Every defined category may have a _Primary.txt file (e.g., Copyright_Rules_Primary.txt). Each_Primary.txt file includes at least one (or more) primary rule(s). The primary rules are expressed in the following format:

	Proximity	Min	Primary	DistaHemang	Secondary	Rule	Substantive	Subject	SM	SM
Weight	Weight	Occurs	Term	Sanghavince	Term2	Display	Area	Matter	Weight	Threshold

[0153] Each primary rule identifies a Primary Term (a word or phrase) that may appear in a given category within a set of documents. For example, the word easement may appear in certain document that should be deemed to fit in the substantive legal area of property documents.

[0154] Additionally, the engine can identify more complex concepts by locating two or three words/phrases near each other. In this case, the engine will find Primary Terms within a certain defined Distance (number of words) from SecondaryTerm1 (a word or phrase) and/or (the and/or is user defined and called the Operator) a Secondary Term2 (a word or phrase). For example, to identify the concept of breach in a contract document, a rule might identify the word breach (Primary Term) within 10 (Distance) words of the words contract (Secondary Term1) or (Operator) agreement (Secondary Term2).

[0155] Each primary rule is assigned a Weight value based on its distinctiveness (the more distinctive or rare, the higher the weight).

[0156] Each primary rule is assigned a MinOccurs (minimum occurrences) value based on the relative frequency of its appearance in a given document set (the more common, the higher the MinOccurs).

[0157] Each primary rule may be assigned a Rule Display, which is the exact text that will be displayed to the end-user when a given rule has been identified and the document has been categorized as falling into that substantive area. For example, to identify the concept of breach in a contract document, a rule might identify the word breach (Primary Term) within 10 (Distance) words of the words contract (Secondary Term1) or (Operator) agreement (Secondary Term2). Rather than display the complex primary rule, the text displayed to the end-user could be Breach of contract. However, a primary rule need not have a Rule Display name. For example, one might look for the word tax to identify documents belonging to the category of Tax Law, but showing the end-user a Rule Display of Tax adds little to their analysis of the document's contents.

[0158] C. Wild Cards:

[0159] In both sets of rules, the Keywords, Primary Terms, and Secondary Terms, can be include wild cards. Wild cards deepen the rule base by defining a Keyword, Primary Term or Secondary Term as a group of words that capture various similar expressions. A rule identifying the concept of capacity to contract could look for the word capacity within 5 words of the word contract. This rule would correctly identify occurrences of capacity to contract, but would not identify the phrase contractual capacity. One could create a new rule to capture every variation of the word contract; however, the SA engine allows a user to define a Keyword, Primary Term or Secondary Term as a group of words to allow one rule to identify multiple variations of the target concept. For example, a user could modify the above rule to look for the word capacity within 5 words of the wild card contract. Placing an exclamation point at the end of a Keyword, Primary Term or Secondary Term tells the engine to lookup the wild card in the WildCards.txt file and substitute all defined terms in place of the wild card to essentially extend the rule in to X number rules (X being the number of words associated with the wild card). In the example above the wild card contract might be defined as: contract, contracting, contracts, contracted, and contractual. Using this expression, the rule would correctly identify occurrences of capacity to contract and contractual capacity.

[0160] Full text searching of a conventional type may be carried out. The full text search uses an application Microsoft Technologies and supports open standards including XML, SOAP. The web server uses IIS 5.0 hosting ASP pages. The middle tier is formed of components running in the COM environment. The data tier uses ADO. The database server is SQL 2000 and search technologies include Indexing Service (comes as a Windows 2000 base service), Full Text Search support provided by SQL 2000.

[0161] SQL Server 2000 uses the same search engine technology used by SharePoint portal Server, benefits from same advanced ranking algorithm and uses a subset of the full-text extensions to SQL used by SharePoint Portal Server.

[0162] Full-text search SQL extension are integrated into the T-SQL language. Users can specify SQL queries that can span structured data from SQL tables, unstructured data from SQL columns, from documents embedded in the columns, and from the file system.

[0163] Other embodiments are intended to be included. For example, while the above has described software modules, it should be understood that the functions described herein could be alternatively implemented in hardware, e.g., using FPGAs or the like.

[0164] All such modifications are intended to be encompassed within the following claims.