System and method for discovering schematic structure in hypertext documents

[0002] 1. Field of Invention

[0003] The present invention generally relates to document processing. The method and apparatus of the present invention have particular application to extracting schematic information from a set of documents.

[0004] 2. Discussion of Prior Art

[0005] The world wide web (throughout this specification,web, www, and world wide web are used interchangeably) is presently growing at an average of 1 million pages per day and is an amazing source of information. All of this information is buried in HTML documents authored by a wide variety of people with differing skills, culture, and purpose and using a wide variety of tools to author these pages. HTML does give structure to the authored documents, but mainly for viewing purposes. HTML has a fixed set of tags which are mostly used to enhance the visual appeal of the documents. Thus, it often happens that HTML pages for the same purpose have a different set of tags. For instance, people often mark-up their resumes in HTML on their home pages. Depending on the styles, these resume documents look significantly different from each other. This difference is acceptable for viewing purposes, but presents great difficulties to automatic programs which try to extract pertinent information from them.

[0006] The web is not just used for browsing anymore; automatic programs, like web crawlers, visit web sites and extract information to serve search engines, or push engines. Comparison shopping engines visit web sites describing similar information, such as prices, and extract semantic information from these sites. Given the format variances possible between topically related web pages, retrieved data is often unhelpful, unrelated or difficult to extract.

[0007] However, the present invention addresses this need to build search engines that allow users to formulate structural queries like “find a student with a Master's degree and a GPA of 3.5 or more and skills in Java.” The present invention allows the extraction of structural information buried in HTML pages which cater to the same topic but are authored with significantly different styles.

[0008] Some specific prior art related to the present invention is discussed below. These references describe methods of investigating the structure of documents and retrieving documents from large databases in response to user queries.

[0009] Two articles which describe attempts to discover structure from semi-structured data are “Identifying Aggregates in Hypertext Structures”, Proceedings of ACM Hypertext '91, pp. 63-74, and “Structural Properties of Hypertext”, Proceedings of the Ninth ACM Conference on Hypertext, pp. 180-187, 1998. These attempts, however, focus on the organization of a set of hypertext documents by following their links rather than considering the schematic nature of the individual documents.

[0010] Three other articles describing related investigations are: “Inferring Structure in Semistructured Data”, Workshop on Management of Semistructured Data, 1997; “Extracting Schema from Semistructured Data”, SIGMOD98, pp. 295-306; and “Discovering Association of Structure from Semistructured Objects”, IEEE Trans. on Knowledge and Data Engineering, 1999. However, these articles do not consider the schematic structure of individual documents or documents which have different schemas.

[0011] The patent to Driscoll (U.S. Pat. No. 5,694,592) teaches a method of querying and retrieving documents from a database using semantic knowledge about the query string to determine document relevancy.

[0012] The patent to Ishikawa (U.S. Pat. No. 5,848,407) describes a method of presenting potentially related hypertext document summaries to a user who is using a search engine that indexes a plurality of hypertext documents.

[0013] Whatever the precise merits and features of the prior art in this field, the earlier art does not achieve or fulfill the purposes of the present invention. The prior art does not provide for automatically identifying schematic structural and tag information from HTML documents and then converting these documents according to the extracted information.

[0014] The present invention describes a system and method that extracts keywords and structural information from hypertext or mark-up language documents (e.g. HTML) and then reformulates them as documents with a common structure and common set of tags. One underlying goal is to convert a collection of HTML documents, written in different styles, into XML documents following a common schema. XML, eXtended Markup Language, defines a web standard for describing schemas for different domains. For instance, one domain might be resumes, and a schema can be defined for describing all resumes. Thus all resume documents are written using the structure and tags described by this schema. Thereafter, keyword based search engines will be able to support queries and retrieve documents that are schematically and semantically closer to the information users are looking for. Using a five-stage process, the common schematic structures are discovered for the set of HTML documents authored in various styles. Prior domain knowledge regarding punctuation, keywords, synonyms and HTML tags is used to 1) break a document up into separate objects, 2) identify the objects corresponding to keywords, 3) regroup objects into hierarchical layers of abstraction, 4) logically order objects at the same level of abstraction, and finally 5) remove any non-keyword related information from the document's discovered schematic structure. The discovered schema supports structural queries from search engines that locate data that are more semantically related to the requested information than data located by simple keyword searching.

[0015] FIG. 1 illustrates an overview of an example web-based environment for the present invention.

[0016] FIG. 2 illustrates an overview of a five step schema extraction process according to the present invention.

[0017] FIG. 3 illustrates an example HTML formatted resume used by the present invention.

[0018] FIG. 4aillustrates the structural effect of the tokenization function of the present invention on an example XML document.

[0019] FIG. 4billustrates the effect of the tokenization function of the present invention on an example XML document's coding.

[0020] FIG. 5aillustrates the structural effect of the keywording function of the present invention on an example XML tree.

[0021] FIG. 5billustrates the effect of the keywording function of the present invention on an example XML document's coding.

[0022] FIG. 6aillustrates the structural effect of the phrasing function of the present invention on an example XML tree.

[0023] FIG. 6billustrates the effect of the phrasing function of the present invention on an example XML document's coding.

[0024] FIG. 7aillustrates the structural effect of the heading function of the present invention on an example XML tree.

[0025] FIG. 7billustrates the effect of the heading function of the present invention on an example XML document's coding.

[0026] FIG. 8aillustrates an extracted schema from an example XML tree resulting from applying the consolidation function of the present invention.

[0027] FIG. 8billustrates the coding of an extracted schema from an example XML tree resulting from applying the consolidation function of the present invention.

[0028] While this invention is illustrated and described in a preferred embodiment, the invention may be produced in many different configurations, forms and materials. There is depicted in the drawings, and will herein be described in detail, a preferred embodiment of the invention, with the understanding that the present disclosure is to be considered as an exemplification of the principles of the invention and the associated functional specifications of the materials for its construction and is not intended to limit the invention to the embodiment illustrated. Those skilled in the art will envision many other possible variations within the scope of the present invention.

[0029] FIG. 1 illustrates an overview of the flow of the present invention, in an example environment of a web-based search engine. This example of the present invention works in the following manner.

[0030] A web crawler 104 traverses the web 102 and retrieves a number of HTML documents 106. These HTML documents 108 are converted, using known methods, to XML documents 110 by HTML-to-XML filter 108. This filter converts the HTML document to “well-formed” XML documents, as defined by present and future versions of the XML specification. Also, representations of domain knowledge such as keywords and matching information 112 as well as HTML tag knowledge 114 are constructed by the users of the present system.

[0031] HTML tag annotations 114, keyword knowledge 112, and XML documents 110 are provided to extractor 116 which extracts schematic information 118 from XML documents 110. The schematic information extractor 116 uses a five step process to extract schematic information from a set of input documents. FIG. 2 illustrates an overview of schematic information extractor 116 operating on a single XML document.

[0032] Knowledge regarding token separators 202, keyword matching information 204, HTML phrase tags 206, and HTML header tags 208 are used appropriately during the five steps of the extraction process: tokenizing 210, keywording 212, phrasing 214, heading 216 and consolidating 218.

[0033] The input to and the output from these five steps is depicted on the left hand side of FIG. 2 and include either an XML document 220 or an XML tree 222.

[0034] Each of the five steps of the schematic extraction process are discussed below in detail. Also, FIG. 3, and FIGS. 4athrough 8adepict an example document and how its structure is transformed by each step of the process. FIGS. 4bthrough 8bdepict the HTML and XML code of an example document as it is transformed by each step of the process. The example document, illustrated in FIG. 3, is as follows. Note that the node h2 is incorrectly marked up to denote the incorrect tag usage found in many web documents.

[0035] <html><body>

[0036] <h1>Resume of John Dole</h1>

[0037] <h2>Education

[0038] <table><tr><td>

[0039] Ph.D. in Comp. Sci., University of California,

[0040] Sept 1996-Present

[0041] </td></tr>

[0042] <tr><td>

[0043] B.S. (Computer Science), Stanford University,

[0044] Sept 1993-Jun 1996, GPA 3.9/4.0

[0045] </td></tr></table>

[0046] <h2>Experience</h2>

[0047] <ul><li>

[0048] Research Assistant, University of California,

[0049] Sept 1997-Present </li>

[0050] <li>

[0051] Teaching Assistant, University of California,

[0052] Sept 1996-Jun 1997

[0053] </li></ul></h2></body></html>

[0054] Hypertext documents, as exemplified above, generally follow an ordered tree model. In formal notation, an ordered tree with root root and immediate children c₁, . . . , c_n, is represented as

[0055] T(root)≡<root, <ST(c₁), . . . , ST(c_n)>>

[0056] where $\mathrm{ST}\left(c_i\right)\equiv \{\begin{array}{cc}T\left(c_i\right)& \mathrm{if}{c}_i\mathrm{is}\mathrm{an}\mathrm{internal}\mathrm{node}\\ c_i& \mathrm{if}{c}_i\mathrm{is}a\mathrm{leaf}\mathrm{node}\end{array}$

[0057] and an empty tree or node is denoted by ⊥.

[0058] This formal notation will be relied upon in describing some of the activities of each of the five steps of the schematic information extraction process. In addition to this formal definition, there are also tree functions which represent manipulations that are routinely performed on tree structures. Some of the ones that are used by the present invention include:

[0059] Function Label: transforms a node into a string by computing the label of the markup tag of a node.

[0060] Function Value: transforms a node into a string by computing the string value of a node if it is a text node. Otherwise it is ⊥.

[0061] Function Root: computes the root of the tree to which a node belongs.

[0062] Function Parent: computes the parent of a node.

[0063] Function PrevSibling: computes the previous sibling of a node, or ⊥ if the node is the first child of its parent.

[0064] Function Children: computes the sequence of immediate child nodes of a node.

[0065] Function Descendants: computes the set of descendant nodes of a node at all levels.

[0066] Function Replace: replaces a subtree of a tree by another, Specifically, Replace(T(node), c, c′) returns the tree the same as T(node) but with the subtree rooted at c, T(c) replaced by the subtree rooted at c′, T(c′). If T(c′)=⊥, subtree T(c) is deleted. Typically, the shorthand Replace(T(node), <c₁, . . . , c_n>, <c′₁, . . . , c′_n>) is used to denote that Replace function is applied consecutively to the tree to replace nodes c_iwith c′_iin the order specified in the sequence.

[0067] Function Delete: deletes a subtree from a tree. Specifically, Delete(T(node), c) deletes the subtree rooted at c from tree T(node). The shorthand Delete(T(node), {c_i}) is used to denote that the Delete function is applied consecutively, in any order, to the tree to remove the subtrees specified in {c_i}.

[0068] Function Insert: inserts a subtree into a tree. Specifically, Insert(T(node), c, c′) inserts the subtree (T(c′) into T(node) as the previous sibling of c. If c=⊥, c′ is inserted as the first child of node. If c′ ∉ Descendants(Root(node)), Delete(T(Parent(c′)), c′) is applied before Insert(T(node), c, c′).

[0069] Function Parent: computes the parent of a node.

[0070] Function PrevParent: computes the previous parent for a node. PrevParent(node) is Parent(node) if no tree reordering function has been applied to T(node). If Replace is applied, PrevParent(node) is the replaced node; in the case of Insert, it is ⊥.

[0071] In the step of tokenization, a set of punctuation is used to extract structural information from each block of text in the XML document 220. The tokenize function gets a set of punctuation marks (e.g. {“;”,“,”, “:”}) and uses them to separate out each block of text into smaller blocks of text. This separating results in unrelated phrases being placed in separate subblocks of text.

[0072] The above list of punctuation marks is one group of marks that handles most general HTML documents expected to be encountered by an information gatherer for a search engine. If a particular domain of documents has particular punctuation marks that are specific to that domain, then the set of applicable marks is modified to include these specific marks.

[0073] A high level object or concept is often described by several objects which are separated by punctuation. For example, in a resume document, if a topic sentence about the educational background is “University of California at Davis, B.Sci. (Computer Science), Awarded June 1996, GPA 3.8/4.0.” The four pieces of information are:

[0074] Organization: University of California at Davis

[0075] Degree: B.Sci. (Computer Science)

[0076] Date: Awarded June 1996

[0077] GPA: GPA 3.8/4.0

[0078] and all are separated by the punctuation “,”. Hence, this heuristic states that a sentence be divided into tokens according to the punctuation. These tokens are then grouped together under a new node labeled item to preserve the fact that the corresponding objects describe the same concept or object of a higher level of abstraction.

[0079] A formal definition of this tokenizing function, Θ(T(node)), is:

[0080] Let T(node)≡<node, <T(t₁), . . . , T(t_n)>>, let TS be a set of punctuation, and let node be tokenized into tokens x={x₁, . . . , x_j}. $\Theta \left(T\left(\mathrm{node}\right)\right)\equiv \{\begin{array}{c}<\mathrm{item},<x_1,\dots ,x_j>>\\ \mathrm{If}\mathrm{node}\mathrm{is}a\mathrm{text}\mathrm{node}\mathrm{and}x\ne 0\\ <\mathrm{node}>\\ \mathrm{If}\mathrm{node}\mathrm{is}a\mathrm{text}\mathrm{node}\mathrm{and}x=0\\ <\mathrm{node},<\Theta (T\left(t_1\right),\dots ,\Theta \left(T\left(t_n\right)\right)>>\\ \mathrm{Otherwise}\end{array}$

[0081] Function Θ states that if a node is a text node, it is tokenized into text nodes x₁, . . . , x_jaccording to TS. The text nodes are grouped under a new node labeled item. If the node is not a text node, function Θ is recursively applied to its child nodes.

[0082] FIGS. 4aand 4billustrate how this function transforms the example document from FIG. 3.

[0083] This function starts with a set of keywords, their synonyms and certain rules for identifying these keywords from blocks of HTML text and markup. The end result is that these keywords are enhanced to create XML elements with these keywords as element names.

[0084] Keywords, as used in the present invention, describe important concepts. Formally, let KI={<k_i, {s_i,j}, whole_i, case_i>} be the domain knowledge about keywords. It consists of a set of keywords K={k_i} and associated with keyword k_iis a set of synonyms {s_i,j}. Keyword k_iis identified from a string by matching its synonyms against words in a string; and words are sequences of characters separated by the space character. If case_iis true, pattern matching is case sensitive; otherwise, it is case insensitive. If whole_iis true, then a string must match exactly in order to identify k_i; otherwise only a portion of the string has to match in order to identify k_i.

[0085] Since tokenization partially identifies the component objects describing a higher level object, applying keyword identification after tokenization typically improves the accuracy of identifying the information the keywords describe. For example, the text “University of California, State University of Fresno” is described by two objects—the names of two universities. Assuming there is a keyword Organization with associated synonym “university”, application of keywording before tokenization results in two Organization keywords being identified. However, which Organization the keyword “State” belongs to is ambiguous without using sophisticated parsing mechanisms. No such complex and sophisticated parsing is needed when keyword identification is applied after tokenization.

[0086] A formal definition of this keywording function Γ(T(node), is:

[0087] Let KI={<k_i, {s_i,j}, whole_i, case_i>} define a set of keywords K={k_i} and their information. Keyword function, Γ(T(node), creates keyword nodes for any keywords identified. A node node is termed as a keyword node if Label(node) ∈ K. It is a non-keyword node otherwise. The switches whole_iand case_iare boolean operators which control how a keyword (and its synonyms) are matched as described above.

[0088] Using a function IK(s), which identifies the set of keywords in a string s; let IK(Value(t_i))={k_i,1, . . . , k_i,n}.

[0089] Γ(T(node)≡<node, <T′(t₁), . . . , T′(t_n)>>where $T^{\prime }\left(t_i\right)\equiv \{\begin{array}{c}<\mathrm{item},<k_{i,1},\dots ,k_{i,n}>>\\ \mathrm{if}{t}_i\mathrm{is}a\mathrm{text}\mathrm{node}\mathrm{and}\mathrm{Label}\left(\mathrm{node}\right)\ne \mathrm{item}\\ <k_{i,1},\dots ,k_{i,n,i}>>\\ \mathrm{if}{t}_i\mathrm{is}a\mathrm{text}\mathrm{node}\mathrm{and}\mathrm{Label}\left(\mathrm{node}\right)=\mathrm{item}\\ \Gamma (T\left(t_i\right)\\ \mathrm{Otherwise}\end{array}$

[0090] Function Γ states that if the child node t_iof node is a text node with keywords k_i,1, . . . , k_i,n,iidentified, a node labeled with the same keyword is created. These keyword nodes are grouped under a node labeled item. Node item is created if it does not exist. If node item exists (because of tokenization), keyword nodes are grouped under the same item node since all of them descend from the same node. If t_iis not a text node, function Γ is recursively applied to its subtree.

[0091] FIGS. 5aand 5bshow the effect of the keywording function on the tokenized tree from FIGS. 4aand 4b. In this example, K={Education, Experience, Degree, GPA, Organization, Date, Title}.

[0092] In this step, certain HTML tags, which are often used to break a document into sections, are identified and used to restructure the document.

[0093] The system identifies tags such as p, hr, h1, h2, etc. as phrase tags. This list of tags is just an example list and various additions and deletions of other HTML tags are also contemplated within the scope of the present invention.

[0094] The set of mark-up tags M, termed “phrase tags”, give hints to the grouping of semantically related nodes in a hypertext document. Block elements (e.g. p, hr), list elements (e.g. dl, ul), heading elements (e.g. h1, h4), and table elements (e.g. tr, td) are often used to divide data into paragraphs or sections. Phrase tags are used to aggregate semantically related nodes by grouping everything between two consecutive nodes, labeled with the same phrase tag, into one new node labeled group.

[0095] A formal definition of the phrasing function, φ(T(node)), is:

[0096] Let P ⊂ M be a set of phrase tags.

[0097] Define $p_1\equiv \{\begin{array}{cc}{t}_k& \mathrm{if}\exists t_k\mathrm{such}\mathrm{that}\mathrm{Label}\left(t_k\right)\in P\mathrm{and}\\ & \left(\nexists t_j\mathrm{such}\mathrm{that}{t}_j\in P\mathrm{and}j<k\right)\\ \perp & \mathrm{otherwise}\end{array}$

[0098] p≡{p₁, . . . , p_m|Label(p_k)=Label(p₁), p_k∈Children(node)} where p₁is the first child node of node labeled with a phrase tag and p is the set of child nodes of node with the same label as p₁.

[0099] Define function MakeChild: $\mathrm{MakeChild}\left(t_i,t_j\right)\equiv \{\begin{array}{cc}<t_i,& <\mathrm{group},<T\left(c_1\right),\dots ,T\left(c_p\right)>>,\\ & T\left(t_{i+1}\right),\dots ,T\left(t_{j-1}\right)>\mathrm{if}i<j\\ T\left(t_i\right)& \mathrm{if}i=j\\ \perp & \mathrm{otherwise}\end{array}$

[0100] Function MakeChild makes the subtrees rooted from t_i+1, . . . , t_j−1child nodes of t_i. The original children nodes of t_i, c₁, . . . , c_p, are grouped under a new node with the tag group.

[0101] Define phrasing function, φ: $\varphi \left(T\left(\mathrm{node}\right)\right)\equiv \{\begin{array}{cc}<\mathrm{node},& <t_1,\dots ,\mathrm{PrevSibling}\left(p_1\right),\\ & \varphi \left(T^{\prime }\left(p_1\right)\right),\dots ,\varphi \left(T^{\prime }\left(p_m\right)\right)>>\\ \mathrm{if}& p\ne 0\\ T\left(\mathrm{node}\right)& \mathrm{otherwise}\end{array}$

[0102] where $T^{\prime }\left(p_i\right)\equiv \{\begin{array}{cc}\mathrm{MakeChild}\left(p_i,p_{i+1}\right)& \mathrm{if}1\le i\le m-1\\ \mathrm{MakeChild}\left(p_i,t_n\right)& \mathrm{if}i=m\end{array}$

[0103] Function φ states that if node has child nodes labeled with a phrase tag, for such a node p_i, nodes from its next sibling until the next node with the same label (or the last sibling if there is none) become its child nodes. Function φ is recursively applied to the subtree rooted at p_i, and nodes before p₁remain the same.

[0104] FIGS. 6aand 6billustrate the operation of function φ on the example document with P={h1, h2}. The result, shown in FIG. 6a, is that h2606 and all its children nodes descend from newly created group node 604 which, itself is a child node of h1602. Also, group node 610 is created under h2608 and nodes 612 and 614 and their children are moved under group node 610.

[0105] This function reformats any header tags that are nested under identical header tags. Header tags are typically used to structurally divide HTML documents into separate sections. By requiring that no two-header tags of the same label can be on the same path from the root of the document, this heuristic modifies every such occurrence such that the two occurrences become siblings of each other.

[0106] These header tags divide data of a document into sections of the same level of abstraction regardless of their markup style. Hence, this heuristic states that objects under nested header tags of the same label are re-ordered to be siblings of each other.

[0107] A formal definition of this heading function, Ψ(T(node)), is:

[0108] Let H ⊂ M be a set of header tags. Heading function Ψ reorders the tree so that there are no two nodes with the same header tags along the same path from the root.

[0109] Given a node t_iwith Label(t_i) ∈ H, define

[0110] where {t_i,1, . . . , t_i,h,i} is the set of child nodes of t_i, labeled with the same header tag which do not have an ancestor labeled with the same header tag except t_iitself, in the subtree T(t_i).

[0111] Ψ(T(node))≡<node, <T′(t₁), Ψ(T(t_1,1)), . . . , Ψ(T(t_1,h,1)), . . . , T′(t_n), Ψ(T(t_n,1)), . . . , Ψ(T(t_n,h,n)>>

[0112] where $T^{\prime }\left(t_i\right)\equiv \{\begin{array}{c}\Psi \left(T\left(t_i\right)\right)\\ \mathrm{If}{t}_i\notin H\mathrm{and}{t}_i\mathrm{is}\mathrm{an}\mathrm{internal}\mathrm{node}\\ T\left(t_i\right)\\ \mathrm{If}{t}_i\notin H\mathrm{and}{t}_i\mathrm{is}a\mathrm{leaf}\mathrm{node}\\ \Psi \left(\mathrm{Delete}\left(T\left(t_i\right),\left\{t_{t,1},\dots ,t_{i,h,i}\right\}\right)\right)\\ \mathrm{If}{t}_i\in H\end{array}$

[0113] Function Ψ states that if a child node t_iis not labeled with a header tag, function Ψ is recursively applied to its subtree. Otherwise, all subtrees under root t_irooted with the same header tag T(t_i,1), . . . , T(t_i,h,i) become siblings of t_i. Function Ψ is applied recursively to all these subtrees.

[0114] The result of applying the heading function to the example document is illustrated in FIGS. 7aand 7b. As an example, H={h2}, and as a result, in FIG. 7a, the h2 nodes 702 and 704 become siblings.

[0115] For a document being analyzed by the present invention, the non-keyword nodes do not necessarily provide critical schematical information regarding the document; however, the structure of the keyword nodes are of prime importance. The consolidation function replaces non-keyword nodes with keyword nodes and this heuristic is driven by the following observations.

[0116] First, very often, the first object of a group of semantically related objects describes the topic of this group of objects. This is analogous to the topic sentence of a paragraph. The heuristic, therefore, replaces a non-keyword node by its first keyword child., Second, an object or concept is often described by a set of objects sharing similar schema and these objects frequently start with the same keyword. Third, if the parent node is a tag known to exhibit list structure (e.g. ul, dl, table), its child nodes are very likely to be objects at the same level of abstraction. Hence, if a sibling of the chosen keyword node is labeled with the same keyword, or their parents have the same tag label, or their parent is a list structure, sibling relationship with the chosen keyword is maintained.

[0117] For example, given the following hypertext document:

[0118] <ul>

[0119] <Degree/>

[0120] <University/>

[0121] <Thesis/>

[0122] <Degree/>

[0123] <University/>

[0124] </ul>

[0125] with its corresponding schematic structure:

[0126] <ul>

[0127] <Degree/>

[0128] <University/>

[0129] <Thesis/>

[0130] <Degree/>

[0131] <University/>

[0132] </ul>

[0133] the schematic structure is identified based on the following:

[0134] 1) <ul> is a tag known to exhibit list structure, so it is likely that it contains a set of child nodes of similar schematic structure;

[0135] 2) node Degree describes the topic of the nodes under <ul> because it is the first keyword child node of <ul>, and

[0136] 3) two of the child nodes of <ul> are labeled with the same keyword Degree, and these nodes are likely to be “topic sentences” for other nodes.

[0137] Also, if the previous parents (i.e. the non-keyword nodes replaced) of two sibling nodes have the same tag label, it is likely that they are marked-up in the same way and hence, share similar schematic structure. Sibling relationship is thus maintained for these nodes. For example, using this heuristic, the HTML fragment:

[0138] becomes

[0139] The two <li> nodes are replaced by Job and Title respectively. These consecutive sibling keyword nodes have the same previous parent <li>, which hints that they contain information sharing similar schematic structure.

[0140] A formal definition of this consolidation function, Δ(T(node)), is:

[0141] Let L ⊂ M be a set of list tags whose child nodes contain information at the same level. Define function beSibling: $\mathrm{beSibling}\left(k_i,k_j\right)\equiv \{\begin{array}{cc}\mathrm{True}& \mathrm{if}\mathrm{Label}\left(k_i\right)=\mathrm{Label}\left(k_j\right)\mathrm{or}\\ & \mathrm{Label}\left(\mathrm{PrevParent}\left(k_i\right)\right)=\\ & \mathrm{Label}\left(\mathrm{PrevParent}\left(k_j\right)\right)\ne \perp \mathrm{or}\\ & \mathrm{Parent}\left(k_i\right)\in L\\ \mathrm{False}& \mathrm{otherwise}\end{array}$

[0142] Function beSibling states that if node k_i, k_jhave the same label, or their previous parents have the same label, or their parent is labeled with a list tag, then the two nodes maintain their sibling relationship.

[0143] Let k≡{k₁, . . . , k_p|Label(k_i)∈K, k_i∈Children(node), k_i<k_i+1, beSibling(k₁, k_i)=true}

[0144] where k is the set of child nodes of node which need to maintain sibling relationship. $\left(T\left(\mathrm{node}\right)\right)\equiv \{\begin{array}{c}T\left(\mathrm{node}\right)\\ \mathrm{If}\mathrm{Label}\left(\mathrm{node}\right)\in K\\ \Delta (\mathrm{Replace}\left(T\left(\mathrm{Parent}\left(\mathrm{node}\right)\right),\mathrm{node},<T^{\prime }\left(k_1\right),\dots ,T^{\prime }\left(k_p\right)>\right)\\ \mathrm{If}\mathrm{Label}\left(\mathrm{node}\right)\notin K\mathrm{and}\mathrm{Parent}\left(\mathrm{Parent}\left(\mathrm{node}\right)\right)\ne \perp \\ \mathrm{Replace}\left(T\left(\mathrm{Parent}\left(\mathrm{node}\right)\right),\mathrm{node},<T^{\prime }\left(k_1\right),\dots ,T^{\prime }\left(k_p\right)>\right)\\ \mathrm{Otherwise}\end{array}$

[0145] where $T^{\prime }\left(k_i\right)\equiv \{\begin{array}{cc}\mathrm{MakeChild}\left(k_i,k_{i+1}\right)& \mathrm{if}i<p\\ T\left(k_i\right)& \mathrm{if}i=p\end{array}$

[0146] Function Δ states that a non-keyword node is replaced by its first keyword child node, k₁, and the other child nodes k₁, . . . , k_pwhose sibling relationship with k₁must also be maintained. Nodes between k_i, k_j+1, become child nodes of k_iin the new subtree T′(k_i). Function Δ is recursively applied to the new tree until a root which is not replaced by any keyword node is reached.

[0147] FIGS. 8aand 8bshow the final form of the example document after applying the consolidation function with L={ul}. The discovered schematic form of this document, as depicted in FIG. 8b, is now:

[0148] One useful application of the schematic structure discovered is to support structural queries in search engines. These search engines are able to use the schematic structure of keywords to locate data that are more semantically related to the information users are searching for. For example, using simple keyword searching, a job recruiter looking for candidates who hold a masters degree from University of California may uncover candidates who have simply lived in California and received a masters degree elsewhere.

[0149] However, using the discovered schematic structure

[0150] <University>

[0151] <Degree/>

[0152] <Date/>

[0153] </University>

[0154] the recruiter is now able to locate only those candidates holding masters degrees from University of California by specifying a match of both the degree and university.

[0155] Very often a set of related hypertext documents share similar schematic structure but their common schema is not explicitly specified. The schematic structure, once discovered using the present invention, from this set of related hypertext documents will serve as the input to discover this common schema. For example, a common schema for a set of hypertext documents similar to the example document, expressed as an XML DTD, would be:

[0156] <!ELEMENT html (Education, Experience)>

[0157] <!ELEMENT Education (Degree, (Organization, Date, GPA*))>

[0158] <!ELEMENT Experience (Title, Organization, Date))>

[0159] <!ELEMENT Degree #PCDATA>

[0160] <!ELEMENT Organization #PCDATA>

[0161] <!ELEMENT Date #PCDATA>

[0162] <!ELEMENT GPA #PCDATA>

[0163] <!ELEMENT Title #PCDATA>

[0164] A system and method have been shown in the above embodiment for the effective implementation of identifying keywords and their schematic structure from hypertext documents using heuristics guided by domain knowledge. While various embodiments have been shown and described, it will be understood that there is no intent to limit the invention by such disclosure, but rather, it is intended to cover all modifications and alternate constructions falling within the spirit and scope of the invention as defined in the appended claims. For example, the present invention should not be limited by computer operating system, computer hardware platform, specific document source language, or specific document target language, In addition, the present invention can be implemented locally on a single PC, connected workstations (i.e. networked-LAN), across extended networks such as the Internet or using portable equipment such as laptop computers or wireless equipment (RF, microwaves, infrared, photonics, etc.).

[0165] Computational Complexity

[0166] Below we give an analysis on the computational complexity of the structuring functions. We show that each structuring function is applied to the nodes in the tree at most once, which implies that the running time of our heuristics is linear to the size of the tree. We also prove that applying each heuristic more than once does not change the tree structure. Hence, we only need to apply the heuristics once to attain stability.

[0167] Theorem 1 (One Pass Property) In computing F(T(node)), F ∈ {Θ, Γ, Φ, Ψ, Δ}, F is recursively applied to each node in T(node) at most once.

[0168] Proof

[0169] F is only recursively applied to the child nodes of node satisfying the conditions defined by F. Since a node cannot be a child of itself, once the function is applied, it is never applied to it again.

[0170] Theorem 2 (One Time Property) F(F(T(node)))=F(T(node)), F ∈ {Θ, Γ, Φ, Ψ, Δ}. Hence, we only need to apply each structuring function to the tree once. Applying each function more than once does not change the tree structure.

[0171] Proof

[0172] Case I: F=Θ

[0173] Θ changes the tree structure of text nodes only. Once a text node is tokenized into tokens, the tokens cannot be further tokenized. Hence, when applying Θ to Θ(T(node))), Θ is not recursively applied to node. The tree structure of the node remains unchanged thereafter.

[0174] Case II: F=Γ

[0175] This is similar to Case I.

[0176] Case III: F=Φ

[0177] In Φ(T(node)), a child p_iof node is either labeled with a phrase tag or its next sibling is labeled with the same phrase tag. Hence, in computing Φ(Φ(T(node))), T¹(p_i)=T(p_i). The tree structure of T(node) is unchanged.

[0178] Case IV:F=Ψ

[0179] This is similar to Case III.

[0180] Case V: F=Δ

[0181] Δ reorders the tree by replacing all non keyword nodes (except the root) by keyword nodes. Hence in Δ(T(node) all nodes except the root are keyword nodes. Applying Δ to Δ(T(node) does not have any effect on the tree.

[0182] Empirical Results

[0183] Experiment Environment

[0184] We implement the heuristics by combining them into two passes, one top down tree traversal for tokenization, keywording, phrasing and heading, one bottom up tree traversal for consolidation.

[0185] Domain knowledge specifies the following annotation of hypertext markup tags:

[0186] TS={′,′,′.′,′,}

[0187] P={h1, h2, h3, h4, h5, h6, h4, div, p, tr, dt, dd, li, title, u, strong, b, ern, i}

[0188] H={h1, h2, h3, h4, h5, h6, hr, div, center}

[0189] L={body, table, dl, ul, ol, dir, menu}

[0190] There are 24 keywords and a total of 233 synonyms specified in domain knowledge.

[0191] The data is obtained from a collection of resumes available as HTML in the web. A web crawler was used to quickly obtain these resumes. The original hypertext documents are first converted to well formed XML documents before their schematic structure is extracted. Hypertext markup tags not considered in the domain knowledge, such as font, a, meta, dir, are ignored.

[0192] We ran our experiment on a Pentium 266 MHz processor with 196 MB main memory and 512 B cache.

[0193] Efficiency

[0194] We ran the algorithm on 100 hypertext documents to evaluate the average running time of one hypertext document. The average number of nodes of a hypertext document in the dataset is 203 of which 52 are keyword nodes. The average file size is 11.7 KB. The average running time was found to be 0.53 seconds.

[0195] We measure the scalability of our heuristics with respect to the sizes of the hypertext documents, the number of nodes in the hypertext documents and the number of keyword nodes.

[0196] The heuristics scaled linearly with the sizes of the documents which estimates the total length of text nodes and the number of nodes in the documents. Tokenize and keywording functions depend on the total length of all text nodes, and all structuring functions depend on the number of keyword nodes. Therefore, the experimental results confirm the One Pass Property.

[0197] Also, the running time does not bear very strong linear relationship with the number of nodes. This may seem to contradict One Pass Property. However, the running time of tokenize and keywording functions depends on the total length of all the text nodes, not just the number of nodes. Running time of other functions like phrasing, heading and consolidation functions depends on the number of keyword nodes, not the number of nodes in the hypertext documents. Hence, we expect the running time to scale linearly with the number of keyword nodes instead, which was indeed observed.

[0198] In sum, the heuristics scales linearly with the document sizes and the number of keyword nodes.