Document search and analyzing method and apparatus

[0001] The present invention relates to a document search and analyzing method and a document search and analyzing system for carrying out a document search and analysis in a communications network like the internet, a corporate intranet, etc.

[0002] From the published international patent application W000-04463 a program logic for displaying text passages relevant to the solution of a task like a search task is known, wherein the relevant text passages are displayed on the screen upon entering a combination of search criteria. The text passages are displayed in concentric order around the combination of the search criteria. The radial distances of the individual text passages express their relevance to the combination of the search criteria.

[0003] Richard H. Fowler et al., Proceedings of the 14th Annual International ACM/SIGIR Conference on Research and Development in Information Retrieval, 1991, pages 142 to 151, describes a system utilizing visually displayed graphic structures and a direct manipulation interface to supply an integrated environment for document retrieval. A common visually displayed network structure is used for query, document content and term relation. A query can be modified to direct manipulation of its visual form by incorporating terms from any other information structure the system displays.

[0004] From John Lamping et al., Chi '95 Conference Proceedings, Denver, May 7 to 11, 1995, pages 401 to 408, a focus and context technique for visualizing and manipulating large hierarchies is known. The technique lays out the hierarchy in a uniform way on a hyperbolic plane and maps this plane on to a circular display region. This supports a smooth blending between focus and context as well as continuous redirection of the focus.

[0005] From Allen Ginsberg, IEEE Expert, October 1993, pages 46 to 56, a knowledge representation framework is known, which uses a lattice-structured version of the traditional thesaurus.

[0006] From M. Hemmje et. al., SIGIR '94 Conference, Dublin, Jul. 3 to 6, 1994, pages 249 to 259, a visualization interface for an abstract information space is known. Visualizations are used to communicate information search and browsing activities in a natural way by applying metaphors of spatial navigation in abstract information spaces.

[0007] G. G. Robertson et al. in Human Factors in Computing Systems Conference Proceedings, Reading, USA, Apr. 27, 1991, pages 189 to 194 describes three 3Dvisualisations of hierarchical information in the form of cone trees. This enables most effective use of the available screen space and enables the visualization of a whole hierarchical structure.

[0008] From U.S. Pat. No. 6,038,562 an interface is known to support state-dependent web applications accessing a relational database.

[0009] The continuing growth of the internet in recent years has made search engine services a popular tool for retrieving documents on the internet and other communication networks. The user normally enters a search term and probably some additional parameters like the document language or the document age and receives from the server, where the search engine service is located, on his or her client computer a so called hit list containing the web addresses of a large number of documents indexed with the search terms and taking into account the additional parameters. In most cases the list of documents is very long and only a few of those placed on top of the hit list will be looked at by the user. The order of the found documents is often determined by so called metatags placed in the documents. Many commercial websites use popular search terms as metatags in order to prominently appear on popular search engines.

[0010] Using the first result list the user can then refine his search by inputting further search terms or parameters. Repeating this operation several times will reduce the search result to a handable size but bares the risk that during the search valuable documents are missed.

[0011] There exists therefore a need for improved document search services in the internet and other communication networks providing the user to perform a more specific search and result analyzing strategy.

[0012] The present invention provides a document search method in a communication network, comprising the steps of providing a hierarchical query data structure containing a plurality of search terms, displaying a graphical representation of the query data structure on a display screen, providing a user interface for selecting search terms from the query data structure using the graphical representation, carrying out a document search based on the query data structure, and outputting the found documents as search result.

[0013] The search results can then be qualified according to the number of search terms of the query data structure that are contained in or assigned to a scanned document. Preferably, the search terms of a query data structure are arranged in different hierarchical levels. So the quality of a document or set of documents is qualified differently for search terms of different hierarchical levels.

[0014] The query data structure may be displayed in a two-dimensional or three dimensional graphical representation. Preferably, the query data structures may be stored in a memory device and every query data structure is assigned a unique identifier.

[0015] Preferably, the search result is also displayed as a graphical representation thereof, wherein the quality or matching properties of a document or document set may be expressed by a linear or circular display position or by a color display or the like. For certain standard search tasks the search system preferably provides, like an expert system, model query data structures. Moreover, it is possible to combine two or more query data structures to form a clustered query data structure.

[0016] The present invention further provides a document search method in a communication network, comprising the steps of providing a query data structure containing a plurality of search terms, carrying out a document search based on search terms selected from the query data structure, generating a graphical representation of the search result dependent on the match properties of the searched documents and a set of additional result parameters, providing a user interface for controlling the graphical representation of the search result dependent on the match properties and/or the result properties, and displaying the graphical representation of the search result on a display medium.

[0017] The two-dimensional or three-dimensional graphical display of the search result reflects the match properties of a particular document or set of documents with respect to the search terms. The result representation can be adapted by the user, for example by differently weighting of search terms or by additionally taking into account result parameters like the document size, language, publication date server address or domain extension.

[0018] One document, which in view of the user fits ideally to the search, may be selected as an ideal document for future search or analyzing purposes.

[0019] Preferably, a number of model result display profiles are provided, which may be modified by the user or automatically adapted to the user's behavior by a learning algorithm.

[0020] For carrying out a continuous watch a search based on a specific query data structure may be carried out repeatedly after a predetermined time period, for example every week or every month. The new results are compared to the old ones and the differences are shown in the graphical representation.

[0021] The method may further include the step of simulating a form wrapper or accessing data bases which acquire a special access form. These forms are preferably updated automatically without requiring further user interaction.

[0022] Preferably, the ontology editor includes functions like automatic check of multiple use of search terms and tracking the building steps of a query structure. It is also possible to provide a thesaurus function for providing synonymous terms, language recognition and translation functions for translating search terms to a different language and for outputting a definition of a selected search term.

[0023] The invention still further provides a system including one or more server computers, comprising a scanner scanning a communications network and providing a scan list, a client interface for selecting, from a client device, search terms from a query data structure containing a plurality of search terms in a hierarchical order, an ontology indexer matching the documents stored in the scan list with the search terms contained in the query data structure (ontology) and indexing the documents dependent on the occurrence of one or more of the search terms in the document, and an output client interface for outputting search results for display on a client device.

[0024] The present invention still further provides a document search method in a communication network, comprising the steps of providing a query data structure containing a plurality of search terms, carrying out a document search based on search terms selected from the query data structure, generating a graphical representation of the search result dependent on the match properties of the searched documents and set of additional result parameters, providing a user interface for controlling the graphical representation of the search result by its dependence on the match properties and/or the result properties, and displaying the graphical representation of the search result on a display medium.

[0025] Further preferred embodiments and variations of the invention are described in the dependent claims.

[0026] The present invention and further objects, features and advantages thereof will become apparent from the following description of preferred embodiments in connection with the drawings in which

[0027] FIG. 1 shows a schematic block diagram of a preferred embodiment of the present invention;

[0028] FIG. 2 shows a flow chart of information retrieval steps of a preferred embodiment of the present invention;

[0029] FIG. 3 shows a flow chart of method steps of handling a client request of a preferred embodiment of the present invention;

[0030] FIG. 4 shows an example of a preferred user screen layout according to a preferred embodiment of the present invention;

[0031] FIGS. 5.1 to 5.4 show flow charts of client method steps according to a preferred embodiment of the present invention;

[0032] FIG. 6 is a flow chart illustrating the function of the result space sub-system of a preferred embodiment of the present invention;

[0033] FIG. 7 is a flow chart showing method steps of the dynamic data filtering function of a preferred embodiment of the present invention;

[0034] FIG. 8 shows a graphical representation of the first hierarchical level of query data structure containing three search terms;

[0035] FIG. 9 shows an example of the graphical representation of the highest level of a search result; and

[0036] FIG. 10 shows the graphical representation of a second level of a search result.

[0037] FIG. 1 shows schematically the basic design of a preferred embodiment of a document search system of the present invention. The modules of the system are divided upon the provider side 100 and the client side 200. It has to lea acknowledged, however, that some modules may be located differently as shown in the embodiment of FIG. 1. It is, for example, also possible to provide the template engine 220 as part of the provider side generating query space 230 and result space 240 for download by the client, 200. Moreover, the provider side 100 need not to be confined to one server computer. The units may be divided upon a plurality of server and database systems. As client side any suitable terminal device like a personal computer, a laptop computer or an internet enabled mobile phone may be employed. Communication between provider side and client side is preferably carried out over the internet or any other network. Alternatively, client and server can run on the same platform, searching the local memory.

[0038] The provider side, on the one hand, comprises the information retrieval unit 110 and, on the other side, the client handler unit 120.

[0039] The information retrieval unit 110 contains those functional blocks dealing with the information retrieval from the internet or a different communication network like a corporate intranet.

[0040] The crawling (downloading of webpages) is done by the so called scanner 111. The scanner reads instructions of a job providing URLs of target websites, visits the provided websites, follows all links on the pages of the website according to the instructions in the job and stores various information about the found links associated with an unique ID in a scan list, which is stored on a storage device. The scan list is the basis for indexing the content of the websites.

[0041] There are provided two methods of indexing the documents. The first indexing method provides a full text index and the second indexing method a so called ontology index.

[0042] The full text indexing function is performed by the full text indexer 114. It fetches each document from the scan list and parses it. When parsing the document, it creates a new entry for each word in the so called word index, that is not yet contained in it, and associates it with a unique word ID. It may also create a prefix tree which is a special version of the word index that enables prefix search. Furthermore, a full text index is created which stores the relation between the documents and the word index.

[0043] The ontology indexing function is performed by the ontology indexer 113. Similar to the full text indexer it fetches each document found in the scan list and parses it. The ontology indexer uses as second input source for the indexing function an ontology which will be described in more detail later. The ontology is comprised of a system of related concepts that describes a certain expert knowledge for carrying out the search. The concepts encapsulate certain terminology that is likely to be used to describe a named concept in the text. The terminology in the concept is encoded in regular expressions. When the ontology indexer passes the web document it matches the regular expressions from the concepts against the text and thus associates match concepts with the document and stores it in an ontology index. The form wrapper 112 simulates filling in a form for database access. The form wrapper monitor 115 recognizes changes of forms and informs the administrator or the form wrapper adjuster 115 which automatically updates the forms wrapper.

[0044] The client handler 120, on the other hand, is responsible for handling client requests. The client handler can be broken down in two major sub-systems, mainly the request handler 121 and the search engine 125.

[0045] The request handler is responsible for inputting client requests, passing these requests on to the other sub-systems for processing and returning the appropriate server response. The request handler may be implemented as Java Servlet or any other server-sided technology (cgi, php3, etc.) attached to a webserver. It is also possible to provide several request handlers for different client requests, for example for full text search or for concept search requests.

[0046] The search engine sub-system 125 is responsible for processing search queries and consists of the concept search engine 126 and the full text search engine 127 for carrying out concept searches and full text searches, respectively. It is, however, also possible in a client request to combine full text and concept search.

[0047] The client side or client 200 comprises a client applet 210 being responsible for the communication with the server, a template engine 220 for generating display representations of the query data structure and the search result and the weight preference profiler 260, the parameter controller 270 and the query monitor 280, which will be described in more detail later.

[0048] The query space builder 221 generates the query space 230, that is the two dimensional or three-dimensional graphical representation of the query data set or ontology. A result builder 222 generates the result space 240 for displaying a graphical representation of the search results in 2D or 3D. Another client, the ontology editor, is provided for administrative purposes.

[0049] FIG. 2 is a flow chart showing the method steps carried out by the information retrieval module 110 for obtaining the necessary information required for obtaining the search results.

[0050] A search system and method of the present embodiment uses a collection of websites as the target of the search. Usually the information or service provider has the URLs of these websites stored in some kind of web directory, either categorized or as hot link lists. In any case, a job is created for each URL of a website which contains beside the address different instructions about how the links of the site should be followed, etc.

[0051] The scanner then carries out the tasks contained in the job and produces the corresponding scan list. The full text indexer 114 uses this scan list to produce a full text index, a word index and/or a prefix tree. The ontology indexer 113 uses the ontology for generating an ontology index of the documents contained in the scan list.

[0052] The operation of the client handler is illustrated in FIG. 3.

[0053] As a starting point, the request handler 121 receives from the client a client request, either containing a request for a certain query space or a search string. The query space is produced by the query space builder 221 and sent to the client. A concept search is handled by the concept search engine 126 using the scan list and the ontology index. The full text query is handed over to the full text search engine 127 for executing a full text search using the word index, full text index and prefix tree (see FIG. 2). The document IDs of those documents in which the search term appears are returned by the search engine as search result to the request handler and subsequently to the client in the form of a result set or a document list.

[0054] The search result is then transformed into a graphical representation by the result builder 222 and display on the client display screen.

[0055] Preferably, the client side runs in a web-browser and is implemented using, for example, Java, Java Script, HTML and VRML (virtual reality modeling language). The communication between the different components in the different frames shown in FIG. 4 is preferably accomplished by using a Java Script bridge. The VRML frame is used for displaying the query data structure (ontology) as well as the search result. The client applet section may contain further sub-sections for providing additional information for the user as well as a parameter control section.

[0056] In the following the method of creating a query data structure or ontology is discussed. When the user logs in to the system he will be presented a screen display corresponding to that shown in FIG. 4. A number of ontologies are offered for user selection. Then the user clicks on one of the presented ontologies, for example the ontology new media law, a graphical representation of the uppermost level of the ontology is displayed on the screen, as is shown on FIG. 8. The uppermost level contains, in this example, the search terms or concepts technology, commerce, and legal issues. Everyone of the three nodes consists of two sub-nodes which may for example be displayed in different colors. Selecting the first sub-node, for example by a mouse click, opens the next lower level of the concept, in the case of technology for example comprising the concepts internet software and interface.

[0057] Clicking the second sub-node selects this concept or search term for the search. Clicking of the second sub-node preferably also initiates the display of an explanation of the selected search term on the screen.

[0058] After selecting one concept of the second level, for example interface the concepts of the next more detailed level are shown, in this example e.g. graphical user interface, programming interface and human computer interface. By selecting the concepts the user can so configure the query for carrying out the document search. A navigation through the three-dimensional virtual ontology space allows the user to intuitively understand and refine his search strategy.

[0059] When the query is finished the server executes the document search as has been described above in connection with FIGS. 2 and 3. The search result is then also provided as a graphical representation of the found documents or document sets dependent on the concept contained in the search ontology. FIG. 9 shows an example of a graphical representation of the uppermost level of the search results. Four result fields are recognizable wherein the arrow on the lower left side points in the direction of the best matching between the search terms and the found documents or document sets. In the shown example the query contains four different search terms or concepts, for example those three shown in FIG. 8 and the additional search term internet. The result field at the tip of the arrow contains the found documents corresponding to all four concepts. The next result space contains those documents with three of the four terms, then followed by three different result sets each containing two of the search concepts and then those documents including one of the search terms. The height of the column represents the number of documents found. Preferably different colors represent different search terms.

[0060] If the user now clicks to one of the result fields or one of the columns shown in FIG. 9 he will be presented the more detailed results of the next lower level of the search result. The example shown in FIG. 10 is the more detailed view of the best results of the right-most result space of FIG. 9. The picture shows three documents which each contain all four concepts represented by differently colored columns. The different heights of these columns show how often the concept or search term appears in the respective document. If the user clicks to one of the three documents shown he will be linked automatically to the address of the respective document. The triangle on the right-most document shows that this document has already been visited by the user.

[0061] In order to improve and personalize the result analysis it is possible to display the result representation also on other parameters than the matching property. These parameters enclose document parameters like the document size, the date of the last modification, the language of the document, document ID etc. and server parameters like the server size, the number of matching documents of one server, the domain extension etc.

[0062] Dependent on these parameters the visualization of the search result can be adjusted in order to optimize the result visualization. The visualization properties which can be varied include the position of a document representation, its orientation, size, form, icon, visibility, color, transparency or assigned labels. For lower level documents the visualization properties include a clustering of objects, ranking the object, focussing and emphasizing objects.

[0063] It is for example possible for a user to include in the displayed results only documents having a size between 5 and 50 pages, being in English, German or French language and being up-dated no longer than twelve months ago. It is also possible to explicitly exclude or include specific servers or domain extensions (corn, org, Ant, at, .de).

[0064] These adjustments are preferably carried out by the parameter controller 270 using an interactive graphical user interface.

[0065] The parameter controller 270 allows a user to change the weight of different concepts for analyzing the results. Different search terms can therefore have different importance for the qualification of the search result. This allows the user to personalize the displayed search result representation. The weight preference profiler 260 is a learning algorithm which automatically adjusts the display parameters depending on the user's behavior.

[0066] With the query monitor 280 it is possible to carry out identical or similar searches on a regular basis, for example every week or every month. The results are then available for the user after logging in to the system. The new results are compared to the old ones and the differences are shown in the graphical representation. FIGS. 5.1 to 5.4 show the method steps of the result visualization and analysis according to a preferred embodiment of the invention.

[0067] The operation shown in FIG. 5.1 is the standard case: The user selects a query using the query space 230 and sends it to the server, the server generates a result and sends it back to the client. There, the template engine 220 produces the (static) visualization model which is then rendered.

[0068] In the operation shown in FIG. 5.2 the user uses the parameter controller 270: The user selects a query using the query space 230 and sends it to the server, the server generates a result and sends it back to the client. There, the template engine 220 produces the (static, for the beginning) visualization model which is then rendered. Until here, the process is exactly the same as in FIG. 5.1. Now, the user modifies parameters using the parameter controller 270. This provokes the template engine 220 to produce an parameterized update of the visualization, which is then rendered.

[0069] In the operation shown in FIG. 5.3 the user uses the weight preference profiler 260. In phase 1, the weight preference profiler 260 learns a profile: The weight preference profiler 260 knows the result, the user modifies parameters using the parameter controller 270. Now the weight preference profiler 260 can either use this modification for learning after the user told him to do so (teaching mode), or he can watch the user's actions automatically (watchdog mode). In both cases, the weight preference profiler 260 saves the combination result/parameter settings. This procedure is repeated until an adequate amount of samples exists.

[0070] In phase 2, the weight preference profiler 260 applies the profile. The user sends a query, the server returns a result, the template engine 220 produces a (static) visualization. Now there are two possibilities: The user can ask the weight preference profiler 260 to adjust the parameters for the new result using the profile, or the weight preference profiler 260 does this automatically. Both actions provoke the template engine 220 to produce an parameterized update of the visualization, which is then rendered.

[0071] In the operation shown in FIG. 5.4 the user uses the query monitor 280 for monitoring a query over a longer period: The user charges the query monitor 280 with a monitoring job. The query monitor 280 saves query and result and parameter settings. The user defines a monitoring frequency. Depending on this frequency, the query monitor 280 sends the query to the server again, and receives a new result. Now the query monitor 280 compares this result with the saved one. If he finds differences, he sends a message to the user. Now the user can call the result including the parameter settings, which contains visualizations of the differences.

[0072] FIG. 6 illustrates the result space sub-system of a preferred embodiment of the present invention.

[0073] The user controls by means of an interactive result set manipulator (preferably on the display screen) the parameter controller 270 to change the result document set in dependence on result parameters like the document size, a language, update age etc. On the other hand the user can also manipulate the visual appearance of the displayed results by navigation through result space.

[0074] FIG. 7 illustrates the dynamic filtering with the parameter controller 270.

[0075] Each property of the data model of the search result is mapped to a property of the visualization-model. A modifier is assigned to each pair of data properties/visualization properties. A value of each modifier can be changed by a manipulator (compare FIG. 6), which is implemented by a user interface component. Each time the value of a modifier is changed, the parameter controller applies this value to the corresponding data property, then reapplies the ranking/sorting/clustering function to the result data model and maps the data model again onto the visualization model. The adjusted result visualization is then displayed on the user display.