Information retrieval system providing secondary content analysis on collections of information objects

[0002] The present invention is related generally to the field of information retrieval from one or more distal information sources, and in particular to techniques for enabling a user to more efficiently generate queries and browse the results of queries.

[0003] With the advent of networking technology, the ability to access information from distal information sources has greatly increased. The explosive growth of the Internet and commercial on-line networks and databases (collectively referred to as information sources) are indicative of the high demand for easily accessible information. When interacting with a large information source, a user typically writes a query that returns on the order of 1000 items (which is referred to here as a medium-sized collection). This is about one and a half orders of magnitude more than can be dealt with directly by the user in most task situations. Thus, the user might view a few items, and then narrow the scope and resubmit the query to decrease the number of result items. This cycle is usually repeated many times until the user arrives at a manageable set of documents to view in detail, typically 30-50 documents. This process is referred to as iterative query refinement.

[0004] This process has a number of well-known problems, most notably that it is time-consuming and that it often ineffective for arriving at a good collection of 30-50 documents. Often, overly-aggressive attempts to reduce the result set to a manageable size can lead to queries that lock in on too narrow a specification of what is sought. This happens because only a few items are browsed per cycle, and thus very little available information is used to make refinement decisions in each cycle.

[0005] Various techniques are used for aiding query refinement. A simple technique is to provide the number of hits for each of the search terms, and/or the number of hits for specified logical combinations of the search terms. This helps the user to determine if there is a problem with a particular search term or logical combination (i.e. it is overly broad or too narrow), or if the search requires more terms for narrowing the number of hits.

[0006] Another technique is known as relevance feedback. The essence of this technique is that a user indicates documents in a result set that are relevant. Typically, an automatic procedure is used to reweight, add, or remove terms in a subsequent query (although some systems permit this to be done by the user). Though relevance feedback is most often used in the context of a ranking model of retrieval (i.e. one that scores and orders results), it has also been used in a Boolean context. For example, The European Nuclear Documentation System (ENDS) automatically constructs a Boolean query based on co-occurence of terms in at least 2 documents selected as relevant.

[0007] Other known techniques for supporting iterative query refinement include scatter/gather and snippet search. Each of these techniques provide more efficient means of browsing a collection in order to refine the query.

[0008] The problem of query refinement is exacerbated when the query is directed towards multiple disparate information sources (databases). This is because each of the information sources may have different functional capabilities, different search engines, operate under different protocols, etc. There are no known systems which been designed to provide aids for query refinement on collections of documents that have been obtained responsive to a query to multiple disparate information sources.

[0009] An information retrieval system which provides for secondary content analysis of retrieved collection of documents is disclosed. The collection of documents is the result of a query to one or more information sources, e.g. databases. The secondary analysis may be performed for various reasons such as summarization of the collection, navigation through the collection, understanding the relationship between the documents in the collection or for query refinement. In the currently preferred embodiment, the secondary analysis is comprised of generating statistical information which may be used for query refinement and for more effective browsing of the ephemeral document collection. The analysis performed is termed secondary since some primary analysis may have been performed at the information sources. The secondary content analysis is performed on an Information Access (IA) client which can couple to the various information sources.

[0010] The general method of query, refinement and browsing enabled by the present invention is comprised of the steps of: a user generating a query, the query translated and transmitted to the respective information sources, the results returned and collected, secondary content analysis performed on the returned collection wherein document level and collection level statistics are obtained, the user selects a query refinement option, the statistics are used to provide the selected option, and the user refines their query as needed.

[0011] When the document collection is comprised of a collection of primarily textual documents, the process of generating the secondary content analysis is generally comprised of the steps of: tokenizing document text, filtering the tokens according to the type of statistics to be collected and collecting the statistics from the remaining tokens.

[0012] In addition to aiding the query refinement process, the performance of secondary content analysis on a collection of documents has other advantages. First, it permits more efficient browsing. Second it permits the use of the query refinement and browsing techniques on documents from information sources that do not support such functions.

[0013] FIG. 1 is a block diagram of an information retrieval system for accessing multiple distal information sources as may be implemented in the currently preferred embodiment of the present invention.

[0014] FIG. 2 is a block diagram of an alternative information retrieval system for accessing multiple distal information sources

[0015] FIG. 3 is a flowchart illustrating the general steps for querying, retrieving and query refinement in the currently preferred embodiment of the present invention.

[0016] FIG. 4 is a block diagram describing the functional components of an Information Access (IA) client as may be utilized in the currently preferred embodiment of the present invention.

[0017] FIG. 5 is a flow diagram illustrating the processing of a statistical content analysis engine of the currently preferred embodiment of the present invention.

[0018] FIG. 6 is a block diagram of a computer based system upon which the IA client may be implemented.

[0019] In an information retrieval system, a means for providing secondary content analysis on retrieved collections of documents is disclosed. The secondary analysis performed is for enabling functions that would facilitate better or more efficient use of the retrieved collection. Such functions would include providing better understanding of the collection on the whole, enhancing navigation through the collection, or for query refinement.

[0020] In the currently preferred embodiment, the secondary analysis performed enables refinement of a query to more precisely obtain the desired information. A subsequent query may be to either the retrieved collection or a new query to the multiple disparate information sources.

[0021] While the currently preferred embodiment is described for use in an information retrieval system capable of obtaining documents comprised primarily of text from multiple disparate information sources, the present invention may be used in any information retrieval system that can obtain a collection of documents. For example, the present invention may be used to perform a secondary analysis on a collection of documents comprised primarily of image information. Further, the currently preferred embodiment provides a Statistical Content Analysis (SCA) engine for operation on ephemeral collections of documents. Such collections are ephemeral for the duration of the query. This is not meant to be limiting as the present invention may also operate on persistent collections of documents.

[0022] Overview of the Information Retrieval Architecture of the Currently Preferred Embodiment

[0023] The information retrieval architecture of the currently preferred embodiment is illustrated with reference to FIG. 1. Referring to FIG. 1, an Information Access (IA) client 101 is coupled to a plurality of information sources 102. Connection to the information sources may be through a network, e.g. databases residing on the Internet, or via a direct connection to the communication source, e.g. direct dial-up to Dialog™ Information Source databases.

[0024] The IA clients 101 provide both browsing and search interfaces to the information sources 102. A browsing interface supports the browsing of items and collections provided by the information sources 103 and potentially navigating from one item to other connected items. A search interfaces supports specifying and refining queries, controlling searches, and utilizing the results of searches in various ways. The ephemeral statistical content analysis engine is functionality which reside within the IA client 101.

[0025] Each of the information sources 102 manage a repository of information (e.g. collections of papers, articles or other document types) and provide access to it through some access protocol. The access protocol may be either one of the de facto standards (e.g. Z39.50) or a server specific protocol (e.g. as used in the Dialog™ On-line information source). Thus, the IA client is capable of communicating using the protocol of the server.

[0026] An alternative information retrieval architecture is described with reference to FIG. 2. The alternative architecture incorporates an intermediary server 201 which mediates access requests between the IA client 101 and various information sources. This alternative information retrieval architecture is described in the article entitled “System Components For Embedded Information Retrieval From Multiple Disparate Information Sources”, Ramana B. Rao, Daniel M. Russell, and Jock D. Mackinlay, Proceedings of 1993 ACM Symposium on User Interface Software and Technology, Atlanta, Ga., November 1993 ACM SIGGRAPH and SIGCHI.

[0027] The intermediary server 201 mediates the operations between IA client 101 and information sources 102. The intermediary server 201 acts as a protocol gateway. The intermediary server 201 communicates with the IA client 101 according to a generic protocol optimized for local interactions between a client and server. The intermediary server relays the operation requests to the information sources using the protocol of the information sources. The intermediary server 201 may also provides various services for enabling a user to make more efficient use of the information sources (e.g. information describing the available information sources or estimates on the cost of performing an operation on particular information sources).

[0028] It should be noted that the intermediary server 102 may also be a location for a statistical content analysis engine.

[0029] FIG. 3 is a flowchart illustrating the basic steps for querying, retrieving and query refinement as may be performed in the currently preferred embodiment of the present invention. Referring to FIG. 3, a user makes a query, step 301. It is assumed at this point that the query was one that will require further refinement. The query is processed by the IA client and “translated” for transmission to the information sources associated with the query, step 302. As described above, each of the various information sources may communicate using a different protocol and each may have different search capabilities. In this step any necessary protocol conversions are performed along with any modifications for a particular information source that are necessary in order to make a “best effort” to perform the query as requested. After some period of time the results of the query to the information sources are returned and collected by the IA client, step 303. In this case the entire documents may be downloaded, so what is obtained is a whole collection of full text documents. In any event, what the user now has is an ephemeral collection of documents that have been gathered from multiple information sources.

[0030] General techniques for processing a collection documents to aid retrieval may now be performed on this ephemeral collection. The statistical content analysis engine within the IA client analyzes the resulting collection of documents and generates document and collection level statistics that may later be used to help a user refine their query, step 304. This analysis is described in greater detail below. In any event, when the results are received, the IA client notifies the use as the number of hits, step 305. From this number of hits the user can determine whether or not the query needs refinement. For purposes of this description, it is assumed that the query needs refinement because of an excessive number of hits (if there were too few hits an entirely new query would be performed.)

[0031] At this point it should be noted that a user may not need all the rights currently associated with the downloading of documents. This ephemeral collection of documents could have been borrowed from the various Information Sources or otherwise obtained in a fashion that did not require the user to pay the full cost of receiving the document. The full cost of retrieving the document may be invoked when the user actually “views” the full text of the document. Thus, the present invention permits new types of billing schemes by the information source which may further drive use.

[0032] In any event, next the user selects one of the options for helping them refine their query, step 306. This would include relevance rating, scatter/gather, snippet search or any other practical technique. The appropriate processing is then performed on the ephemeral collection and associated statistics, step 307, and the information displayed to the user in the appropriate manner, step 308. From this, the user may then decide the best means to refine the query, step 309. Or as will be described below, sometimes the refined query is generated automatically responsive to user interactions.

[0033] IA Client

[0034] FIG. 4 is a block diagram illustrating the functional components of an IA client. The IA client 401 is typically comprised of a searching interface 402, a browsing interface 403, a query translation part 404, one or more information source connection interfaces 405 and statistical content analysis engine 406.

[0035] The searching interface 402 provides a means by which queries may be submitted to the various information sources. Typically, this would be a string of keywords each having some logical relationship. The way the resulting set of documents would be presented to the user may also be specified.

[0036] The browsing interface 403 provides a means by which the results of a query are viewed by the user. Typically, the browsing interface will render the data representing the document onto the display coupled to the computer based system on which the IA client resides. The rendering may also occur for audio based data. As will be described in more detail below, the browsing interface 403 may also use data generated by the ephemeral statistical content analysis engine 406 to create a more effective “high” level browsing which is used for refining the documents to be viewed.

[0037] The query translation part 404 processes a query into the protocol(s) utilized by each of the various information sources being accessed. The query translation part 404 may also modify the query for a particular information source in the event that the information source cannot support the original request (e.g. a request made that is restricted to a particular field).

[0038] The information source connection interface part 405 is the means by which the respective information sources are accessed. This may be by a direct dial-up connection or via a network connection (e.g. the Internet).

[0039] Finally, the statistical content analysis engine 406 processes an ephemeral collection of documents to create various statistics which may be used for refining a query and for more efficient searching of the information in the collection of documents.

[0040] Typically, the IA client is embodied in a computer based system controlled by operating software and which is interfaced with through a display and various input devices such as a pointer device or keyboard.

[0041] Statistical Content Analysis Engine

[0042] In the currently preferred embodiment, the IA clients are augmented with a special-purpose statistical content analysis (SCA) engine called an ephemeral SCA engine. It is the ephemeral SCA which performs a secondary content analysis. An ephemeral SCA engine is inherently useful since it permits the use of various query modification techniques on information sources that don't directly support such features.

[0043] An SCA engine processes documents by treating their contents as a sequence of tokens and gathering collection and document level object and token statistics (most notably token occurrence). A contiguous character string representing a word is an example of a token. So in the currently preferred embodiment of the present invention, an ephemeral collection of documents is processed by the SCA engine to yield various indexes and index terms that will supplement query refinement. A suitable process for analysis of a collection of documents (or database) is described in section 5 of an article entitled “An Object-Oriented Architecture for Text Retrieval”, Doug Cutting, Jan Pedersen, and Per-Kristian Halvorsen, Xerox™ PARC technical report SSL-90-83.

[0044] The ephemeral SCA engine is specialized for usage on short-lived, dynamically-generated medium-sized collections that arise during interaction with the information sources. A medium sized collection is defined as approximately 1000 documents. In particular, this means that the tokenizing and indexing processes are tuned for rapidity, not persistence, reliability, or large collections as is typically the case for full-text statistical content analysis engines. Moreover, the tokenizing and indexing processes may be tuned for the specific types of query refinement tools supported.

[0045] The process utilized by the ephemeral SCA engine is generally described with reference to the flow diagram of FIG. 5. Referring to FIG. 5, an ephemeral collection of documents 501 provide input to a tokenzier 502. It should be noted that each document is handled independently. The tokenizer 502 extracts tokens by parsing the text of a document and generating a token 503 for each contiguous sequence of characters (e.g. a word). The tokens are then processed through a series of filters 504. The filters will cause some of the tokens to be dropped out and some tokens to be combined. The filters perform such functions as generating stop lists, stemming, part of speech tagging or phrase spotting. Some of the filters may also be parameterized based on the terms used in the query. In any event, after going through the series of filters 504 the remaining tokens 505 that are output are input to a statistics collector 506 for collecting the document level statistics 507 and collection level statistics 508. These statistics may include token occurrences or proximity of words or phrases.

[0046] In order to operate in the environment of the present invention, the SCA engine provides for processing 1000 documents of 1000 tokens in 5-10 seconds. These numbers are estimates about a level of performance that would enable very powerful uses, though many tasks can withstand lesser performance. Such performance number are certainly attainable when one considers the ever increasing processing power of microprocessors as well as the corresponding increases in density of memory devices. As an example, there already exist various memory resident database systems. In such systems both the corpus of the data and the software instructions for accessing the database reside in the internal memory of a computer based system.

[0047] Query Refinement/Browsing Tools

[0048] An ephemeral SCA engine can allow the user to more easily utilize the information obtained by a query to formulate a more restrictive query and to evaluate higher-level strategy options. This is done by the use of various query refinement and browsing tools. The SCA engine of the currently preferred embodiment supports versions of relevance feedback, scatter/gather, snippet search, and similarity search. It is anticipated that other search aids which use the ephemeral SCA engine may be developed as appropriate.

[0049] In relevance feedback a user indicates documents in the ephemeral collection that are relevant. Typically, an automatic procedure is used to reweight, add, or remove terms in a subsequent query (although some systems permit this to be done by the user).

[0050] In scatter/gather and snippet search the focus is on “browsing” the result set to help refine a query. In scatter/gather a result set is automatically “scattered” into small collections of document-groups, along with a short summary. Based on the summaries, one or more of the groups is selected. The selected groups are then “gathered” to create a new result set. The new result set is then “scattered” as before and the process is repeated until groups of sufficient groups are obtained. Scatter/gather is described further in “Constant interaction-time scatter/gather of very large document collections, Cutting, D. R., Karger, D. R., and Pedersen, J. O., Proceedings of SIGIR'93 June 1993, and in “Scatter/gather: A cluster-based approach to browsing large document collections”, Cutting, D. R., Karger, D. R., Pedersen, J. O., and Tukey, J. W., Proceedings of SIGIR'92, June 1992.

[0051] In scatter/gather, the user can take advantage of aggregate properties and a multitude of individual properties of matching items, rather than relying on ad hoc browsing of a few items. The user can then see how the result partitions and utilize terms from cluster term summaries or central documents to manually or automatically refine the query.

[0052] In snippet search portions of context surrounding a search term are displayable. This aids query reformulation by permitting indication of other discriminating search terms. Snippet search is further described in “Snippet search: A single phrase approach to text access”, Pedersen, J. O., Cutting, D. R., and Tukey, J. W., Proceedings of the 1991 Joint Statistical Meetings, American Statistical Association (1991).

[0053] Snippet search provides more detailed evaluation of how exactly items matched the query. For example, snippet search or other proximity techniques can be used to locate interesting passages of matching documents to evaluate them, rather than relying on summary information like titles.

[0054] A similarity search find documents that are near by in a vector space using some distance metric. The vector space is an n-dimensional space with each word as a dimension, and a documents coordinate on each dimension being the number of occurrences of that word. Typically, the distance metric is a cosine of the angle between the vectors.

[0055] Other Uses of Secondary Content Analysis

[0056] Providing secondary content analysis at the IA client level has a number of other advantages:

[0057] 1) If the initial query is believed to be of high recall, further interaction with the information sources can be curtailed, since further queries can be performed locally with respect to the secondary content analysis and the available query refinement and browsing tools.

[0058] 2) The various techniques and advantages (e.g. scatter/gather, relevance feedback) apply when dealing with multiple information sources, together or in sequence. Thus, items obtained from different sources can be browsed using scatter/gather or items found in one source can be used to formulate a query for another source.

[0059] 3) User-generated information like reports or papers can be processed by a secondary content analysis and used to find related information in external sources. For example, a paper describing an invention could be processed and used to generate a query on a database containing issued patents.

[0060] 4) Structuring and analyzing medium-sized collections using the statistics provided by the secondary-content analysis may be of direct value in the larger process of information sense-making, independently of its value in retrieval.

[0061] 5) Secondary content analysis supports new models of billing for the information sources. As described above, an information source may provide documents for the purpose of performing a secondary analysis at a lower cost then viewing. When a document is to be viewed, a different higher price may be used. This may encourage higher use of the information source.

[0062] 6. Secondary content analysis may be used for summarizing the contents of the collection. Such summarization may be used to determine if the collection is worth keeping.

[0063] 7. Secondary content analysis may be used for understanding the relationship between the documents in the collection.

[0064] 8. Secondary content analysis may be used for picking individual documents within the collection. This may be done through an analysis which provides a visualization of the collection.

[0065] Computer Based System of the Currently Preferred Embodiment of The Present Invention

[0066] The computer based system on which the currently preferred embodiment of the present invention may be implemented is described with reference to FIG. 6. Referring to FIG. 6, the computer based system is comprised of a plurality of components coupled via a bus 601. The bus 601 illustrated here is simplified in order not to obscure the present invention. The bus 601 may consist of a plurality of parallel buses (e.g. address, data and status buses) as well as a hierarchy of buses (e.g. a processor bus, a local bus and an I/O bus). In any event, the computer system is further comprised of a processor 602 for executing instructions provided via bus 601 from Internal memory 603 (note that the Internal memory 603 is typically a combination of Random Access or Read Only Memories). The processor 602 and Internal memory ROM 603 may be discrete components or a single integrated device such as an Application Specification Integrated Circuit (ASIC) chip.

[0067] Also coupled to the bus 601 are a keyboard 604 for entering alphanumeric input, external storage 605 for storing data, a cursor control device 606 for manipulating a cursor, and a display 607 for displaying visual output. The keyboard 604 would typically be a standard QWERTY keyboard but may also be telephone like keypad. The external storage 605 may be fixed or removable magnetic or optical disk drive. The cursor control device 606 will typically have a button or switch associated with it to which the performance of certain functions can be programmed.

[0068] Optional elements that could be coupled to the bus 601 would include scanner 608, printer 609, facsimile element 610, audio subsystem 611 and network connection 612. The scanner 608 provides a means for creating a bit-mapped representation of a medium (i.e. a scanned document image). The printer 609 could be used to print the bit-mapped representation after it has been edited. The facsimile element 610 could be used for facsimile transmittal of the bit-mapped representation after it has been edited. Note that the facsimile element 610 could utilize the functionality of the scanner 610 and printer 611 to create the full functionality of a facsimile device. The audio subsystem 611 may be used to render and replay audio based data. The network connection 612 would be used to receive and/or transmit data containing a bit-mapped representation of a medium.

[0069] Thus, an information retrieval system providing secondary content analysis on retrieved collections of documents is disclosed. While the present invention is described with respect to a preferred embodiment, it would be apparent to one skilled in the art to practice the present invention with other configurations of information retrieval systems. Such alternate embodiments would not cause departure from the spirit and scope of the present invention.