Metasearch technique that ranks documents obtained from multiple collections

[0002] The present invention is generally directed to the field of information search and retrieval, and more particularly to techniques for locating and ranking documents and/or other forms of information that are contained in multiple collections accessible via a network.

[0003] Computer networking technology has made large quantities of digital content available to users, resulting in a phenomenon popularly known as information overload; users have access to much more information and entertainment than they can absorb. Significant practical and commercial value has therefore been provided by search technologies, whose goal is to identify the information that is of greatest utility to a user within a given content collection.

[0004] The quality of a search is typically quantified by two measures. First, a search should find all the information in a collection that is relevant to a given query. Second, it should suppress information that is irrelevant to the query. These two measures of success correspond to recall and precision, respectively. A search is considered less effective to the extent that it cannot maximize both measures simultaneously. Thus, while one may be able to increase recall by relaxing parameters of the search, such a result may be achieved only at the expense of precision, in which case the overall effectiveness of the search has not been enhanced.

[0005] A metasearch combines results from more than one search, with each search typically being conducted over a different content collection. Often, the various content collections are respectively associated with different information resources, e.g. different file servers or databases, in which case the metasearch is sometimes referred to as a distributed search. The present invention is concerned with the difficulty of maximizing both the recall and precision of a metasearch, particularly one that is conducted via distributed resources. The following discussion explains the sources of such difficulty.

[0006] For simplicity of exposition, the issues will be discussed herein with reference to keyword-based queries of text-based content. As practitioners familiar with the field will recognize, the disclosed principles are easily generalized to queries of text-based content that are not purely keyword-based (such as natural-language queries into parsed documents), as well as to queries of content that is not text-based (such as digital sounds and images). The applicability of the present invention to methods for processing such queries will be readily apparent to those skilled in the art.

[0007] To facilitate an understanding of the invention, the following definitions are used in the context of exemplary keyword-based searches that are employed to describe the invention. A “term” is defined to be a word or a phrase. A “query” is a set (mathematically, a bag) of terms that describes what is being sought by the user. A “document” is a pre-existing set of terms. A “collection” is a pre-existing set of documents. A “metacollection” is a pre-existing set of collections.

[0008] A ranked search is the procedure of issuing a query against a collection and finding the documents that score highest with respect to that query and that collection. The dependence of each score on the entire collection often stems from the well-known technique of weighting most strongly those search terms that are least common in the collection. For example, the query “high-tech farming” would be likely to select the few documents in a computer collection that contain the term “farming”, and the few documents in an agriculture collection that contain the term “high-tech”.

[0009] A metasearch is the procedure of responding to a query against a metacollection by combining results from multiple searches. For the metasearch to be maximally precise, it should find the documents that score highest with respect to the metacollection, not those that score highest with respect to the individual collections in which they reside. For example, in a metasearch over the two aforementioned collections, if a query contains the term “computer,” an incorrect implementation would give undue weight to computer-related documents that appear in the agriculture collection. The practical impacts of this effect are substantial to the extent that a metacollection is used to cull information from diverse collections, each with a different specialty or focus.

[0010] A process that executes an individual search is called a search engine. A process that invokes search engines and combines results is known as a metasearch engine. FIG. 1 depicts the general components of a metasearch system. Typically, the user presents a query to a metasearch engine 10. The metasearch engine forwards this query on to multiple search engines 12a, 12b. . . 12n, each of which is associated with a collection 14a-14nof information content, e.g. documents 15. Most documents are likely to appear in only one collection. However, some documents can appear in more than one collection, as depicted by the overlap of the sets of documents 15 in collections 14band 14n. In such a case, multiple references to a document can appear in the results of a metasearch which employs both of these collections. A well-designed metasearch engine attempts to remove duplicates whenever possible.

[0011] The relationship between search engines and collections need not be one-to-one. For example, as depicted in FIG. 1, two different search engines 12band 12cmay both execute a query against the same collection 14b. In the context of the present invention, this situation is considered to be within the meaning of executing a query on different collections, namely the collection 14bas processed by the search engine 12b, and the collection 14bas processed by the search engine 12c. In some cases, the two search engines could operate with different sets of heuristics. In such a situation the two search engines might produce different results, e.g., different rankings within the respective documents of the same collection. In the particular situation depicted in FIG. 1, since some documents are common to collections 14band 14n, three references to those documents could be returned to the metasearch engine by search engines 12b, 12cand 12n, respectively.

[0012] The metasearch engine 10 and the various search engines 12 execute on computers that communicate with one another via a network. In a fully distributed metasearch, each engine 10, 12 executes on a different machine. In a less distributed system, two or more of these engines may execute on the same machine. For instance, the search engines 12aand 12bmay execute on the same computer 16, or the metasearch engine 10 and one or more of the search engines 12 may execute on the same computer. Similarly, the various collections 14 may reside in different respective storage systems, or any two or more of them can share a common storage facility. The efficiency with which information is exchanged between the metasearch engine 10 and the various search engines 12 via the network is a significant factor in the overall user experience.

[0013] In a system that implements metasearch capability, it is desirable to identify the documents that score highest with respect to the metacollection, i.e. the totality of the collections 14a-14n. The more significant components of the system are the search engines, the metasearch engine, and the protocol by which they communicate. When the search engines exist on different machines in a distributed network, it is further desirable for the communication protocol to minimize the amount of latency perceived by the user, as well as the resource burden in terms of bandwidth and processing power.

[0014] Numerous metasearch implementations exist in the commercial world and in the academic literature. Because of fundamental differences in approach, these vary significantly in precision. FIG. 2 illustrates a taxonomy of the various implementations for metasearch techniques. Before discussing these implementations, however, one distinctive concept should be noted. Much of the prior art operates by centralized indexing, which is not a form of metasearching. With centralized indexing, the original documents remain in their distributed locations but an index database is stored in a central location. The index is built by “crawling”, i.e. copying each document to the centralized facility to be indexed. Unless the copy of a document is required for future retrieval from the central location, it can be destroyed after it has been indexed. Relative to metasearching, central indexing schemes have three main disadvantages. First, because the indexing process takes time, the index is more difficult to keep up to date. For example, crawling the entire Internet takes weeks. Second, unless a protocol is in place specifically to skip the indexing of unchanged documents, the indexing process wastes processing time at the central index and network bandwidth. Third, the hardware resources required to store the central index and to execute queries against it grow at least linearly with the size of the collection being indexed. Thus, metasearch techniques provide significant practical and commercial advantages.

[0015] Turning now to FIG. 2, metasearch techniques can be classified into two broad categories, Boolean searches and ranked searches. The weakest metasearch techniques apply Boolean rules that either accept or reject each document, and do not supply a score. Thus, they cannot prioritize results in the event that the result set is too large for the user to consume. In contrast, ranked metasearch techniques permit such prioritization.

[0016] The category of ranked metasearches can be further divided into centralized and decentralized ranking. In centralized ranking implementations, each search engine sends candidate documents to the metasearch engine, which then ranks them as a single, new collection. Some variants of centralized ranking transmit each document to the metasearch engine in its entirety. Others make more economical use of network bandwidth by transmitting only the minimal amount of statistical information about each document that is needed to compute the necessary scores. In many cases, a Boolean search initially conducted by each search engine eliminates from consideration any completely irrelevant documents.

[0017] Centralized ranking is either network intensive or extremely imprecise. It is network intensive if each search engine returns all documents, or all documents that satisfy the Boolean pass. It is extremely imprecise if a limit is placed on the number of documents returned by each search engine; without performing any scoring or ranking, a search engine cannot prioritize its result list.

[0018] To help overcome the extreme imprecision and high bandwidth utilization described above, decentralized ranking techniques are preferred. Decentralized ranking can be carried out by using either local statistics or global statistics. In metasearch implementations that employ local statistics, each search engine initially returns the results that score best with respect to the given query and the individual collection. These results are subsequently combined and manipulated for presentation to the user. Within this rubric there are many subsidiary variations.

[0019] a) Some metasearch implementations do not attempt to re-rank results. Instead, they either group results by search server or interleave them.

[0020] b) Others attempt to remove duplicates and re-rank results, applying scoring rules with heuristic measures such as the number of duplications across search servers, rank within each search, and concentration of search terms within the title or summary received from each search server.

[0021] c) Still others perform a final, centralized ranking at the metasearch engine, treating the union of results returned from the search engines as a single collection.

[0022] Regardless of these variations, these implementations suffer from lack of precision when applied across collections with disparate statistics. The scoring function applied within each collection does not in general match the scoring function applied across the metacollection because the local statistics do not match. Thus, in the example described previously, the relatively few documents from the agriculture collection might be ranked higher than the potentially more relevant documents from the computer collection, due to the limited focus of each search engine. Consequently, decentralized ranking with local statistics is most appropriate when the constituent search engines are not under control of the metasearch provider, and each search engine has access to substantially the same content. Those conditions hold true, for example, when combining multiple search engines on the Internet.

[0023] To obtain a correct ranking in a network-efficient manner, decentralized ranking with global statistics is most preferable. In this approach, some portion of the computation is executed by the individual search engines using global statistics (statistics that depend on the entire metacollection). The desirability of such metacollection-level statistics has been recognized in the published literature since 1995. See, for example, C. Viles and J. French, “Dissemination of Collection Wide Information in a Distributed Information Retrieval System,” Technical Report CS-95-02, University of Virginia, Jan. 6, 1995. Particular methods for accomplishing such computations have been refined during the intervening years.

[0024] One class of approaches is to precompute all necessary metacollection-level statistics and store them at each search engine, as taught, for example, in U.S. Pat. No. 6,163,782. This approach eliminates request-response delays perceived by the user that would otherwise be incurred during the computation of metacollection-level statistics. However, it is appropriate only if the same metacollection is used for every metasearch. In general, that condition does not hold. With each query, the user might specify a different set of collections to be included in the metacollection. Even if the user does not make such a specification, many metasearch systems perform a database selection process that determines the collections which are most likely to have appropriate results for a given query. Each query might include conditions, such as access rights, that cause different documents to be included within the metacollection. The set of available documents itself might be dynamic, with documents being continuously added, deleted, and edited. Each of these conditions makes it impractical to use pre-computed metacollection statistics.

[0025] In a variation of such an approach, only certain metacollection-level statistics, such as document frequencies, are precomputed and stored at each search engine. Such approaches are subject to similar limitations.

[0026] Thus, for correctness and practicality it is preferable for at least some of the metacollection-level statistics to be dynamically computed in response to each query. It is the goal of the present invention to provide such correctness and generality. It is a further goal to do so in a manner that minimizes both perceived latency and consumed memory, processing, and network resources. These goals become increasingly important as the scale of the collections, metacollection, and result sets increases.

[0027] In accordance with the invention, these objectives are achieved by means of a multi-phase approach in which local and global statistics are exchanged between the search engines and the metasearch engine. In the first phase, the query is transmitted to the search engines from the metasearch engine, and each search engine computes or retrieves previously-computed local statistics for those terms in its associated document collection. In the second phase, each search engine returns its local statistics. A third phase consists of computing metacollection level statistics at the metasearch engine, based upon the information contained in the local statistics. The metacollection level statistics are disseminated to the search engines. In the final phase, the search engines determine scores for the documents in their respective collections pursuant to the metacollection level statistics, and transmit document references to the metasearch engine. The metasearch engine merges the results from the individual search engines, to produce a single ranked results list for the user.

[0028] By dynamically computing the local and metacollection statistics at the time a query is presented, the present invention maximizes precision for the collections that are being searched, without compromising recall. In addition, the amount of data that is transmitted over the network is kept to a minimum, thereby providing efficient bandwidth utilization and reduced latency. Furthermore, the topology of the search mechanism readily supports a multi-tier hierarchy of search engines, thereby facilitating the scalability of the metasearch system to any number of document collections and search engines.

[0029] Additional features of the invention which contribute to both the efficacy of the search and the efficiency of network communications are described hereinafter with reference to exemplary embodiments depicted in the drawings.

[0030] FIG. 1 is a general block diagram of the basic components of a typical metasearch system;

[0031] FIG. 2 is an illustration of the topology of various metasearch techniques;

[0032] FIG. 3 is a flow diagram of the four fundamental phases of a metasearch in accordance with the present invention;

[0033] FIG. 4 is a expanded flow diagram of the phases of a metasearch in accordance with the present invention, illustrating additional features that can be incorporated;

[0034] FIG. 5 is a flow chart of the distributed sort and chunking features of an embodiment of the present invention;

[0035] FIG. 6 is a block diagram of the manner in which abstract iterators can be employed in the context of the invention; and

[0036] FIG. 7 is a block diagram of a hierarchical arrangement of search engines.

[0037] To facilitate an understanding of the invention, a description will first be provided of one example of a methodology that can be employed to score each document that is retrieved in response to a query, for purposes of ranking the relevance of the documents to the terms in the query. In the description which follows, a document can be a file which contains textual terms. Other examples of “documents” that contain searchable content include web pages, images, videos, audio files, etc. In these latter examples, the elements of a query can be any suitable description for the content of the document. For instance, in the case of a video, a query term might be a scene identifier, and in the case of an audio file it might be a particular sequence of musical notes. In a more complex image-retrieval system, the contents of an image might be characterized in terms of wavelet transformations. Information-retrieval techniques, which are well known in the art, call for the scoring of documents based on statistics that are dependent on the query, each document, each collection, and the metacollection.

[0038] Collection-level statistics depend only on the query and a single collection. Some commonly employed statistics include:

[0039] Collection size (n_i): The number of documents or the number of terms in collection i.

[0040] Local mean document length (mdl_i): The mean document length in collection i.

[0041] Local document frequency (df_ij): The number of documents in collection i that contain term j.

[0042] Local collection frequency (cf_ij): The number of occurrences of the term j in collection i.

[0043] Metacollection-level statistics depend on the query and the entire metacollection. These are:

[0044] Metacollection size (N): The number of documents or the number of terms in the metacollection.

[0045] Mean document length (MDL): The mean document length across the metacollection.

[0046] Document frequency (DF_j): The number of documents in the metacollection that contain term j.

[0047] Inverse document frequency (IDF_j): The scarcity of documents containing term j, often equal to log(N/DF_j).

[0048] Collection frequency (CF_j): The number of occurrences of the term j in the metacollection.

[0049] Inverse collection frequency (ICF_j): The scarcity of term j in the metacollection.

[0050] Document-level parameters depend on an individual document and the query. These are:

[0051] Document length (dl_k): The length of document k.

[0052] Term frequency (tf_jk): The number of times term j appears in document k.

[0053] Other statistics are known as well, and can be employed in addition to, or in place of, those defined above.

[0054] In the so-called vector-space representation, each document is represented as a vector (r_1k, r_2k, . . . ), in which each element (r_jk) is computed for term j and document k by a function. An example of such a function which is based on some of the foregoing statistics could take the following form:

[0055] r_jk=IDF_j*f(tf_jk,dl_k,NP)

[0056] where f( ) is a function whose precise form varies among implementations, and NP is an optional normalization parameter based on document-level statistics.

[0057] As another example, ICF can be used in place of IDF in many cases. It can be seen that the vector representation depends on metacollection-level statistics as well as document-level parameters. Many implementations may also normalize the vector to lie on the unit sphere, i.e., have Euclidean length of one.

[0058] The query is similarly represented as a vector, computed as if the query were a document in the metacollection. The score of a document is quantified in terms of its proximity to the query in vector space. Proximity is commonly measured as the cosine of the angle between two vectors, which is efficiently computed as a dot product between the two vectors.

[0059] While various metasearch implementations may use different techniques for scoring documents, it is widely acknowledged that dependence of the score on metacollection-level statistics (typically IDF_jor ICF_j) produces a practical level of precision in response to a metasearch across multiple collections. The challenges described previously exist in any metasearch wherein document scores depend on metacollection-level statistics.

[0060] In accordance with the general principles of the present invention, a metasearch is carried out in four phases, which comprise two cycles of request and response. These four phases are depicted in the flow diagram of FIG. 3. Referring thereto, the metasearch engine 10 first receives a query containing parameters for a ranked search and disseminates it to each search engine 12, at step 20. In response, in the second phase each search engine computes collection-level, or local, statistics at step 22 and returns those statistics to the metasearch engine. In the third phase, the metasearch engine computes and disseminates metacollection-level, or global, statistics (step 24). Finally, in the fourth phase, each search engine scores each document using the global statistics at step 26, and returns a result to the metasearch engine. The metasearch engine collates results from the search engines and produces a single ranking at step 28.

[0061] Thus, in the present invention, at least some of the global statistics are computed after receiving the user's query, and are based upon local statistics determined in accordance with the query. In other words, the document scores are dynamically computed, rather than approximated. In some implementations, it may be feasible to calculate some of the global statistics and disseminate them on a periodic basis, rather than compute them for each query. For instance, if all of the documents in a metacollection have lengths that vary within a small range, it may be preferable to compute MDL once per day, and transmit that statistic at a suitable time that would not impact network traffic. For other global statistics, however, particularly IDF and ICF, the best results are obtained when they are computed dynamically for each query.

[0062] The data that is returned by the search engine in phase four comprises a document reference. The document reference can take any of a variety of forms. For example, it could be a full copy of the document itself, an identification of the document, a link to the location of the document, e.g. a URL, a link plus a short descriptor or summary of the document, etc. The returned result can also include the relevancy scores for the documents whose references are being supplied to the metasearch engine.

[0063] An exemplary embodiment of this technique is presented hereinafter with reference to one example of the typical scoring methodology previously described. In the first phase, the search parameters transmitted at step 20 comprise a set of terms j₁, j₂, j₃. . . , that appear in the query presented by the user. In the second phase, each search engine returns the collection size n_i, the local mean document length mdl_i, and, for each term j in the query, the document frequency df_ij.

[0064] In the third phase, the metasearch engine sums the n_ivalues over i to compute the total collection size N. It computes the mean document MDL as a weighted mean of the local mean document lengths mdl_i, e.g., MDL=[Σ(n_i*mdl_i)]/N. For each term j in the query, it sums the df_ijvalues over i to compute DF_j. From these quantities it computes the vector representation of the query, r_q. It disseminates the vector query, N, MDL, and every DF_jto each search engine along with the original query. Of particular significance, it is only necessary to disseminate those statistics that are needed for the correct calculation of document scores, namely the statistics for those terms j that appear in the query. Relative to other metasearch techniques that require statistics for every term in the collection, or every term that appears within documents matching the query, the present invention achieves a significant savings in communication overhead.

[0065] In the fourth phase, each search engine produces the following parameters for each document k. It computes or fetches a cached value of dl_k, and for each term j, the term frequency tf_jk. From DF_j, tf_jk, dl_k, and MDL it computes the vector representation of each document, and hence the score of each document against the query and entire metacollection. The statistics that are disseminated enable the same scoring results to be achieved as those which are attainable through a central indexing approach.

[0066] In search engines and metasearch engines, it is often desirable to normalize scores, to remove the effects of false differentiators. In the foregoing example, the mean document length MDL is employed as the normalization parameter NP for the computation of the vector representation. Thus, the scores of the documents are normalized across the collections in accordance with the mean document length MDL for the metacollection. When the scores are not normalized in accordance with document length, the rankings could be biased toward longer documents, if the lengths of the documents vary significantly across the collections. Hence, this normalization can be employed to provide more exact results over the entire metacollection.

[0067] Other parameters that are based on document-level statistics can also be employed in lieu of, or in addition to, document length as a normalization parameter. One example is tf_max, the number of times the most frequent term in the document appears in that document. Another example is the number of unique terms in a document. Other document-level statistics will be apparent as suitable normalization factors as well.

[0068] For the ranking computed in phase four to be mathematically correct, IDF is preferably identical at all search engines. One way to accomplish this objective is to compute IDF from DF at the metasearch engine, to thereby guarantee that it will be identical at all of the search engines. However, to provide greater flexibility and provide the ability to accommodate search engines from different vendors, it may be preferable to compute IDF from DF at the individual search engines, rather than at the metasearch engine. In many installations, where metasearch techniques are employed across search engines of diverse origin, the procedures for computing IDF may vary in some details. For most practical purposes, such minor variations are acceptable within the context of the present invention. Furthermore, by computing IDF at the search engine, the conversion of independent variables into dependent variables occurs at a later stage of the process, so that the original data remains intact and available throughout a greater part of the process.

[0069] For similar reasons, it may be preferable to calculate other values at the individual search engines, rather than at the metasearch engine. For instance, since the vector representation of the query, r_q, is a dependent variable, like IDF, it is a good candidate for such distributed calculation.

[0070] In one embodiment of the invention, the query string that is entered by the user can be parsed at the metasearch engine, and each term in the parsed query is transmitted to the search engines during phase one. In a metasearch, however, practical difficulties can arise when different search engines parse the same query string differently. For example, an English language search engine finds word breaks at spaces, whereas a French language search engine is sensitive to contracted articles. Thus, the query “1′enfant” would be parsed by the English language search engine as one word, whereas the French language search engine would parse it as two words. To accommodate for such differences, in the practice of the invention, query parsing is preferably delayed until phase two, and hence occurs at the individual search engines according to their own parsing conventions. In phase three, each search engine returns collection-level statistics together with the parse results in the form of a character offset and length for each term.

[0071] In a simple implementation, each distinct combination of character offset and length could be treated as separate terms. Thus, in the preceding example, the term “1'enfant” as identified by the English language search engine would be regarded as distinct from the terms “1′” and “enfant” as identified by the French language search engine. In a more preferred embodiment, the metasearch engine identifies specific matches between differently parsed terms that are co-located in the query string, and aggregates statistics accordingly. For example, the metacollection-level statistics for “enfant” would include the collection-level statistics for “enfant” from the French search engine and those for “1′enfant” from the English one. Similarly, the metacollection-level statistics for “1′enfant” would include collection-level statistics for “1′enfant” from the English search engine and those for “enfant” from the French one, assuming that “enfant” is less common than “1′”.

[0072] The fundamental procedure of the present invention comprises four phases consisting of two request-response pairs. Depending upon the particular implementation, either or both of the request-response pairs might itself comprise plural request-response cycles which are carried out to perform the function of one or more phases of the process, as described hereinafter. In addition, other steps can be included with the fundamental phases to enhance the practicality and value of the metasearch process. Some of these enhancements are set forth below, and depicted in the expanded flow chart of FIG. 4.

[0073] As one feature of the invention, phase two may include an additional step 21 of filtering from the collection all documents that do not satisfy a given Boolean condition. The Boolean condition underlying the filter may include arbitrary complex functions that employ non-Boolean operators, as long as the result definitively specifies each document's inclusion or exclusion in the collection to be considered. Examples of such operators include concept matching, usage statistics, database selection, and the like. The Boolean condition can be specified together with the search parameters during phase one, or it may have been established prior to sending the query, in which case it need not be explicitly specified. It may be used to accomplish a variety of practical aims. In the case of a keyword-based query, for example, the Boolean filter can eliminate from consideration any documents with no chance of entering the result set or affecting the collection-level statistics. For example, a Boolean filter can be employed to cull out all documents in a collection that do not contain any of the terms in the query. As another example, the filter might be used to exclude any documents that possess or lack some attribute of exogenous origin, such as having been paid for by the user.

[0074] The remaining phases of the process operate on the subset of the collection that survives the Boolean search. Of particular significance is the fact that the Boolean filtering takes place prior to the computation of the statistics that are used to determine relevancy scores for the documents. The collection which remains under consideration after the filtering could generate significantly different local statistics from those which would be generated from the original collection as a whole. For instance, in a search for “water irrigation” a Boolean filter could be used to exclude all documents that do not contain either of the terms “water” or irrigation”. Within the remaining set, it is likely that most of the documents will contain the word “water”, since irrigation normally deals with water. As a result, the term “water” will likely have a very low IDF, whereas its IDF in the full collection, or metacollection, could be much larger. Hence, the dynamic computation of the local statistics at the time that a query is received provides more reliable results than pre-query computation of a fixed set of statistics for a collection, particularly when Boolean filtering is used to modify the collection.

[0075] In another aspect of the invention, the third phase can include a further step 25 of assigning a key to each document and sorting results within each collection. The sort criterion can be any suitable value-based key, such as the relevancy score itself, or some other attribute such as alphabetically by author name, or temporally by modification date. The metacollection-level sort performed in the fourth phase is then accomplished via a multi-way merge sort that is itself known in the art. Thus, the system performs a distributed sort, avoiding the need to incur the high network costs associated with central ranking.

[0076] In a further extension of this feature of the invention, network traffic generated by the merge sort is minimized by “chunking” the result set, i.e., by dividing it into subsets of fixed size L, and passing those subsets only as needed (step 27). Numerous web-based search engines employ a form of chunking at the user-interface level, by displaying results to the user in groups of ten or twenty at a time. The present invention employs chunking at the level of data transfer from each search engine to the metasearch engine in phase four. For example, in response to a user-interface request for the first ten documents, the metasearch engine can request only ten of the sorted document references from each search engine at a time (step 29).

[0077] This chunking mechanism avoids undue communication between the search engines and the metasearch engine. Referring to FIG. 5, in response to the user's query, the metasearch engine requests the K highest-ranked document references, e.g. the first ten in the sorted order, from each search engine (step 30), and stores the references in a cache memory. The documents from all of the search engines are then merged in sorted order, in accordance with their relevancy rankings (step 32), and the top M merged documents are returned to the requesting entity, e.g. displayed to the user, at step 34. When the user requests the next group of N documents (step 35), the metasearch engine retrieves N documents one by one from its cache (step 36). For each document that is retrieved, the metasearch engine determines if that document is the lowest-ranked document that has been returned by one of the search engines (step 37). If so, the next ranked group of K documents is requested from the corresponding search engine at step 38, and added to the cache. Once all of the necessary documents are stored in the cache, the highest-ranked N document references that have not yet been displayed are presented to the user, at step 39.

[0078] As an illustrative example, if K=M=N=5, when the first five document references from each of three search engines are merged and sorted, the documents from Engine A may have rankings of 1, 3, 4, 6 and 8, those of Engine B may be ranked 2, 7, 10, 13 and 15, while those from Engine C are ranked 5, 9, 11, 12 and 14. In response to the initial query, ranked documents 1-5 are displayed to the user. In response to a request for the next five documents, the metasearch engine determines at step 36 that all of the remaining documents from Engine A, namely the documents that are ranked 6 and 8, are needed to fulfill the request. In this case, the metasearch engine requests the next five documents from Engine A, to account for the fact that some or all of these additional documents may have a score which is higher than that of cached document 9 from Engine C and/or document 10 from Engine B. However, no additional documents are requested from Engines B or C, since they each contributed cached documents that are not needed to fulfill the user's current request (ranked documents 11-15). Thus, network traffic is minimized by utilizing the distributed sort technique in combination with the chunking of documents, so that transmissions to the metasearch engine only occur as needed.

[0079] In the foregoing example, the number of documents that are presented to the user is equal to the chunk size. However, this need not be the case. Rather, it is possible for the number of documents M that is initially presented to the user to be greater or less than the number of documents K that are sent in each chunk. Similarly, the number of documents N that are displayed to the user in response to a request for additional documents could be different from either of the values for M and K. In such a case, however, the foregoing technique can still be utilized to determine when additional chunks of document references are required from each search engine, and thereby minimize network bandwidth utilization.

[0080] To accomplish the merge sort with minimal code complexity and maximum implementation flexibility, in accordance with another variant of the invention the protocol between the metasearch engine and the search engine can employ an abstract iterator together with a value-based sort key. An abstract iterator is well known to software engineers as a uniform interface to a variety of dissimilar data structures such as the array, linked list, and doubly linked list. Such data structures differ in the operations they support. For example, an array supports direct access to the k'th element, whereas the other two do not. An abstract iterator provides the smallest set of common operators, such as “move to first element”, “move to next element”, and “test for last element”. Thus, an abstract iterator permits the implementation of a data structure to be changed without affecting the code that uses the data structure.

[0081] Referring to FIG. 6, abstract iterators are employed in the present invention at three locations, and each may be associated with a cache in memory or on disk. The iterators at the three locations respectively comprise a metasearch iterator 40, remote search iterators 42, and local search iterators 44. The two fundamental operators in each such iterator are “initialize” and “return next K elements”, where K is a whole number. The behavior of each iterator is defined by its response to these two operators. For clarity of exposition, termination conditions are not described, as they are readily understood by the skilled software engineer.

[0082] The metasearch iterator 40 performs the multi-way merge sort. It contains a cache having one memory cell associated with each remote search iterator. Upon initialization, it fills each cell in its cache with the highest ranking document reference from the associated remote search iterator. When requested to return the next element, it returns the highest ranking document reference in its cache and replaces the contents of the vacated cell with the next highest ranking document reference from the associated remote search iterator. When requested to return the next K elements, it returns the next element K times.

[0083] The remote search iterator 42 is designed according to a tradeoff between memory consumption at the metasearch engine, the number of request-response cycles invoked between metasearch engine and search engine, and the total network bandwidth consumed. If it contains no cache, it simply passes all requests directly to its associated local search iterator. If it does contain a cache, the cache may be of fixed or unlimited size. Upon initialization, it fills its cache with the highest-ranking document references from its associated local search iterator. When requested to return the next element, it returns the highest ranking document reference in its cache. If the cache is empty, it first fills its cache with the next highest ranking document references from its associated local search iterator. When requested to return the next K elements, it performs the corresponding logic K times, but using only one request via the network.

[0084] The local search iterator 44 is designed according to a tradeoff between memory consumption and processing burden at the search engine. If it contains a cache, then the cache may be of fixed or unlimited size. Upon initialization it scores and sorts all documents in the collection, subject to any Boolean filtering; then it fills its cache with the highest ranking document references from its associated collection. When requested to return the next K elements, it serves the responses from its cache. If the cache is exhausted, or if the iterator is designed without a cache, then the local search iterator reinitializes and discards as many of the highest ranking document references as it has already returned.

[0085] In a preferred embodiment of the invention, the local search iterator contains a cache of unlimited size. The remote iterator contains a cache of fixed size, which corresponds to the chunk size, i.e., a fixed number of document references fetched upon each request.

[0086] The use of value-based sort keys provides further benefits in the context of statelessness. In this further aspect of the invention, each search engine is stateless but cached. From the foregoing description it can be seen that each search engine responds to multiple requests in the course of servicing a single query. In particular, the four fundamental phases of operation are divided into two cycles of response and request, and the second cycle might itself involve multiple request-response cycles if the local sorting and chunking techniques are employed. The response to each request depends on results from previous requests.

[0087] Statelessness refers to the capability of the search engine to respond correctly to each request even if it has no memory of previous requests. The benefits of statelessness are known in the field of Internet protocols, for example. They include reduced exposure to memory overflows, resource interchangeability for purposes of scaling and load balancing, and greatly reduced complexity in implementing hierarchical topologies. Caching refers to the use of memory to avoid recomputations whenever memory capacity allows. The present invention accomplishes these goals by means of value-based sort keys in combination with collection-level caches. The cache stores the original query, the list of Boolean results computed in phase one, the metacollection-level statistics disseminated in phase two, and the scores and sort order computed in phase three.

[0088] As an illustrative example, the metasearch engine can increment an iterator, resulting in an indication that it requires more results from a particular search engine. Upon detecting this condition, it sends the metacollection-level statistics to that search engine together with the current value-based sort key. If the search engine finds no cache corresponding to the given query, it re-executes the query and sorts the results to refill the cache. It seeks to the correct position in the cache using the value-based sort key and returns the next chunk of results. The term “value-based” implies that the sort key contains only data computed from the query and document, such as its score or a representation of its metadata. Thus, the seek operation works correctly even if documents are added to or removed from the collection.

[0089] The protocol that is employed in the context of the present invention enables a metasearch engine to be treated as a search engine within a larger metasearch, in accordance with another aspect of the invention. More particularly, the metacollection-level statistics N, MDL, and DF_jthat are generated in a smaller metasearch can form intermediate level statistics that are treated as the collection-level statistics n_i, mdl_i, and df_ijin the context of a larger metasearch. In other words, whether a particular search engine is delivering local statistics for a single collection or global statistics for a subsidiary metasearch is inconsequential to the higher-level search engine that is receiving those statistics. By employing this concept in a recursive manner, engines can be arranged in a hierarchy, as illustrated in FIG. 7, so as to permit a query to incorporate the metasearch engine, its neighbors, the neighbors of its neighbors, and so forth.

[0090] Referring to the figure, local search engines 12 perform searches on their respective collections 14, and report local statistics n_i, mdl_iand df_ito associated intermediate engines 50 during phase two. These intermediate engines process the local statistics in the same manner as the metasearch engine to produce intermediate statistics, e.g. semi-global values. These intermediate statistics are then forwarded to a higher level engine, in this case the metasearch engine 10, which processes them to compute the global statistics. During phase three, these global statistics are transmitted to the intermediate engines 40, which in turn forward them to the local search engines 12. The local search engines utilize the global statistics to score the documents, and report the scores to the intermediate engines 40 during phase four. The intermediate engines merge the scores from their associated search engines, and forward them to the metasearch engine 10, which performs the final merge and display to the user.

[0091] This ability to arrange the search engines among multiple tiers is particularly advantageous in the case of geographically distributed collections. When the root node of a hierarchy is separated from a leaf node by multiple network links, the properties of chunked sorting and statelessness become especially valuable in reducing latency, memory overhead, and overall code complexity. More particularly, the ranking of documents and transmission of partial results, utilizing chunking, can be carried out at both the local search engines and the intermediate engines, to thereby minimize network traffic. Each intermediate engine merges the lists of documents that it receives from its associated search engines, and then sends the top-ranked chunks of the merged list to the metasearch engine as necessary.

[0092] From the foregoing, therefore, it can be seen that the present invention provides a metasearch technique in which local and global statistics are dynamically computed in response to a received query, through multiple request-response cycles between the metasearch engine and the individual search engines. Through this technique, precision and recall are both maximized, while at the same time minimizing network traffic, and thereby reducing latency. Furthermore, the manner in which local and global statistics are computed and calculated enables the architecture of the metasearch system to be implemented in any desirable number of hierarchical tiers, thereby providing the ability to scale the system to accommodate any number and configuration of document collections.

[0093] It will be appreciated by those of ordinary skill in the art that the present invention can be implemented in other forms without departing from spirit or essential characteristics thereof. For instance, the invention is not limited to searching over text documents. The techniques described herein apply to any kind of ranked search that uses ranking criteria that depend on metacollection-level statistics.

[0094] The presently disclosed embodiments are therefore considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents thereof are intended to be embraced therein.