Generating query refinements using query components

[0001] This specification relates to digital data processing and, in particular, to generating query refinements using query components.

[0002] Internet search engines provide information about Internet accessible resources (e.g., Web pages, images, text documents, multimedia content) that are responsive to a user's search query by returning a set of search results in response to the query. A search result includes, for example, a Uniform Resource Locator (URL) and a snippet of information for resources responsive to a query. However, the set of search results that are presented to a user are not always what the user is trying to find.

[0003] This specification describes technologies relating to generating query refinements using query components. A search system identifies components in a query. The components are, for example, non-compositional compounds. The system determines component refinements for each component. The system then combines the component refinements, for example, in the order of the components as they were in the original query, or in various other permutations of combinations, thus creating candidate query refinements for the original query. The system filters the candidate query refinements to remove candidates determined to be poor candidates, e.g., candidates that users are not likely to select.

[0004] In general, one aspect of the subject matter described in this specification can be embodied in a method that includes the acts of receiving a query comprising a plurality of terms; identifying first and second components of the query, wherein each component comprises one or more of the terms of the query and the components do not share a term from the query, and wherein the first component appears before the second component in the query; determining, for each of the first and second components, a plurality of different respective component refinements; and combining the component refinements to create a plurality of query refinements for the query, including combining a first component refinement for the first component with a second component refinement for the second component to create a query refinement so that the first component refinement appears before the second component refinement in the query refinement. Other implementations include corresponding systems, apparatus, computer program products, and computer storage media.

[0005] These and other implementations can optionally include one or more of the following features. Identifying first and second components of the query comprises: dividing the query into a plurality of possible combinations of n-grams; for each possible combination of n-grams: determining, for each n-gram in the combination, a probability that the n-gram is a related phrase; and determining a sum of the probabilities for the n-grams in the possible combination; determining that a first possible combination of the possible combinations has the largest sum of the possible combinations; and identifying the n-grams of the first possible combination as components of the query. Identifying first and second components of the query further comprises determining that the sum for the first possible combination exceeds a threshold sum. Determining the probability that the n-gram is a related phrase is based on a function of the n-gram's relative frequency in training data.

[0006] Combining the component refinements into the plurality of query refinements comprises filtering a plurality of candidate query refinements, including: determining for each candidate query refinement an initial score based on user preference data; and removing from the plurality of candidate query refinements any candidate query refinements having an initial score beyond a threshold score. An initial score for the candidate query refinement is based on a number of searches for the candidate query refinement. An initial score for the candidate query refinement is based on a click-through rate for the candidate query refinement.

[0007] Combining the component refinements into a plurality of query refinements comprises filtering a plurality of candidate query refinements, including ranking the candidate query refinements by commonality with highest inverse document frequency components of the query. Combining the component refinements into a plurality of query refinements comprises filtering a plurality of candidate query refinements, including filtering the candidate query refinements based on syntactic similarity with the components of the query.

[0008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Query refinements for an initial query can be identified by using, in some implementations, conventional query refinement techniques on separate components of the query. The query refinements can be filtered so that only high quality refinements are provided to users. The query refinements can help a user quickly find relevant search results by suggesting related queries which may contain results the user is looking for, and by allowing the user to explore and obtain new information that was not present with the user's initial query. The query refinements can also help users explore a topic of interest by showing them other queries related to the topic they are searching for.

[0009] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0027] Like reference numbers and designations in the various drawings indicate like elements.

[0028] FIG. 1 illustrates an example graphical user interface 100 that presents search results 104 in response to a query 102, and also presents possible query refinements 106 for the query 102. The query “new york times square” 102 is submitted to a search engine through the graphical user interface 100, and a set of search results 104 that are responsive to the query 102 are generated by the search engine and presented in the user interface. While FIG. 1 show a visual display of search results, presenting search results can include various forms of presentation including, for example, transmitting search results to a user's computer for presentation to the user, transmitting search results to another device, transmitting sounds corresponding to the search results, providing haptic feedback corresponding to the search results, or transmitting signals comprising haptic feedback corresponding to the search results to a user's computer for presentation to the user. Other methods of presenting search results are possible.

[0029] Sometimes, the user's query will be a starting point for the exploration of a topic, and sometimes the user's query will be an attempt to find specific information. When users are exploring a topic, they are often looking for a general overview of the topic that may not be provided by search results alone. When the user is looking for specific information, the search results will often be what the user is looking for, and a user will click (e.g., select with a mouse or other input device, for example, a keyboard, or a tactile or an oral input device) one or more of the search results 104. However, a user will not always be satisfied with the search results 104 generated in response to a query. Users can be unsatisfied, for example, when the queries they submit are too broad. For example, when a user submits “New York” but is really looking for “restaurants in New York,” the search engine may identify search results that are very relevant to New York, but not relevant to restaurants in New York. Users can also be unsatisfied, for example, when the queries they submit use non-standard or incorrect terminology. For example, a user may submit a query for “New York,” but really be interested in information on “New York City.” Other reasons for user dissatisfaction are also possible.

[0030] To help users that were not presented with the search results they wanted (or to help users interested in exploring a topic of interest), the user interface 100 includes a group of query refinements 106, e.g., related queries that a user may find have responsive search results that are more relevant to the user's interests. A query refinement of a given query is another query that is related to the given query. Query refinements may, but need not be, superstrings of the given query. For example, the query refinements 106 for the query “New York” 102 include “New York Vacation” (a superstring of “New York” because it includes “New York” as well as the additional word “Vacation”) and “Brooklyn Bridge” (not a superstring of “New York” because it does not include the phrase “New York”).

[0031] Some of the query refinements 106 are identified using conventional query refinement techniques, e.g. query refinements techniques based on user preference data or superstrings. FIGS. 2-11 illustrate example systems, software components, and processes for identifying query refinements.

[0032] Some of the query refinements 106 are identified using component-based query refinement techniques. For these techniques, the search engine (or another system) identifies components in the query 102 and, in some implementations, uses conventional query refinement techniques to identify query refinements for the components. Other query refinement techniques are possible. For example, the system can identify “new york” and “times square” as components, or “new york times” and “square.” The search engine combines the component refinements to create candidate refinements for the query. In some implementations, the search engine further filters the candidate refinements based on, e.g., user preference data (e.g., click data gathered by a search engine, as described below with reference to FIGS. 2-4).

[0033] When a user selects one of the query refinements 106, the search engine presents a new set of search results responsive to the query refinement in the user interface 100 and may optionally present a new group of query refinements for the selected query refinement.

[0034] FIG. 2 illustrates an example search system 214 for providing search results relevant to submitted queries as can be implemented in an internet, an intranet, or another client and server environment. The search system 214 is an example of an information retrieval system that can be used to generate the search results shown in FIG. 1, as well as to collect user preference data that the optional query refinement engine 260 can use to identify query refinements.

[0035] A user 202 interacts with the search system 214 through a client device 204. For example, the client device 204 can be a computer (e.g., a personal computer, a mobile phone, etc.) coupled to the search system 214 through a wired or wireless local area network (LAN) or wide area network (WAN), e.g., the Internet. In some implementations, the search system 214 and the client device 204 are one machine. For example, a user can install a desktop search application on the client device 204. The client device 204 will generally include a random access memory (RAM) 206 and a processor 208.

[0036] A user 202 submits a query 210 to a search engine 230 within a search system 214. When the user 202 submits a query 210, the query 210 is transmitted through a network to the search system 214. The search system 214 can be implemented as, for example, computer programs running on one or more computers in one or more locations that are coupled to each other through a network. The search system 214 includes an index database 222 and a search engine 230. The search system 214 responds to the query 210 by generating search results 228, which are transmitted through the network to the client device 204 in a form that can be presented to the user 202 (e.g., as a search results web page to be displayed in a web browser running on the client device 204).

[0037] When the query 210 is received by the search engine 230, the search engine 230 identifies documents that match the query 210. The search engine 230 will generally include an indexing engine 220 that indexes documents (e.g., web pages, images, multimedia content, or news articles on the Internet) found in a corpus (e.g., a collection or repository of content), an index database 222 that stores the index information, and a ranking engine 252 (or other software) to rank the documents that match the query 210. The indexing and ranking of the documents can be performed, for example, using conventional techniques. Other indexing and ranking techniques are possible. The search engine 230 transmits the search results 228 through the network to the client device 204 for presentation to the user 202.

[0038] In some implementations, the search system further includes a refinement engine 260. In some implementations, the refinement engine 260 generates query refinements from user preference data gathered by the search system 214 and stores them, for example, in a refinement database 262. An example of such a query refinement engine 260 is described in more detail below with reference to FIG. 5. In some other implementations, the refinement engine 260 generates query refinements using component-based techniques. An example of such a query refinement engine 260 is described in more detail below with reference to FIG. 12. The search engine can optionally retrieve refinements for the query 210 from the refinement database 262 and transmit them to the user 202 along with the search results 228.

[0039] FIG. 3A illustrates example components of an information retrieval system. These components include an indexing engine 3010, a scoring engine 3020, a ranking engine 3030, and a rank modifier engine 3070. The indexing engine 3010 functions as described above for the indexing engine 220. The scoring engine 3020 generates scores for document results based on many different features, including content-based features that link a query to document results, and query-independent features that generally indicate the quality of documents results. The content-based features include, for example, aspects of document format, such as query matches to title or anchor text in an HTML (Hyper Text Markup Language) page. The query-independent features include, for example, aspects of document cross-referencing of the document or the domain. Moreover, the particular functions used by the scoring engine 3020 can be tuned, to adjust the various feature contributions to the final information retrieval (IR) score, using automatic or semi-automatic processes.

[0040] The ranking engine 3030 produces a ranking of document results 3040 for display to a user based on IR scores received from the scoring engine 3020 and one or more signals from the rank modifier engine 3070. A tracking component 3050 is used to record information regarding user behavior such as individual user selections of the results presented in the ranking 3040. In some implementations, the tracking component 3050 is embedded JAVASCRIPT code included in a web page ranking 3040 that identifies user selections of individual document results and also identifies when the user returns to the results page, thus indicating the amount of time the user spent viewing the selected document result. In other implementations, the tracking component 3050 is a proxy system through which user selections of the document results are routed. The tracking component can also include pre-installed software at the client (e.g., a toolbar plug-in to the client's operating system). Other implementations are also possible, for example, an implementation that uses a feature of a web browser that allows a tag/directive to be included in a page, which requests the browser to connect back to the server with message(s) regarding link(s) clicked by the user.

[0041] The recorded information is stored in result selection logs 3060. In various implementations, the recorded information includes log entries that indicate user interaction with each result document presented for each query submitted. For each user selection of a result document presented for a query, the log entries indicate the query (Q), the document (D), the user's dwell time (T) on the document, the language (L) employed by the user, and the country (C) where the user is likely located (e.g., based on the server used to access the IR system). The log entries also records negative information, such as the fact that a document result was presented to a user, but was not selected. Other information such as position(s) of click(s) (i.e., user selection(s)) in the user interface, information about the session (e.g., existence and type of previous clicks, and post-click session activity), IR scores of clicked results, IR scores of all results shown before click, the titles and snippets shown to the user before the click, the user's cookie, cookie age, Internet Protocol address, user agent of the browser, and so on, can also be recorded.

[0042] In various implementations, the time (T) between the initial click-through to the document result and the user's returning to the main page and clicking on another document result is also recorded. An assessment is made about the time (T) regarding whether this time indicates a longer view of the document result or a shorter view of the document result, since longer views are generally indicative of quality or relevance for the clicked through result. This assessment about the time (T) can further be made in conjunction with various weighting techniques.

[0043] The components shown in FIG. 3A can be combined in various manners and implemented in various system configurations. For example, the scoring engine 3020 and the ranking engine 3030 can be merged into a single ranking engine, such as the ranking engine 252 of FIG. 2. The rank modifier engine 3070 and the ranking engine 3030 can also be merged, and in general, a ranking engine includes any software component that generates a ranking of document results after a query. Moreover, a ranking engine can be included in a client system in addition to (or rather than) in a server system.

[0044] FIG. 3B shows another example information retrieval system. In this system, a server system 3150 includes an indexing engine 3160 and a scoring/ranking engine 3170. A client system 3100 includes a user interface 3110 for presenting a ranking, a tracking component 3120, result selection logs 3121 and a ranking/rank modifier engine 3140. For example, the client system 3100 can include a company's enterprise network and personal computers, in which a browser plug-in incorporates the ranking/rank modifier engine 3140. When an employee in the company initiates a search on the server system 3150, the scoring/ranking engine 3170 can return the search results along with either an initial ranking or the actual IR scores for the results. The browser plug-in then re-ranks the results locally based on tracked page selections for the company-specific user base.

[0045] FIG. 4 is a diagram illustrating the building of an example model database 402 for use with an information retrieval system. The model database 402 is one or more databases that aggregate information indicative of user behavior on at least a per-query and per-document basis, for instance. In various implementations, information in the model database 402 is used as input to a process that identifies query refinements. In some implementations, the information is also used as input to a process that ranks documents or enhances the ranking. For example, the model information can be used as input to interactive or non-interactive Internet search engines, image search engines, video search engines, book search engines, consumer product search engines, other suitable processes capable of identifying query refinements or ranking documents, and combinations of these. In various implementations, a document is referenced by a URL in the model database 402; however, other types of document references are possible. For instance, a document can be a webpage on the World Wide Web referenced by a URL, or can be referenced by a globally unique user id (GUID) determined by the information retrieval system.

[0046] As shown in FIG. 4, a user submits a query 406a “used car,” for example, to an information retrieval system, such as a search engine, through a graphical user interface 409, as presented on a personal computer, a mobile telephone, or other device. The user interface can be, for example, the user interface 100 shown in FIG. 1, or another user interface. A query includes one or more terms. For example, a query can include the terms “bus,” “river” and “hotel.” In addition to dictionary words, terms can include special characters, numbers, mathematical expressions, Boolean expressions, slang terms, names, titles, images, sounds, videos, other suitable query terms, and combinations of these. Moreover, query terms can be in more than one language.

[0047] In response to the user selecting the search button 422, for example, a search engine returns a ranking or result list 408 which is an ordered list of references to documents that are responsive to the query 406a. The result list 408 includes a set of document references URL A, URL B, URL C (401a), and URL D. The result list 408 can contain the text of the URL itself, a short description of the information found within each document, a snapshot of the portion of the document which contains the query, other suitable information, or a combination of these. If a user selects (e.g., clicks) URL C 410a from the result list 408, for example, the selected URL can cause the user interface 409 (e.g., a web browser) to present the associated document 412. Once the user has finished viewing the document, the user can navigate back to the result list 408.

[0048] In various implementations, the model database 402 is built as users interact with search engines. Each presentation of a document reference in the result list in response to a query is recorded as an “impression” of the referenced document in connection with the query. Other standards for recording impressions can also be used. For example, an impression can be recorded when a user selects the document, or a document below the document in the result list. An impression can also be recorded when a user selects the document directly above the document in the result list. In some implementations, a weighted count of impressions is calculated for each document presented in response to the query. For example, if a user selects the document or a document below the document in the result list, 1 is added to the weighted count. If a user selects a document above the document, a value of less than 1 is added to the weighted count. The value can be a constant or can be determined, for example, by dividing 1 by the number of documents between the document and the selected document in the search result list.

[0049] Each selection of the presented document reference is recorded as a “click” for the document in connection with the query. A timer can be used to track how long a user views or “dwells” on the document. For example, a longer time spent dwelling on a document (i.e., a long click) can indicate that a user found the document to be relevant for the query. A brief period viewing a document (i.e., a short click) can be interpreted as a lack of document relevance. In some implementations, the click data 414 is a count of clicks of all types (e.g., long, medium, short) for a particular query 406 and document 410 combination. In some implementations, the click data 414 includes a count of each click type (e.g., long, medium, short) for a particular query 406 and document 410 combination. Aggregated click data from queries for a given document can be used to generate a quality of result statistic for the document as a search result for the query. In general, the quality of result statistic estimates how responsive users found a given document to be as a search result for a given query. The quality of result statistic can be used to rank or enhance a ranking of that document. For example, a quality of result statistic can be an average of the weighted long clicks for a given document and query pair, a sum of the weighted clicks for a given document and query pair, or a click fraction for a given document and query pair. Various weights are also possible, for example, the clicks can be weighted by the length of the clicks, by how quickly a user clicked on a result, and whether a user clicked on a result first, second, third, etc. Various types of click fractions are possible. For example, in some implementations, the click fraction is calculated by dividing the number of clicks (or the number of weighted clicks) by the number of impressions. In other implementations, the click fraction is calculated by dividing the sum of the weighted clicks by the total number of clicks. In still other implementations, the click fraction is the sum of weighted clicks for the document presented in response to the query divided by sum of weighted clicks for all documents presented in response to the query. Other click fractions are also possible.

[0050] A search engine or other process creates a record 416 in the model database 402 for documents that are selected by users in response to a query. Each record 416 within the model 402 (herein referred to as a tuple <document, query, quality of result statistic>) is at least a combination of a query 406 submitted by users, a document reference 410 selected by users in response to that query, and an aggregation of click data 414 for all users that select the document reference 410 in response to the query 406. The aggregated click data is viewed as an indication of document relevance. In various implementations, model data is specific to a geographic location (e.g., a city, metropolitan region, state, country, or continent), specific to a language preference of users (e.g., as specified by the users in a profile or as determined from user search behavior, for example, the language of the queries issued by the user), or specific to a locale (e.g., specific to a geographic location or a set of geographic locations, and optionally a language preference). Other extensions of model data are possible.

[0051] FIG. 5 illustrates an example refinement engine 560. In general, the search system analyzes user preference data 504, for example, data stored in the model database 402 of FIG. 4, and generates a group of one or more query refinements for queries in the user preference data 504. In some implementations, the refinement engine 560 is included in the search system 214 illustrated in FIG. 2. In alternative implementations, the refinement engine is separate from the search system, and merely provides refinements to the search system.

[0052] The refinement engine 560 includes a query-document map generator 506, a candidate refinement selector 508 and a candidate refinement filter 510. The boxes shown in FIG. 5 logically represent executing software components or modules. These components can be combined or subdivided in ways other than those shown in FIG. 5 and can be distributed on two or more computing devices.

[0053] The query-document map generator 506 receives the user preference data 504 and generates a document-to-query-to-document map 512 from the user preference data. In general, the document-to-query-to-document map models relationships between the documents and the queries (and the queries and the documents) that are derived from the user preference data 504. The document-to-query-to-document map is described in more detail below, with reference to FIG. 6.

[0054] The candidate refinement selector 508 receives the map 512 and generates a set of candidate refinements 514 for queries in the map 512. The candidate refinement selector 508 is described in more detail below with reference to FIG. 7.

[0055] The candidate refinement filter 510 receives the map 512 and the candidate refinements 514 and filters the candidate refinements 514 based on data stored in the map 512, resulting in a final set of query refinements 516 including one or more query refinements for some of the queries in the map. The candidate refinement filter 510 is described in more detail below with reference to FIG. 10.

[0056] FIG. 6 illustrates an example map 602 generated from user preference data 504. The user preference data 504 includes tuples associating a query, a document, and a quality of result score for the document (e.g., tuple (Query A, Document A, 0.1) 606). The query-document map generator 506 processes the tuples 504 to generate a map such as the map 602. The map identifies which queries are related to which documents, and which documents are related to which queries. In general, a document is related to a query (and a query is related to a document) when there is a tuple associating the query and the document in the user preference data 504. However, in some implementations, the map generator 506, will require an additional threshold requirement to be met before a query is related to a document (or a document is related to a query). While FIG. 6 illustrates the map as a graph where nodes correspond to documents and queries and edges correspond to relationships between documents and queries, other types of maps are possible. For example, in some implementations, the map stored as a database. Also, while the map is a convenient way of storing the relationships, the relationships identified in the map can alternatively be generated as needed from the user preference data 504 without creating the map.

[0057] The map 602 illustrated in FIG. 6 has two overlapping regions. The first region 608 corresponds to relationships between documents and queries that are identified from the user preference data 504. For example, entry 606 in the user preference data is a tuple associating query A with document A. Therefore, the first region 608 includes an arrow (indicating a relationship) from a node 610 representing document A to a node 612 representing query A. In some implementations, the query-document map generator 506 relates each document in the first region to all queries for which there is a tuple in the search history data corresponding to the document and the query. In other implementations, the query-to-document map generator 506 further requires that a query satisfy a given threshold before a document is associated with the query. For example, in some implementations, the query-to-document map generator 506 further requires that a query have a quality of result statistics for a given document and a given query that satisfies a threshold (e.g., a click fraction that exceeds 0.05, or a weighted clicks score that exceeds 50), before the document is related to the query. In other implementations, the query-to-document map generator 506 further requires that a query have a sum of quality of result statistic across all documents in the user preference data that satisfies a threshold in order for a document in the first region 608 to be related to the query. The thresholds can be determined empirically, for example, to balance between number of candidate query refinements and quality of query candidate refinements. Other requirements are also possible.

[0058] The second region 614 in the map corresponds to relationships between queries and documents that are identified from the user preference data. For example, the second region 614 includes an arrow (indicating a relationship) from the node 612 representing query A to a node 616 representing document A, because there is a tuple 606 in the user preference data 504 that associates query A and document A. In some implementations, the query-to-document map generator 506 relates each query in the second region to all documents for which there is a tuple in the search history data corresponding to the query and the document. In other implementations, the query-to-document map generator 506 further requires that a document satisfy a given threshold before a query is related to the document. For example, in some implementations, the query-document map generator 506 further requires that a document have a quality of result statistic for a given query (as opposed to all queries) that satisfies a threshold (e.g., a click fractions that exceeds 0.05, a weighted clicks score that exceeds 50) before the query is related to the document. In other implementations, the query-document map generator 506 further requires that a document have a sum of quality of result statistics across all queries in the user preference data that satisfies a threshold (e.g., a sum of click fractions that exceeds 0.05, or a sum of weighted clicks that exceeds 50). The threshold can be determined empirically, for example, to balance between number of candidate query refinements and quality of query candidate refinements, and may, but need not be, the same as the threshold for the first region 608. Other requirements are also possible.

[0059] In some implementations, the query-to-document map generator 506 further filters queries, for example, to remove queries with text included in a list of phrases (e.g., words and symbols) that should be filtered out, for example text used in URLs or advanced operators that have special meaning to the search engine from the map. For example, the query-to-document map generator 506 can remove one or more of “www,” “.com,” “.net,” and “:” (where a colon has a special meaning that causes a search engine to search only within the URL that follows the colon). Other phrases can also be filtered, for example, pornographic, offensive, and hate speech phrases.

[0060] FIG. 7 illustrates an example candidate refinement selector 508. The candidate refinement selector 508 includes a query pair identifier 702, a quality filter 704, and a diversity filter 706. The boxes shown in FIG. 7 logically represent executing software components or modules. These components can be combined or subdivided in ways other than those shown in FIG. 7 and can be distributed on two or more computing devices.

[0061] The query pair identifier 702 receives the map 512 and generates a group of query pairs 708 from the map. Each query pair has a first query and a second query. The candidate refinement selector 508 then processes each of the identified query pairs to determine if the second query is a candidate query refinement for the first query.

[0062] The query pair identifier 702 identifies the query pairs as follows. First, the query pair identifier 702 selects a group of queries that a given document is related to. For example, in the first region 608 of the map 602 illustrated in FIG. 6, document A is related to query A, query B, and query C. Therefore, the query pair identifier 702 would select queries A, B, and C as related to document A. The query pair identifier 702 then generates the query pairs from queries to which a document is related. For example, the query pair identifier 702 can generate query pairs: (query A, query B), (query A, query C), (query B, query A), (query B, query C), (query C, query A), and (query C, query B) from queries A, B, and C. In some implementations, the query pair identifier 702 generates query pairs for all possible combinations of the queries to which the document is related. In alternative implementations, the query pair identifier 702 generates query pairs from a subset of the possible combinations, for example, by considering combinations where one or both queries have a quality of result statistic for the document that exceeds a threshold. Other implementations are also possible. For example, in some implementations, the query pair identifier 702 only generates query pairs from queries having at least one word in common. In other implementations, the query pair identifier 702 only generates query pairs from queries where the second query is a superstring of the first query. In some implementations, the query pair identifier 702 does not use the map 512 to identify the group of queries to which a document is related, but instead uses the user preference data itself to identify relationships between documents and queries.

[0063] The quality filter 704 receives the query pairs 708 and the map 512, calculates a quality score for each query pair, and then filters the query pairs based on their quality scores. In some implementations, the quality of result statistics are included in the map 512, and in some implementations, they are provided to the quality filter 704 instead of, or in addition to, the map 512. Each quality score is determined from user preference data (e.g., quality of result statistics) for documents responsive to both the first and the second query in the query pair. In general, the quality filter 704 calculates the quality score from quality of result statistics for documents responsive to the first query, as search results for the second query. The quality score generally reflects whether the first query and the second query share documents frequently selected by users.

[0064] In some implementations, the quality filter 704 performs the following operations. First, the quality filter 704 identifies documents responsive to the first query in the query pair. In some implementations, the quality filter 704 identifies documents responsive to the first query in the query pair from relationships in the document map (e.g., from the relationships in the second region 614 of the map 602 shown in FIG. 6). For example, in the second region 614 of the map 602, the nodes 616 and 618 for documents A and B are related to query A. Therefore, these documents are responsive to query A. In other implementations, the quality filter 704 identifies documents that are related to the first query and that have a quality of result statistic satisfying a threshold (e.g., 0.1). The threshold can be determined, for example, empirically to balance between having the threshold set too low (where insignificant documents that may not really be relevant to the query are included) and having the threshold set too high (where relevant documents will be excluded). Other methods for determining documents responsive to the first query are also possible. For example, in some implementations, the quality filter 704 identifies the documents directly from user preference data (e.g., by selecting the documents with a tuple corresponding to the query, or with a quality of result statistic for the query that satisfies a threshold). As another example, in other implementations, the quality filter 704 identifies a top number of documents identified as responsive to the first query by a search system, such as the search system 214 described above with reference to FIG. 2.

[0065] Second, the quality filter 704 determines quality of result statistics for those documents as search results for the second document in the query pair. In some implementations, the quality filter 704 determines quality of result statistics for the identified documents as search results for the second document in the query pair from the map 512, for example, the map 602 illustrated in FIG. 6. For example, the quality filter 704 can determine which of the identified documents are related to the second query from the relationships shown in the second region 614 of the map, and then retrieve the quality of result statistic for those documents as search results for the second query, either from the map itself, or from the user preference data. Other implementations are also possible.

[0066] Third, the quality filter 704 combines the quality of result statistics to generate the quality score (e.g., by summing the quality of result statistics). In some implementations, the quality filter 704 combines the quality of result statistics to generate the quality score by summing the quality of result statistics. However, other combinations are possible. For example, in some implementations, the quality filter 704 averages the quality of result scores.

[0067] Finally, the quality filter determines whether the query pair should be included in a group of filtered query pairs 710 provided to the diversity filter 706, or excluded from the group of filtered query pairs 710 (e.g., based on the quality score). In some implementations, the quality filter determines whether the query pair should be included in a group of filtered query pairs 710 provided to the diversity filter 706, or excluded from the group of filtered query pairs 710 by determining whether the quality score for the query pair satisfies a threshold (e.g., exceeds a threshold, when the threshold is a minimum value for the quality score, or is below a threshold, when the threshold is a maximum value for the quality score). The threshold can be determined, for example, empirically, to balance the tradeoff between the number of candidate query refinements and the quality of the candidate query refinements. In some implementations, the threshold is lower (e.g., 0) when the second query is a superstring of the first query. In some implementations, the quality filter automatically includes a query pair in the group of filtered query pairs 710 when the second query is a superstring of the first query.

[0068] The diversity filter 706 receives the filtered query pairs 710 and the map 512 and determines whether the search results responsive to the second query in the pair are diverse enough from the search results responsive to the first query in the pair for there to be value in using the second query as a query refinement for the first query. If there is little diversity between the search results responsive to the two queries, then the second query is not a good query refinement because it will not help users find search results that they could not have found with the first query.

[0069] The diversity filter 706 calculates a diversity score for the query pair from user preference data (e.g., quality of results statistics, or other data) for documents that are responsive to the second, but not the first, query. The diversity score thus measures how good the different search results that would be seen in response to the second query are.

[0070] The diversity filter 706 generates the group of documents that are responsive to the second, but not the first, query by identifying a top group of documents responsive to the first query and a top group of documents responsive to the second query, and then generating a group of documents that contains documents from the top group of documents responsive to the second query that are not in the top group of documents responsive to the first query. The top group of documents for each query includes, for example, a fixed number of the top documents (when the documents are ordered, for example, based on quality of result statistics for the document and the query), or all documents whose quality of result score for the query satisfies a threshold. The thresholds can be determined empirically, for example, to balance between having the threshold set too low (where insignificant documents that may not really be relevant to the query are included) and having the threshold set too high (where relevant documents will be excluded). The threshold or fixed number of documents may, but need not, be the same for the first and the second query. Alternatively, the top group of documents for one or both queries can be documents whose quality of result statistic for the query satisfies a threshold (e.g., 0.025 or 0.05). Other methods of generating the group of documents that are responsive to the second, but not the first, query are also possible. For example, in some implementations, the diversity filter 706 takes the difference of the set of all documents responsive to the second query and the set of all documents responsive to the first query.

[0071] Once the group of documents responsive to the second, but not the first, query are identified, the diversity filter 706 calculates the diversity score from user preference data for the documents as search results for the second query, for example, by summing or averaging the diversity scores. The diversity filter 706 then filters the query pairs based on the diversity scores, so that query pairs whose diversity scores satisfy a diversity threshold are output as the set of candidate query refinements 514. The set of candidate query refinements 514 associates one or more candidate query refinements with queries from the map. Each query is a first query in a query pair, and each candidate refinement is a second query in a query pair. The diversity threshold can be determined empirically, for example, to balance the tradeoff between the number of candidate query refinements and the quality of the candidate query refinements. For example, when possible diversity thresholds range from 0 to 1, a middle value, such as 0.4 or 0.5 can be chosen. In some implementations, for example, when the quality score and the diversity score are both sums of the quality of result statistics, the quality threshold and the diversity threshold are chosen with reference to the other (e.g., when the quality threshold is higher, the diversity threshold is lower, and vice versa).

[0072] In some implementations, candidate refinement selector 508 generates candidate refinements for queries that are specific to a given locale (e.g., specific to a geographic location or a set of geographic locations, and optionally a language preference). For example, the user preference data 504 or the map 602 can only include data specific to a given locale, or the candidate refinement selector 508 can filter the data it uses to generate the query refinements.

[0073] FIG. 8A illustrates an example of identifying documents responsive to both the first and the second query in a query pair. FIG. 8A includes a group of documents 8002 responsive to query A (along with the quality of result statistics for the document as a search result for query A) and a group of documents 8004 responsive to query B (along with the quality of result statistics for the document as a search result for query B). The two groups of documents have three documents in common: document A 8006, document B 8008, and document F 8010. Although the two queries share three documents, in some implementations, the quality filter 704 will use a subset of the shared documents (e.g., document A 8006 and document B 8008) to calculate a quality score for a query pair of (query A, query B), because the quality of result statistics for the other shared documents as a search result for query A (e.g., document F 8010 having a quality of result statistic of 0.01 for query A) do not satisfy the threshold used by the quality filter.

[0074] FIG. 8B illustrates an example of identifying documents responsive to the second, but not the first, query in a query pair. FIG. 8B includes the same group of documents 8002 responsive to query A, as well as the same group of documents 8004 responsive to query B. The documents 8004 responsive to query B include three documents that are not in the documents 8002 responsive to query A: document H 8104, document J 8106, and document K 8108. Although the documents 8004 responsive to query B include three documents not in the group of documents 8002 responsive to query A, in some implementations, the diversity filter 706 will use a subset of the unique documents (e.g., document H 8104 and document J 8106) to calculate a diversity score for the query pair (query A, query B), because the quality of result statistics for the other unique documents as a search result for query B (e.g., document K 8108, having a quality of result statistic of 0.01 for query B) do not satisfy the threshold used by the diversity filter.

[0075] FIG. 9 illustrates an example method 900 for selecting query refinements from query pairs. For convenience, the example method 900 will be described in reference to a system that performs the method 900. The system can be, for example, the candidate refinement selector 508.

[0076] The system obtains a group of query pairs (step 902). Each query pair includes a first query and a second query. The system obtains the query pairs, for example, as described above with reference to FIG. 7.

[0077] The system determines a quality score for each query pair (step 904), for example, as described above with reference to FIG. 7.

[0078] The system then determines a diversity score for each query pair having a quality score above a quality threshold (step 906), for example, as described above with reference to FIG. 7.

[0079] In some implementations, the system determines a diversity score for each query pair having a quality score above the quality threshold and a minimum number of documents included in the top documents for both the first and the second query. The minimum number can be determined, for example, empirically.

[0080] In some implementations, the system uses a lower quality threshold for query pairs where the second query is a superstring of the first query than for query pairs where the second query is not a superstring of the first query. In some implementations, the threshold is 0. When the threshold is 0, the system can optionally skip calculating the quality score and treat the query pair as if the quality score satisfies the threshold.

[0081] The system associates, for each query pair having a quality score satisfying the quality threshold an a diversity score satisfying a diversity threshold, the second query of the query pair with the first query of the query pair as a candidate refinement for the first query (step 1008), for example, as described above with reference to FIG. 7.

[0082] In some implementations, the system performs additional filtering before associating the second query with the first query as a candidate refinement (or after associating the second query with the first query as a candidate refinement, to reduce the number of candidate refinements associated with the first query). For example, in some implementations, the system does not store candidate refinements for a group of the most popular queries, because these queries are mostly navigational in nature (i.e., a user is searching for the name of a popular website to go to that website, rather than seeking information about the website). The system can identify whether a given query is one of the most popular queries, for example, from user preference data, from information received from a search engine, or from a blacklist that it maintains.

[0083] As another example, in some implementations, the system filters out candidate refinements that are a substring of the original query for which they are a refinement.

[0084] As another example, in some implementations, the system filters out any candidate refinements that include phrases (e.g., one or more words or symbols) on a list of phrases to filter. For example, the system can filter queries that are pornographic (e.g., contain pornographic text or will lead to pornographic results), that contain offensive or racist or hate speech, or that contain text associated with URLs or advanced search engine operators (e.g., “www,” “.com,” “.net,” or “:”). For example, in some implementations, the system maintains a list of known pornographic queries or phrases (or offensive or racist or hate speech queries or phrases, or known URL text and advanced search engine operations), compares each candidate refinement to the list, and removes candidate refinements that include text on the list. In some implementations, the list is pre-defined. In alternative implementations, the list is generated using a machine learning engine that processes lists of known phrases to identify other phrases that should be filtered out.

[0085] As another example, in some implementations, the system filters out candidate refinements that include a geographic location when the query that they are a refinement for does not include a geographic location. For example, if the query is “department store” and a candidate refinement is “department store near San Francisco,” the system would filter out the candidate refinement, because “San Francisco” is a geographic location, and the query “department store” does not include a geographic location. This filtering is useful when a concept is popular in many locations (for example, when the original query is for a restaurant chain). In these situations, it is unlikely that the particular geographic location included in the query refinement will match the user's geographic location, and the user is likely to be frustrated if they are presented with completely irrelevant refinements. In some further implementations, the system does not filter out the candidate search result, but instead, flags it. At a later time when a user submits the query, the user's geographic location is estimated (for example, using an IP address of the computer from where the query is submitted, and the search system determines whether that location is close enough in the query refinement (e.g., within 100 miles). If the locations are close enough, the refinement is presented to the user. If the locations are not close enough, the refinement is not presented to the user.

[0086] As another example, in some implementations, the system normalizes the candidate refinements for a given query (for example, by applying a stemming algorithm to reduce the terms of the refinements to their base form or by removing stop words, e.g., known terms that do not contribute to the topicality of the query such as “a” or “the,” or “of”). The system then filters the refinements to remove syntactic duplicates (e.g., so that only one refinement of the refinements that normalize to the same query is associated with each query. For example, when the system determines that multiple refinements normalize to the same query, the system can associate the refinement from the query pair having the best quality score with the query.

[0087] As yet another example, in some implementations, the system filters out candidate refinements that have a measure of usefulness that does not satisfy a threshold. For example, the system can filter out refinements whose click through rate does not satisfy a threshold, or can filter out click through rates whose overall number of weighted clicks across all documents presented in response to a query for the refinement divided by the overall number of weighted impressions across all documents presented in response to a query for the refinement satisfies a threshold. The threshold can be determined empirically to balance between quality of candidate refinements and number of candidate refinements. For example, in some implementations, the threshold is 0.34.

[0088] In some implementations, the system also checks the spelling of the candidate refinements, for example, by processing the refinements with a batch spell checker, and corrects the spelling in any misspelled refinements. In these implementations, the system can use either the user preference data for the correctly spelled refinement, or the misspelled refinement.

[0089] FIG. 10 illustrates a candidate refinement filter 510 that is part of the refinement engine 560 illustrated in FIG. 5.

[0090] The candidate refinement filter 510 includes a refinement ranking engine 1002 and a refinement filtering engine 1004. The boxes shown in FIG. 10 logically represent executing software components or modules. These components can be combined or subdivided in ways other than those shown in FIG. 10 and can be distributed on two or more computing devices.

[0091] The refinement ranking engine 1002 receives the candidate query refinements 510 generated by the candidate refinement selector 508, as well as the user preference data 504, and then ranks the refinements for each query using the user preference data, resulting in a ranked group of candidate refinements for each query 1006. The refinement ranking engine 1002 can use various metrics to rank the results, depending, for example, on whether it is desired to have query suggestions that are useful queries in general, or query suggestions that are closely related to the query they are suggestions for. In some implementations, the refinement ranking engine 1002 ranks the candidate query refinements for each query according to the quality scores associated with the query pairs for each query and the candidate refinements for the query (e.g., in order from highest to lowest quality score). In other implementations, the refinement engine 1002 ranks the candidate query refinements for each query according to the quality of result statistics (e.g., click fractions or weighted clicks) for the candidate refinements (e.g., in order from highest to lowest quality of result statistic). In still other implementations, the refinement ranking engine 1002 ranks the candidate refinements for each query according to the click-through-rate for the refinement, that is, the total number of clicks on documents presented in response to the query, divided by the total number of impressions for documents presented in response to the query. In still other implementations, the refinement ranking engine 1002 ranks the candidate refinements according to a weighted click-through-rate for the refinement, for example, where the clicks on each document are weighted by how long the documents were viewed. Other ranking metrics are also possible. For example, in some implementations, the candidate refinements are ranked by a measure of their popularity as queries (e.g., how often do users issue the candidate refinements as queries). In other implementations, compound metrics are used. For example, in some implementations, all query refinements that are superstrings of the query are ranked higher than query refinements that are not superstrings of the query. Within the superstring and not-superstring groups, the refinements are ranked according to one of the other metrics. As another example, in some implementations, the candidate refinements are ranked based on the number of words they have in common with the query refinement. Ties are broken using one of the metrics described above.

[0092] The refinement filtering engine 1004 receives the ranked candidate refinements for each query 1006 and processes them in order, to select the best group of query refinements for each query. These query refinements are then associated with the query as confirmed query refinements for the query 516. While only one confirmed refinement for each query is shown in the query refinements 516, multiple refinements can be associated with each query.

[0093] The refinement filtering engine 1004 selects the best group of query refinements for a given query as follows. First, the refinement filtering engine 1004 initializes a group of seen documents. In some implementations, the refinement filtering engine 1004 initializes the group of seen documents by selecting a top number of documents responsive to the query (e.g., a top number of the documents responsive to the query, ranked for example, by a quality of result statistic, or all documents responsive to the query having a quality of result statistic satisfying a threshold). In alternative implementations, the refinement filtering engine 1004 selects the top-ranked candidate refinement for the query, identifies the refinement as a confirmed query refinement, and initializes a set of seen documents to be a top number of documents responsive to the confirmed query refinement (e.g., a top number of the documents responsive to the confirmed query refinement, ranked for example, by a quality of result statistic, or all documents responsive to the query having a quality of result statistic satisfying a threshold).

[0094] The refinement filtering engine 1004 then processes the candidate refinements (or the rest of the candidate refinements, if the first refinement has already been selected) in order, until a desired number of query refinements have been confirmed, or all candidate refinements have been considered. For each additional query refinement, the refinement filtering engine 1004 calculates an intra-suggestion diversity score between the seen documents (or a subset, such as a top number (e.g., 5) of the seen documents) and documents (or a subset of the documents, such as a top number (e.g., 5)) responsive to the additional query refinement, e.g., much as the diversity filter 706 calculates the diversity score for documents responsive to a first and second query in a query pair. If the intra-suggestion diversity score satisfies a threshold, the refinement filtering engine 1004 associates the additional query refinement with the query as another confirmed query refinement and adds a top number of documents responsive to the additional query refinement to the set of seen documents. If the intra-suggestion diversity score does not satisfy the threshold, the refinement filtering engine 1004 moves on to the next candidate refinement. The threshold can be determined, for example, empirically.

[0095] FIG. 11 illustrates an example method for selecting a group of confirmed query refinements for a query from a group of candidate refinements for the query. For convenience, the example method 1100 will be described in reference to a system that performs the method 1100. The system can be, for example, the candidate refinement filter 510. In FIG. 11, i is a counter variable that tracks which of the candidate refinements is being processed, confirmed_refs is the set of confirmed refinements that have been selected for the query, seen_docs is the set of top documents responsive to each of the confirmed refinements, additional_ref is the candidate reference that is currently being processed, top_docs is the top documents responsive to a candidate refinement, desired_refs is the desired number of refinements, and query is the query for which the candidate refinements are being identified.

[0096] The system begins by setting a counter variable i to 0, setting the group of confirmed refinements to the empty set, and initializing the seen documents to the top documents for the query (step 1102), for example, as described above with reference to FIG. 10. In alternative implementations, the system initializes i to be 1, the confirmed refinements to be the first refinement in the order, and the seen documents to be the top documents for the first refinement in the order, for example, as described above with reference to FIG. 10.

[0097] The system then processes the candidate refinements. The system selects the candidate refinement identified by the counter as the additional_ref (step 1104). The system then calculates an intra-suggestion diversity score between a top number of documents responsive to additional_ref and the seen_docs, and compares the intra-suggestion score to a threshold (step 1106), for example, as described above with reference to FIG. 10. If the intra-suggestion diversity score satisfies the threshold (e.g., exceeds the threshold), then additional_ref is added to the group of confirmed_refs, and the top documents for additional_ref are added to the group of seen_docs (step 1108). The system continues to process the additional references by incrementing i (step 1112), until the system determines that a desired number of confirmed refinements have been identified (step 1110) or until all candidate refinements have been processed (step 1114).

[0098] FIG. 12 illustrates an example component-based query refinement engine 1200. In general, the engine 1200 identifies components of a query and generates a group of one or more query refinements for the query based on refinements of the components. In some implementations, the engine 1200 is included in the search system 214 illustrated in FIG. 2 (e.g., as refinement engine 260). In alternative implementations, the engine 1200 is separate from the search system, and merely provides refinements to a search system.

[0099] The engine 1200 includes a component identifier 1202, a component refinement engine 1204, a component refinement combiner 1206, and a candidate refinement filter 1208. The boxes shown in FIG. 12 logically represent executing software components or modules. These components can be combined or subdivided in ways other than those shown in FIG. 12 and can be distributed on two or more computing devices.

[0100] The component identifier receives queries and, for each query, identifies components of the queries. The components are, for example, non-compositional compounds. An example method for identifying components is described in more detail below, with reference to FIG. 13.

[0101] The component refinement engine 1204 determines, for each component of each query, a variety of different component refinements. The component refinements are query refinements of the components. In some implementations, a component refinement is determined using conventional techniques. Other techniques for determining component refinements are possible. In some implementations, the component refinement engine 1204 employs the refinement engine 560 of FIG. 5 to determine component refinements. The refinement engine 560 uses user preference data 504 to determine component refinements. In some implementations, the component refinement engine 1204 uses a query refinement database, e.g., the query refinement database 516 of FIG. 5, to determine component refinements. Various techniques are possible for determining component refinements.

[0102] The component refinement combiner 1206 combines component refinements to create candidate query refinements for the queries. For example, the component refinement combiner 1206 can combine component refinements in the order that the components were found in an original query from the queries 1210. An example method for combining component refinements is described in more detail below, with reference to FIG. 14.

[0103] The candidate refinement filter 1208 filters the candidate query refinements generated by the component refinement combiner 1206, resulting in a final set of query refinements 1212 for the queries 1210. The candidate refinement filter 1208 uses, for example, user preference data 504 to filter the candidate query refinements. An example method for filtering candidate query refinements is described in more detail below, with reference to FIG. 15.

[0104] The final query refinements 1212 are typically stored for later retrieval by a search engine. When the search engine receives a query having stored query refinements 1212, the search engine presents the query refinements in response to the query along with the search results. In the case where the component-based refinement engine 1200 receives a query and the component identifier 1202 only identifies one component in the query, the refinement engine 1200 operates like refinement engine 560 of FIG. 5.

[0105] FIG. 13 is a flow diagram of an example method 1300 for identifying components in a query. For convenience, the example method 1300 will be described in reference to a system that performs the method 1300. The system can be, for example, the component identifier 1202 of FIG. 12.

[0106] The system divides the query into possible combinations of n-grams (step 1302). An n-gram is a sequence of n consecutive tokens, e.g., words or characters. An n-gram has an order, which is the number of tokens in the n-gram. For example, a 1-gram (or unigram) includes one token; a 2-gram (or bi-gram) includes two tokens. As an example, a 2-gram can be the term “new york”, where the tokens are words. As another example, a 3-gram can be the term “new york times”.

[0107] In some implementations, the system divides the query into all possible combinations of n-grams that are in order as they were in the query. Consider the example query [a b c] where a, b, and c are each tokens. All the possible n-grams that are in order are as follows: [a], [b], and [c]; [a b] and [c]; [a] and [b c]; and [a b c]. In some other implementations, the system only divides the query into some combinations of n-grams. In further implementations, the system divides the query into a subset of all possible combinations of n-grams.

[0108] For each possible combination of n-grams, the system determines a sum of probabilities for each n-gram that the n-gram is a related phrase (step 1304). A related phrase is a group of terms that are found together more frequently than those terms would be found together if they were not associated with each other. For each possible combination of n-grams, the system determines, for each n-gram in the combination, a probability that the n-gram is a related phrase. The system then adds the probabilities for the n-grams in the possible combination to determine the sum.

[0109] In some implementations, the probability that an n-gram is a related phrase is calculated as a function of n-gram relative frequency in training data. Examples of training data include dictionaries, web pages, search query logs, emails, blogs, instant message (IM) scripts, and news articles. A string of L tokens is represented as W₁^L=(w₁, w₂, . . . , w_L). A probability can be assigned to the string W₁^L as:

[0110] $P\left(w_1^L\right)=\prod _{i=1}^{L}P\left(w_i|w_1^{i-1}\right)\approx \prod _{i=1}^L\hat{P}\left(w_i|w_{i-n+1}^{i-1}\right),$
where the approximation is based on a Markov assumption that only the most recent (n−1) tokens are relevant when predicting a next token in the string, and the “^” notation for P indicates that it is an approximation of the probability function.

[0111] The strings and associated probabilities can be stored in a language model. The system can use the language model to identify the one or more n-grams. As an example, the system can divide a query “new york times square” into two n-grams, i.e., two 2-grams, “new york” and “times square,” based on probabilities of the n-grams each occurring as a term in training data.

[0112] For example, consider the query [a b c] and the possible combination of n-grams [a b] and [c]. The system determines the sum of the probabilities by determining the probability that [a b] is a related phrase and the probability that [c] is a related phrase and then adding the probabilities together.

[0113] The system determines the combination having the largest sum (step 1306). For example, consider the query [a b c]. If the sum of probabilities for [a b] and [c] is greater than the sum for the other possible combinations, then the system determines that the possible combination of [a b] and [c] has the largest sum.

[0114] In some implementations, the system compares the sum for the combination having the largest sum to a threshold sum. If the sum does not exceed the threshold sum, or is beneath another threshold, then the system determines that the query as a whole is one component. The threshold sum can be adjusted to control how tightly related the tokens of an n-gram must be for the n-gram to be identified as a component.

[0115] For example, a relatively higher threshold can be used so that the system identifies components as non-compositional compounds. A non-compositional compound is an n-gram where the tokens of the n-gram are generally found together rather than separately. “New York” is an example of a non-compositional compound. In contrast, “red book” is not a non-compositional compound because the words “red” and “book” are frequently found separately.

[0116] The system identifies the n-grams from the combination having the largest sum as the components of the query (step 1308). Continuing with the above example, [a b] would be identified as one component and [c] would be identified as another component of the query [a b c].

[0117] FIG. 14 is a flow diagram of an example method 1400 for combining component refinements for components of a query. For convenience, the example method 1400 will be described in reference to a system that performs the method 1400. The system can be, for example, the component refinement combiner 1206 of FIG. 12.

[0118] The system receives component refinements for a first component of the query (step 1402). The component refinements are received from, for example, the component refinement engine 1204 of FIG. 12. The system receives component refinements for a second component of the query (step 1404). The second component of the query appears after the first component in query. For example, consider the query “new york times square,” and suppose that the component identifier 1202 identifies “new york” and “times square” as components. “new york” appears before “times square” in the query.

[0119] The system combines the component refinements for the first component with the component refinements for the second component (step 1406), thereby creating candidate refinements for the query. For each component refinement for the first component, the system combines that component refinement with the second component and the component refinements for the second component so that the component refinement for the firs component appears first. Continuing the above example of “new york” and “times square,” the component refinements of “new york” would appear before “times square” and the refinements of “times square” in the candidate refinements for the query. Similarly, the component refinements of “times square” would appear after “new york” and the refinements of “new york.”

[0120] Although the method 1400 is illustrated for two components of a query, the method can be extended for queries having more than two components. For example, for a query having three identified components, the component refinements can be combined so that the component refinements appear in the same order as the identified components and so that all possible permutations of the component refinements are created as candidate refinements for the query.

[0121] Moreover, in some implementations, additional candidate refinements are created by combining the component refinements in an order different from the order of the components in the query, e.g., reverse order. In some implementations, individual terms from the component refinements can be dropped or re-ordered to create additional candidate refinements for the query. Creating additional candidate refinements generally requires more processing time and computation resources but increases the number of possibly good candidate refinements.

[0122] Consider an example query having three terms, “a,” “b,” and “c,” denoted [a b c]. Suppose a component identifier identifies [a b] as one component and [c] as another component. Further suppose that a component refinement engine 1204 determines that a component refinement for [a b] is [d e] where “d” and “e” are terms that are different from “a” and “b.” The system can combine [d e] with [c] in order to create the following candidate query refinements: [d e c], [d c], and [e c]. The system can combine [d e] with [c] in reverse order to create the following candidate query refinements: [c d e], [c d], [c e].

[0123] FIG. 15 is a flow diagram of an example method 1500 for filtering candidate query refinements for a query. For convenience, the example method 1500 will be described in reference to a system that performs the method 1500. The system can be, for example, the candidate refinement filter 1208 of FIG. 15.

[0124] The system receives candidate query refinements for a query (step 1502). The system receives the candidates from, for example, the component refinement combiner 1206 of FIG. 12.

[0125] The system filters the candidates based on initial scores for the candidates (step 1504). The system determines an initial score for each candidate and removes candidates having an initial score below a threshold score. The initial score can be based on various factors, for example, from the user preference data 504 of FIG. 5.

[0126] In some implementations, the initial score is the number of times a user has searched for the candidate over a period of time. In other implementations, the initial score is based on quality of result statistics (e.g., click fractions or weighted clicks, described above). In still other implementations, the initial score is based on a click-through-rate for a candidate, that is, the total number of clicks on documents presented in response to the candidate, divided by the total number of impressions for documents presented in response to the candidate. In still other implementations, the initial score is a weighted click-through-rate for a candidate, for example, where the clicks on each document are weighted by how long the documents were viewed. Other scores are also possible. In some implementations, the initial scores are based on multiple factors.

[0127] The system ranks the candidates by commonality with the highest Inverse Document Frequency (IDF) components of the query (step 1506). The IDF for a given component is the number of documents in a corpus of documents divided by the number of documents in the corpus of documents that include the component. The corpus can be, for example, the corpus of documents indexed by the search system 214.

[0128] In some implementations, the system determines the IDF of each component of the query and then ranks the components according to their IDFs. The system then compares each candidate to the highest IDF components and places candidates that are similar (e.g., identical, or having a certain number of common characters) to the highest IDF component at the top of the ranking. The system then compares candidates that were not similar to the highest IDF component to the next highest IDF component, and so on.

[0129] Consider the example query [a b c] where [a b] and [c] are identified as components. Suppose that [a b] has a higher IDF than [c]. Candidates that contain [a b], or are similar to [a b], will be ranked above candidates that are not similar to [a b].

[0130] In some implementations, the system further filters candidates based on user preference data 504 (step 1508). If a candidate has three or more terms, then the system determines a weighted number of clicks on search results for the candidate, e.g., where each click on a search result is weighted by the amount of time spent viewing the search result. The system determines whether the weighted number of clicks exceeds a threshold number of weighted clicks. The system then determines a top search result for a search for the candidate (e.g., using the search engine 230 of FIG. 2) and determines whether a click-through rate for the top result in response to searches for the candidate exceeds a threshold click-through rate. If the candidate's click-through rate for its top result exceeds the threshold click-through rate, and the candidate's weighted number of clicks exceeds the threshold number of weighted clicks, then the system determines to keep the candidate as a candidate query refinement.

[0131] If the system determines that the candidate does not meet the above criteria, the system determines whether the candidate's weighted number of clicks exceed a different threshold number of weighted clicks. In some implementations, the different threshold depends on the query length, e.g., the number of terms or characters in the query. The system also determines a sum of click-through rates for the top five search results for a search for the candidate (e.g., using the search engine 230 of FIG. 2) and determines whether the sum exceeds a threshold sum. If the sum exceeds the threshold and the candidate's weighted number of clicks exceed the different threshold, and the candidate's click-through rate for its top result exceeds its threshold (discussed above), then the system determines to keep the candidate as a candidate query refinement. Otherwise, the system removes the candidate as a candidate query refinement.

[0132] The system filters candidates based on syntactic similarity with the query (step 1510). The system determines a syntactic similarity score for each candidate and removes candidates having a syntactic similarity score below a threshold.

[0133] In general, syntactic similarity refers to whether the candidates appear similar to the query rather than whether the candidates have a meaning similar to the query (referred to as semantic similarity). In some implementations, the syntactic similarity score for a candidate is the edit distance (sometimes referred to as a hamming distance) between the candidate and the query. The edit distance is the number of edits (e.g., character changes, deletions, and the like) required to change the candidate into the query. Various techniques are possible for determining edit distances.

[0134] FIG. 16 illustrates an example architecture of a system 1600. The system generally consists of a server 1602 and one or more user computers such as user computer A 1604. The server 1602 and user computers are connected through a network 1606.

[0135] The server 1602 consists of one or more data processing apparatus. While only one data processing apparatus is shown in FIG. 16, multiple data processing apparatus can be used. The server includes various modules, e.g. executable software programs, including a refinement engine 1200 and a search system 1608. Each module runs as part of the operating system on the server, runs as an application on the server, or runs as part of the operating system and part of an application on the server.

[0136] The refinement engine 1200 includes a component identifier 1202, a component refinement engine 1204, a component refinement combiner 1206, and a candidate refinement filter 1208. The refinement engine 1200 is described above with reference to FIGS. 12-15.

[0137] The optional search system 1608 includes a search engine 1610, for example, the search engine 230 described above with reference to FIG. 2, and a refinement selector 1612. The search system 1608 uses the search engine 1610 to generate search results responsive to queries, and builds a model of user preference data from user behavior, for example, as described above with reference to FIGS. 2-4. The refinement selector selects confirmed refinements that have been selected for a user-submitted query by the refinement engine 1200, and presents them to a user along with search results responsive to the query.

[0138] The server 1602 can also have hardware or firmware devices including one or more processors 1614, one or more additional devices 1616, computer readable medium 1618, a communication interface 1620, and one or more user interface devices 1622. Each processor 1614 is capable of processing instructions for execution within the server 1602. In some implementations, the processor 1614 is a single-threaded processor. In other implementations, the processor 1614 is a multi-threaded processor. Each processor 1614 is capable of processing instructions stored on the computer readable medium 1618 or on the storage device 1630. The server 1602 uses its communication interface 1620 to communicate with one or more computers, for example, over a network. Examples of user interface devices 1622 include a display, a camera, a speaker, a microphone, a tactile feedback device, a keyboard, and a mouse.

[0139] The server 1602 can store instructions that implement operations associated with the modules described above, for example, on the computer readable medium 1618 or one or more additional devices 1616, for example, one or more of a floppy disk device, a hard disk device, an optical disk device, or a tape device. In some implementations, the data processing apparatus 1602 also stores additional data, for example, user preference data, document-to-query-to-document maps, candidate refinements for queries, confirmed refinements for queries, or other data, on the computer readable medium 1618 or one or more additional devices.

[0140] User computers such as user computer A 1604 are connected to the server 1602 through the network 1606. Users submit search queries to the server 1602 and select search results received from the server 1602 using user computers such as user computer A 1604, for example, using a web-browser running on the computer. The server generates user preference data from the actions of the users, for example, by tracking what search queries are submitted by users and what search results responsive to a query are selected by users. In some implementations, this user preference data is used by the refinement engine 1200 to generate query refinements for a given query, for example, as described above with reference to FIGS. 12-15. Users can also use user computers such as user computer A 1604 to submit search queries to the server 1602 and to view search results responsive to their queries, as well as query refinements for the queries.

[0141] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on a propagated signal that is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0142] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0143] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0144] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0145] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0146] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0147] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0148] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0149] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0150] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0151] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0152] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.