Determining a relevance function based on a query error derived using a structured output learning technique

[0001] 1. Field of the Invention

[0002] The present invention relates to improved ranking of search results.

[0003] 2. Background Art

[0004] A search engine is an information retrieval system used to locate documents and other information stored on a computer system. Search engines are useful at reducing an amount of time required to find information. One well known type of search engine is a Web search engine which searches for documents, such as web pages, on the “World Wide Web.” The World Wide Web is formed by all publicly accessible websites, which primarily includes websites hosted on computer systems that are accessible over the Internet. Other types of search engines include personal search engines, mobile search engines, and enterprise search engines that search on intranets.

[0005] Development of a search engine that can index a large and diverse collection of documents, yet return to a user a short, relevant list of result documents in response to a query has long been recognized to be a difficult problem. A user of a search engine typically supplies a short query to the search engine, the query containing only a few terms, such as “hazardous waste” or “country music.” The search engine attempts to return a list of relevant documents. Although the search engine may return a list of tens or hundreds of documents, most users are likely to only view the top few documents on the list. Thus, to be useful to a user, a search engine is desired to be able to determine, from potentially billions of documents, the two or three documents that a user would be most interested in, according to the query submitted by the user.

[0006] Previously, search engine designers have attempted to construct relevance functions that take a query and a document as their input and return a relevance value. Relevance values may be used, for example, to create a list of documents indexed by the search engine. The list ranks the documents in order of relevance to the query. For the top two or three documents on this list to be useful to a user, the underlying relevance function must be able to accurately and quickly determine the relevance of a given document to a query.

[0007] A user's perception of true relevance of a document to a query is influenced by a number of factors, many of which are highly subjective. A user's preferences are generally difficult to capture in an algorithmic set of rules defining a relevance function. Furthermore, these subjective factors may change over time, such as when current events are associated with a particular query term. Changes over time in the aggregate content of the documents available in the Internet may also alter a user's perception of the relative relevance of a given document to a particular query. A user who receives a return list from a search engine that contains documents that the user does not perceive to be highly relevant may become frustrated, and may potentially abandon the use of that particular search engine.

[0008] Thus, what is needed are techniques for determining a document relevance function that reflects one or more human users' perceptions of document relevance to a query, and can rank documents quickly and efficiently.

[0009] Methods, systems, and apparatuses for generating relevance functions are provided. The generated relevance functions may be used to rank documents obtained in searches based on queries having one or more search terms. The searches may be performed in any medium, including on a computer system, on an intranet, and/or on the World Wide Web.

[0010] In a first example, one or more features to be used as predictor variables in the construction of a relevance function are determined. The relevance function is parameterized by one or more coefficients. A query error is defined that measures a difference between a relevance ranking generated by the relevance function and a training set relevance ranking based on a query and a scored set of documents. Values for the coefficients of the relevance function are determined that substantially minimize an objective function that depends on the defined query error. In some examples, the objective function may be a sum over a set of queries of the defined query error, a weighted sum over the set of queries of the defined query error, or a regularized sum over the set of queries of the defined query error.

[0011] The query error, as defined herein, is a continuous upper bound of an ideal query error. For a provided relevance measure, such as the Discounted Cumulative Gain (DCG), the ideal query error is defined as the difference between the best possible value of the relevance measure and the value corresponding to the ranking induced by the relevance function. Because the ideal query error depends on a sorting operation, the ideal query error is not continuous. Therefore, it is conventionally difficult to find the coefficients of the ranking function which minimize the sum of the query errors. In an aspect of the present invention, the continuous upper bound of the ideal query error can be optimized efficiently by gradient descent. This upper bound can be derived using the structured output learning framework, for example.

[0012] In a further aspect, the relevance function having the determined values for the one or more coefficients may be tested. If the relevance function does not satisfy the testing, one or more coefficients of the relevance function may be adjusted, other parameters may be adjusted, and the determination of values for the coefficients of the relevance function may be repeated.

[0013] In another example, a system for determining a relevance function is provided. The system includes a relevance function constructor and a relevance function tuner. The relevance function constructor is configured to construct a relevance function based on one or more features used as predictor variables and one or more coefficients. The relevance function tuner is configured to determine values for the one or more coefficients of the relevance function by gradient descent to substantially minimize an objective function that depends on a query error. The query error is a continuous upper bound on an ideal query error, the ideal query error being a difference between a relevance measure on a ranking generated by the relevance function and a ranking based on a training set.

[0014] In a further aspect, the system for determining a relevance function may include a relevance function tester configured to test the relevance function having the values for the one or more coefficients determined by the relevance function tuner.

[0015] These and other objects, advantages and features will become readily apparent in view of the following detailed description of the invention. Note that the Summary and Abstract sections may set forth one or more, but not all exemplary embodiments of the present invention as contemplated by the inventor(s).

[0025] The present invention will now be described with reference to the accompanying drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Additionally, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears.

Introduction

[0026] The present specification discloses one or more embodiments that incorporate the features of the invention. The disclosed embodiment(s) merely exemplify the invention. The scope of the invention is not limited to the disclosed embodiment(s). The invention is defined by the claims appended hereto.

[0027] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0028] Embodiments of the present invention provide methods and systems for determining relevance functions. The determined relevance functions may be used to determine the relevance of documents returned from searches based on queries, in any type of search environment. For illustrative purposes, embodiments are described below in the context of web-based document searches performed by search engines. However, embodiments of the present invention are not limited to such web-based searches, and are also applicable to other types of searches, and to searches performed in other environments, including searches performed on a single computer system, searches performed on an intranet, and searches performed in other environments.

[0029] Embodiments overcome problems that exist in conventional document searching and ranking systems. The quality of a web search ranking is frequently measured through complex relevance measures/scores, such as the Discounted Cumulative Gain (DCG). One representation of the DCG (the DCG at rank p) is shown as follows, as Equation 1:

[0030] $\begin{array}{cc}\frac{1}{n_q}\sum _{q=1}^{n_q}\sum _{j\in \mathrm{Uq}}G\left(y_j\right)D\left(r\left(x_j\right)\right)& \mathrm{Equation}1\end{array}$
where

[0031] n_q=a number of queries,

[0032] Uq=the set of documents retrieved for the query q,

[0033] y_j=a training relevance grade for document j, such as a value selected from the set {0, 1, 2, 3, 4} by a human editor,

[0034] G(y_j)=a mapping function for y, such as

• • [0035] G(y)=2^y−1,

[0036] $D\left(r\right)=\frac{1}{{\log }_2\left(r+1\right)}$
if r is less or equal to p; 0 otherwise,

[0037] x_j=a feature vector containing evaluated feature information for document j and the query, and

[0038] r(x_j)=the relevance ranking function evaluated on a feature vector x_j.

[0039] The DCG is intended to allow for multi-levels of relevance, where each document contributes some “gain” corresponding to the level of relevance. The term “cumulative” in “DCG” refers to measuring the overall utility of n documents by the sum of the gain of each relevant document. The term “discounted” in “DCG” refers to discounting the gain of a document ranked low so that a highly ranked document will count more toward the gain. Alternatively, one can use the “normalized” DCG (NCDG) which is equal to the DCG divided by the best achievable value of the DCG.

[0040] Machine learning algorithms aim to optimize the DCG quantity, but have difficulty in doing so because it is not continuous (e.g., the DCG depends on the result of the ranking itself). Thus, standard optimization techniques cannot be used. This problem is more fully described in C. J. C. Burges, R. Ragno and Q. V. Le, “Learning to Rank with Non-Smooth Cost Functions”, Neural Information Processing Systems (NIPS) Conference, Dec. 4, 2006, which is incorporated by reference herein in its entirety.

[0041] A conventional optimization technique for maximizing the DCG is to use a non-continuous optimization algorithm such as the Nealder-Mead simplex algorithm. However, this technique works only for a limited number of variables.

[0042] Embodiments of the present invention provide advantages over prior techniques. Embodiments enable learning of a relatively large number of coefficients/parameters of a new ranking model. Embodiments provide an advantage in optimizing the quantity of interest (DCG) over a large number of variables, which can lead to a significant increase in ranking performance.

[0043] The example embodiments described herein are provided for illustrative purposes, and are not limiting. Further structural and operational embodiments, including modifications/alterations, will become apparent to persons skilled in the relevant art(s) from the teachings herein.

[0044] FIG. 1 shows a document retrieval system 100, according to an example embodiment of the present invention. System 100 includes a search engine 106, a relevance function determining system 108, and a database 114. In system 100, one or more computers 104, such as first-third computers 104a-104c, are connected to a communication network 105. Network 105 may be any type of communication network, such as a local area network (LAN) or a wide area network (WAN), or combination of communication networks. In embodiments, network 105 may include the Internet and/or an intranet. Computers 104 can retrieve documents from entities over network 105. In embodiments where network 105 includes the Internet, a collection of documents, including a document 103, which form a portion of World Wide Web 102, are available for retrieval by computers 104 through network 105. On the Internet, documents may be located by a uniform resource locator (URL), such as http://www.av.com, and/or by other mechanisms. Computers 104 can access document 103 through network 105 by supplying a URL corresponding to document 103 to a document server (not shown in FIG. 1).

[0045] As shown in FIG. 1, search engine 106 is coupled to network 105. Search engine 106 typically is coupled to a database 114 that indexes documents, such as documents of World Wide Web 102. A user of computer 104a that desires to retrieve a document relevant to a particular topic, but does not know the URL of such a document, may submit a query 112 to search engine 106 through network 105. Search engine 106 receives query 112, and analyzes database 114 to find documents relevant to query 112. For example, search engine 106 may determine a set of documents indexed by database 114 that include terms of query 112. The set of documents may include any number of documents, including tens, hundreds, thousands, or even millions of documents. Search engine 106 may use a ranking or relevance function 116 to rank documents of the retrieved set of documents in an order of relevance to the user. Documents of the set determined to likely be most relevant by relevance function 116 may be provided at the top of a list of the returned documents in an attempt to avoid the user having to analyze the entire set of documents.

[0046] Search engine 106 may be implemented in hardware, software, firmware, or any combination thereof. For example, search engine 106 may include software/firmware that executes in one or more processors of one or more computer systems, such as one or more servers. Examples of search engine 106 that are accessible through network 105 include, but are not limited to, Yahoo! Search™ (at http://www.yahoo.com), Ask.com™ (at http://www.ask.com), and Google™ (at http://www.google.com).

[0047] As shown in FIG. 1, relevance function determining system 108 is coupled to search engine 106. In embodiments, relevance function determining system 108 is used to generate relevance function 116 used by search engine 106. Relevance function determining system 108 determines relevance function 116, and transmits relevance function 116 to search engine 106. Relevance function 116 is a learning architecture with associated coefficients. Relevance function determining system 108 selects values for the coefficients to minimize a query error, thereby generating relevance function 116.

[0048] Relevance function determining system 108 may optionally be coupled to network 105. Relevance function determining system 108 may be implemented in hardware, software, firmware, or any combination thereof. For example, relevance function determining system 108 may include software/firmware that executes in one or more processors of one or more computer systems, such as one or more servers. In embodiments, relevance function determining system 108 may be implemented in a different, or in the same computer system(s) that implement search engine 106. When implemented in different computer systems, relevance function determining system 108 may communicate with search engine 106 via any suitable communication link, including network 105 or other network or communication link.

[0049] In embodiments, to generate relevance function 116, one or more training queries are used to generate one or more corresponding sets of documents. In one example, the plurality of training queries are determined, at least in part, based on logs of queries submitted by users of computers 104 to search engine 106. In an embodiment, the resulting sets of documents may be generated by submitting the training queries to one or more search engines 106 and receiving a response, typically a list of URLs, therefrom.

[0050] Relevance function determining system 108 receives the training queries and corresponding sets of documents. Relevance function determining system 108 additionally receives training information 118 from one or more human subjects 110. For example, human subjects 110 may be used to provide a set of training relevance scores for the documents in the result sets relative to their respective training queries. Based on the training queries, the documents in the result sets, and the training relevance scores, relevance function determining system 108 generates relevance function 116.

[0051] FIG. 2 shows an example query 112 that may be submitted by a user of one of computers 104a-104c of FIG. 1. Query 112 includes one or more terms 202, such as first, second, and third terms 202a-202c shown in FIG. 2. Any number of terms 202 may be present in a query. FIG. 3 shows a plurality 302 of training queries that includes example queries 112a (“unibomber”), 112b (“Australian weather”), and 112c (“Star Wars”). FIG. 3 also shows a plurality 306 of result sets 304a-304n corresponding to training queries 112a-112n of plurality 302. In an embodiment, each training query 112 of plurality 302 is submitted to one or more search engines 106 to generate a respective result set 304. For example, FIG. 3 shows a Query 1 having a result set 304a containing documents 308a-308p. Each document in a result set 304 typically contains one or more of the terms in the respective training query 112.

[0052] FIG. 4 shows a flowchart 400 for determining a relevance function, according to an example embodiment of the present invention. The steps of flowchart 400 do not necessary occur in the order shown in FIG. 4, in all embodiments. Furthermore, not all steps of flowchart 400 need to be performed in all embodiments. Further structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart 400. Flowchart 400 is described as follows.

[0053] Flowchart 400 begins with step 402. In step 402, one or more features to be used as predictor variables in the construction of a relevance function are determined. “Feature,” as used herein, is a term used to quantify an aspect of the relationship of a query to a document, or of an aspect of the document itself. With a document as input, and possibly a query, a feature returns a value. Features that return a value based only the contents of a document itself are referred to as query-independent features. Query-independent features may depend on properties of the document itself. Additionally, query-independent features may depend on properties of the server on which the document is located, and possibly properties of the relationship of the server to other servers on the Web. Features that require both a document and a query to return one or more values are referred to as query-dependent features. Throughout this document, the ith feature selected to relate a document to a query is denoted x_i. The set of features selected, including one or more individual features are, for notational convenience, denoted x, where x={x₁, x₂, . . . , x_d}, and d is the number of features so selected.

[0054] Any type of feature may be used as a predictor variable in embodiments of the present invention. For examples of query-independent and query dependent features applicable to embodiments of the present invention, refer to U.S. Pat. No. 7,197,497 to David Cossack, titled “Method and Apparatus for Machine Learning a Document Relevance Function,” which is incorporated by reference herein in its entirety.

[0055] Referring to FIG. 4, in step 404, the relevance function is parameterized by one or more associated coefficients. These coefficients are encoded in a vector w. For example, in an embodiment, relevance function 116 is initially generated as a combination of one or more base functions that incorporate one or more features, and associated coefficients. In an example embodiment, the initial relevance function may be parameterized as a linear function of features, such as:

[0056] $\begin{array}{cc}f\left(x,w\right)=\sum _{i=1}^d{x}_i{w}_i& \mathrm{Equation}2\end{array}$
where

[0057] f(x,w)=the relevance function,

[0058] x=a feature vector representing a (query, document) pair,

[0059] w=is a vector of coefficients, and

[0060] d=a number of coefficients and features.

In another embodiment, the parameterization of the relevance function may involve more complex functions of the features, such as:

[0061] $\begin{array}{cc}f\left(x,w\right)=\sum _{i=1}^d{w}_i{a}_i\left(x\right)& \mathrm{Equation}3\end{array}$
where

[0062] a_i(x)=is a base function.

For example, a_i(x) may be a base function that receives as input a portion or all of the set of features, and outputs a value. The base function may include quadratic functions of some or all the set of features.

[0063] In some embodiments, the base functions may include wavelet functions, with each wavelet function related to a mother wavelet through a process of dilation, translation, or other possible processes, the precise relationship determined by additional coefficients. In some embodiments, the base functions may include neural networks. In embodiments, the base functions themselves may depend on coefficients that have to be learned, such as a radial basis function with center c_i:
a_i(x)=exp(−dist(x,c_i)),  Equation 4
where

[0064] dist=a distance function, and

[0065] c_i is vector of coefficients and is thus part of the coefficient vector w.

[0066] In some embodiments, a base function includes a regression tree. For example, a base learner may be a regression tree 500, as depicted in FIG. 5. Referring to FIG. 5, a regression tree in block 502a examines a first “splitting variable,” x₁, and compares its value to a “splitting location” b₁. For example, the splitting variable may be equal to the value of one of the features, x₁. In another example, the splitting variable may be a linear or nonlinear function of one or more of the features in the set of selected features. Depending on the result of the comparison performed in block 502a, the flow of tree 500 proceeds to either block 502b or block 502c. Each block 502 has an associated splitting variable and a splitting location. The splitting variables (e.g., 508a) and splitting locations (e.g., 508b) are coefficients 508 that are needed to determine the base function represented by tree 500.

[0067] Based on the comparisons made in blocks 502, the flow of tree 500 proceeds to one of the four terminal nodes 506a-506d. In each terminal node, a terminal value (e.g., 508c), is assigned as the value of the base function. The terminal values (for example, 508c) are also coefficients 508 that are needed to determine the base function represented by tree 500.

[0068] In some embodiments where the base functions are regression classification trees, the relevance function may be determined as a function of a linear combination of the terminal values returned from application of each base function to the set of selected features.

[0069] Referring to FIG. 4, in step 406, a query error is defined. In embodiments, the query error is an error related to a particular relevance measure, such as a discounted cumulative gain (DCG) error. For a provided relevance measure, such as the DCG, the ideal query error is defined as the difference between the best possible value of the relevance measure and the value corresponding to the ranking induced by the relevance function. Because the ideal query error depends on a sorting operation, the ideal query error is not continuous. Therefore, it is conventionally difficult to find the coefficients of the ranking function which minimize the sum of the query errors. In an embodiment, the continuous upper bound of the ideal query error can be optimized efficiently by gradient descent, as further described below.

[0070] Embodiments of the present invention incorporate aspects of a structured output learning framework. For further description of a structured output learning framework, refer to Tsochantaridis et al., “Large Margin Methods for Structured and Interdependent Output Variables,” Journal of Machine Learning Research, Vol. 6, September 2005, pages 1453-1484, which is incorporated by reference herein in its entirety. In an embodiment, the structured output learning framework is applied where the input corresponds to a set of documents, the output is a ranking, and the discrepancy measure is the difference with respect to the relevance measure of two rankings.

[0071] In an embodiment, the query error is defined according to Equation 5 shown below:

[0072] $\begin{array}{cc}\underset{\pi }{\max }\sum _{i\in U_q}\left(A\left(\pi \left(i\right)\right)-A\left({\pi }^0\left(i\right)\right)\right)f\left(x_i,w\right)+\left(D\left({\pi }^0\left(i\right)\right)-D\left(\pi \left(i\right)\right)\right)G\left(y_i\right)& \mathrm{Equation}5\end{array}$
where

[0073] D=a discount function,

[0074] NDq=a number of documents in the set of documents for the query,

[0075] Uq=the set of documents for the query q,

[0076] y_i=a training relevance score for document i,

[0077] G(y_i)=a mapping function for y_i,

[0078] A=a non-increasing function,

[0079] π=a permutation from Uq to 1-NDq, and

[0080] π⁰=a permutation from Uq to 1-NDq obtained by sorting y_i in descending order.

Equation 5 has been derived using the structured output learning framework mentioned above where the scoring function between the input {x_i, i ε U_q}, and the output permutation π is defined as Σ_iεUqA(π(i)f(x_i,w).

[0081] Note that when π is a permutation obtained by sorting f(x_i,w) (i.e., when π is the predicted relevance ranking), the first term Σ_iεUq(A(π(i))−A(π⁰(i)))f(x_i,w) in Equation 5 is non-negative, while the second term Σ_iεUq(D(π⁰(i))−D((i)))G(y_i) is a difference between the best DCG and the predicted DCG. As a consequence, Equation 5 is an upper bound on this difference. If f is a continuous function of the parameter vector w, so is Equation 5. Likewise, if f is a convex function in w, Equation 5 is convex because of a pointwise maximum of convex functions is convex.

[0082] Discount function D(j) may be configured to discount documents to any degree and/or manner, as desired for a particular application. For example, in an embodiment, the discount function D(j) may be expressed as shown below in Equation 6:

[0083] $\begin{array}{cc}D\left(j\right)=\frac{1}{{\log }_2\left(j+1\right)}& \mathrm{Equation}6\end{array}$
D(j) can also be optionally truncated at a certain rank p, such that D(j)=0 for j>p.

[0084] Training relevance rankings y_i may be generated in any manner, including being assigned by human subjects 110 that are provided with the associated query and retrieved documents, as described above. Any range of values may be assigned to relevance rankings y_i, such as by selecting values from the set {0, 1, 2, 3, 4}, where a ranking of 0 indicates little to no relevance, and a ranking of 4 indicates extremely high relevance for the document being ranked.

[0085] Mapping function G(y_i) is configured to map values of training relevance rankings y_i to a second set of values. For example, in an embodiment, mapping function G(y) may be 2^y−1.

[0086] The function A measures the importance of documents at different positions and can be any non-decreasing function. It can for instance be chosen to be equal to D, but in embodiments, other choices for function A may be made. In particular, function A may be tuned as described in step 416 below.

[0087] Feature vector x_i contains evaluated information for a number of features, as described above or referenced elsewhere.

[0088] In another embodiment, the query error is defined according to Equation 7 shown below:

[0089] $\begin{array}{cc}\underset{\pi }{\max }\sum _{i\in U_q}\left(A\left(\pi \left(i\right)\right)-A\left({\pi }^1\left(i\right)\right)\right)f\left(x_i,w\right)+\left(D\left({\pi }^0\left(i\right)\right)-D\left(\pi \left(i\right)\right)\right)G\left(y_i\right)& \mathrm{Equation}7\end{array}$
where

[0090] π¹=a permutation from Uq to 1-NDq obtained by sorting f(x_i,w) in descending order.

[0091] As in Equation 5, Equation 7 is a continuous upper bound on the difference between the best DCG and the predicted DCG. Moreover, this upper bound is tighter than the one provided in Equation 5. However, even if f is convex in w, Equation 7 may not necessarily be convex.

[0092] In step 408, an objective function that depends on the defined query error is defined. The objective function may be defined to be a sum over a set of queries of the defined query error, a weighted sum over the set of queries of the defined query error, or a regularized sum over the set of queries of the defined query error, in some example embodiments.

[0093] In an embodiment, Equation 9 shown below may be used to calculate a sum of query errors based on the defined query error of Equation 5:

[0094] $\begin{array}{cc}\sum _{q=1}^{n_q}\left(\underset{\pi }{\max }\sum _{i\in U_q}\left(A\left(\pi \left(i\right)\right)-A\left({\pi }^0\left(i\right)\right)\right)f\left(x_i,w\right)+\left(D\left({\pi }^0\left(i\right)\right)-D\left(\pi \left(i\right)\right)\right)G\left(y_i\right)\right)& \mathrm{Equation}9\end{array}$

[0095] In an embodiment, Equation 10 shown below may be used to calculate a sum of query errors based on the defined query error of Equation 7:

[0096] $\begin{array}{cc}\sum _{q=1}^{n_q}\left(\underset{\pi }{\max }\sum _{i\in U_q}\left(A\left(\pi \left(i\right)\right)-A\left({\pi }^1\left(i\right)\right)\right)f\left(x_i,w\right)+\left(D\left({\pi }^0\left(i\right)\right)-D\left(\pi \left(i\right)\right)\right)G\left(y_i\right)\right)& \mathrm{Equation}10\end{array}$

[0097] In step 410, values for the coefficients of the relevance function are determined to substantially minimize an objective function that depends on the defined query error. For example, in an embodiment where the objective function is a sum over a set of queries of the defined query error, this minimization may be carried out through the process shown in FIG. 6. FIG. 6 shows a flowchart 600 for determining values for the coefficients of the relevance function in step 410 of flowchart 400, according to an example embodiment of the present invention. Flowchart 600 determines values for the coefficients according to an example gradient descent technique. In further embodiments, techniques other than shown in FIG. 6, such as those described in the previous paragraph, may be used to determine values for the coefficients. Further structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart 600. Flowchart 600 is described as follows.

[0098] Flowchart 600 begins with step 602. In step 602, the one or more coefficients is/are initialized. The weight vector w, which contains the coefficients of the relevance function, can be initialized to a random vector, or may be initialized to vector that has already been determined (e.g., by a different optimization technique). For instance, w may be initialized to a value of w₀ determined by a regression solution, such as where w is selected to minimize the mean squared error, as described in U.S. Pat. No. 7,197,497 to David Cossack, which was incorporated herein by reference in its entirety further above.

[0099] In step 604, a permutation maximizing the query error is calculated for each query. When the query error is defined as a maximum over permutations such as Equation 5 or Equation 7, step 604 computes for each query the permutation that maximizes the query error. For Equations 5 and 7, this is a linear assignment problem (also called bipartite graph matching) which can solved in polynomial time using suitable algorithms, such as the Kuhn-Munkres algorithm. Linear assignment problem solving techniques, such as the Kuhn-Munkres algorithm, are well known to persons skilled in the relevant art(s). When several permutations maximizing the query error exist, one of them may be selected for further processing according to flowchart 600 in any manner, including selecting one of them at random.

[0100] In step 606, an objective function and a gradient of the objective function are calculated. The objective function defined in step 408 is calculated over the number of queries, n_q, with the relevance function having coefficients as initialized in step 602 (first iteration) or as modified as described below (e.g., as modified in step 606 for a subsequent iteration). Because the permutations maximizing the query errors have be found in step 604, it is straightforward to compute the objective function and its gradient.

[0101] In step 608, the one or more coefficients is/are modified in a negative direction of the calculated gradient. Values of the one or more coefficients may be modified by any amount, as appropriate for the particular application. In one example embodiment, the values of the coefficients may be modified through a “line search,” where an approximate minimum of the function along the gradient direction is found.

[0102] Note that the objective function is non-differentiable. Non-differentiability occurs when the permutation maximizing the query error is not unique. As explained above, in this case, we may select any maximizer (e.g., selected at random). Thus, to be more precise, the described technique is not a typical gradient descent algorithm, but instead is a subgradient descent algorithm (which may be considered a form of gradient descent algorithm). However, such a technique can be slow to converge, and therefore, an alternative non-differentiable optimization technique may be used, such as the so-called bundle method. Bundle methods are well known to persons skilled in the relevant art(s).

[0103] In step 610, whether the training is complete is evaluated. For instance, in an embodiment, if the norm of the calculated gradient is less than a predetermined value, the training is determined to be complete. If the norm of the calculated gradient is greater than the predetermined value, the training is determined to not be complete. If the training is determined to be complete in step 610, operation proceeds to step 612. If the training is determined to not be complete in step 610, operation proceeds to step 604. In such case (training not complete), step 604 can be repeated (recalculating for each query the permutation maximizing the query error), step 606 can be repeated (recalculating the objective function and recalculating the gradient), and step 608 can be repeated (modifying the one or more coefficients in a negative direction of the recalculated gradient). Step 610 may then be repeated to determine whether training is complete, with steps 604, 606, and 608 being repeated until training is determined to be complete in step 610.

[0104] In step 612, flowchart 600 is completed. In an embodiment, as shown in FIG. 4, operation may proceed to step 412 if it is desired to test the relevance function generated by steps 402-410 of flowchart 400. Alternatively, operation of flowchart 400 may be complete after completion of flowchart 600.

[0105] In step 412, the relevance function is tested. The relevance function may be tested in any manner, as would be known to persons skilled in the relevant art(s). For example, in an embodiment, training information 118 generated by human subjects 110 in FIG. 1 may be used to test the relevance function. In an embodiment, a first portion of training information 118 (e.g., a first set of queries, associated documents, and associated relevance scores) may be used to determine relevance function 116 as described above, and a second portion of training information 118 may be used to test the relevance function in step 412. For example, the second portion of training information 118 may be a holdout validation set generated by randomly selecting a set of queries/documents/relevance scores from training information 118, while the remainder of training information 118 is the first portion that is used to determine relevance function.

[0106] According to step 412, in an embodiment, the relevance function may be used to rank the documents of the one or more test queries. Any measure can be used to test the ranking produced by the relevance function. For example, in an embodiment, the average DCG of the test queries is computed.

[0107] In decision step 414, whether the relevance function performance is sufficient is evaluated. The performance of the relevance function may be evaluated in any manner, based on the results of the test of the relevance function in step 412. For example, in an embodiment, if the average DCG is above a certain threshold, the performance of the relevance function is determined to be sufficient. If the performance is determined to be sufficient in step 414, operation proceeds to step 418. If the performance is determined to not be sufficient in step 414, operation proceeds to step 416.

[0108] In step 416, at least one parameter of the defined query error is adjusted. For example, the function A of Equation 5 or 7 may be adjusted. After step 416 is complete, operation proceeds to step 410. Steps 416, 410, and 412 may be repeated as many times as necessary, until the performance of the relevance function is determined to be sufficient in step 414.

[0109] In step 418, the relevance function is enabled. For example, relevance function 116, as generated according to flowchart 400 of FIG. 4, may be provided to search engine 106 as shown in FIG. 1, to be used to rank documents returned by search engine 106 in response to received queries.

[0110] For example, relevance function 116, as generated according to flowchart 400, may be provided to search engine 106. A user may submit a query to search engine 106. A set of documents is retrieved from database 114 by search engine 106 that are to be ranked for relevance with respect to the query. In some embodiments, only documents including one or more of the terms in the query are including in the set of documents. In other embodiments, other criteria may be used to select the set of documents. Relevance function 116 performs the ranking of the set of documents. For each document, values of the features of relevance function 116 are evaluated, as paired with the query. Relevance function 116 is used to determine a relevance value for each document as paired with the query. After the relevance value has been determined for each document in the set by relevance function 116, an ordered list of the set of documents may be produced. The ordered list may be determined so that the documents on the list are ordered in a sequence of decreasing relevance. Thus, the document that appears first on the ordered list will have the numerically greatest relevance value of all the documents in the set and the document that appears last of the ordered list will have the minimal relevance score of all the documents in the set.

[0111] In embodiments, systems may be configured in numerous ways to implement flowchart 400. For example, FIG. 7 shows a block diagram of a relevance function determining system 700, which is an example of relevance function determining system 108 shown in FIG. 1, according to an embodiment of the present invention. As shown in FIG. 7, system 700 includes a relevance function constructor 702, a relevance function tuner 704, a relevance function tester 706, and a feature evaluator 712. Relevance function constructor 702, relevance function tuner 704, relevance function tester 706, and feature evaluator 712 may be implemented in hardware, software, firmware, or any combination thereof. For example, in an embodiment, each of relevance function constructor 702, relevance function tuner 704, relevance function tester 706, and feature evaluator 712 may be implemented as code that executes in a processor. Relevance function constructor 702, relevance function tuner 704, relevance function tester 706, and feature evaluator 712 are described in detail as follows.

[0112] Relevance function constructor 702 is configured to construct a relevance function (to result in relevance function 116) based on one or more features used as predictor variables and one or more coefficients, as described above. Thus, in an embodiment, relevance function constructor 702 may perform portions or all of steps 402 and 404 of flowchart 400. For example, as shown in FIG. 7, relevance function constructor 702 receives a feature indication 714, which indicates to relevance function constructor 702 the features to be used as predictor variables in the relevance function. Relevance function constructor 702 generates a base relevance function 708 corresponding to a specified learning machine architecture (e.g., see step 404 of flowchart 400 in FIG. 4). The coefficients of base relevance function 708 are initialized to particular values (e.g., see step 602 of flowchart 600 in FIG. 6). In embodiments, relevance function constructor 702 may perform its functions with interaction by a human user (e.g., through a computer user interface), or on an automated basis.

[0113] Feature evaluator 712 is configured to receive queries and associated documents 710, which includes one or more sets of documents associated with one or more queries. Feature evaluator 712 also receives feature indication 714. Feature evaluator 712 generates a feature vector (x_i) 720 for each document of the received documents. Feature vector 720 includes evaluations of each of the features indicated by feature indication 714.

[0114] Relevance function tuner 704 is configured to determine values for the one or more coefficients of the relevance function to substantially minimize a defined objective function, such as an objective function of the sum of query errors defined by Equation 5 shown above. As shown in FIG. 7, relevance function tuner 704 includes a query error function 724, which may be Equation 5. Thus, in an embodiment, relevance function tuner 704 may store/utilize the query error defined in step 406, and may perform step 410 to determine values for the coefficients of base relevance function 708. As shown in FIG. 7, relevance function tuner 704 receives base relevance function 708, feature vectors 720 for each document of each query to be processed, and a first portion 722a of training information 118 having training queries, retrieved documents, and relevance scores (e.g., generated by human subjects 110). In embodiments, relevance function tuner 704 determines values for the one or more coefficients of base relevance function 708 using a gradient descent technique (e.g., according to flowchart 600 of FIG. 6), a gradient boosting procedure, or other technique described elsewhere herein or otherwise known, such as utilizing Equation 9 or Equation 10 (if a regularized sum is used).

[0115] For example, in an embodiment, relevance function tuner 704 may be configured to initialize the one or more coefficients, to calculate an objective function based on the defined query error for a plurality of queries, to calculate a gradient of the calculated objective function, and to modify the one or more coefficients in a negative direction of the calculated gradient (e.g., perform flowchart 600).

[0116] Relevance function tuner 704 generates a determined relevance function 716 formed from base relevance function 708 with coefficients having values as determined by relevance function tuner 704. Relevance function tester 706 receives determined relevance function 716. Relevance function tester 706 is configured to test determined relevance function 716. For example, in an embodiment, relevance function tester 706 may perform steps 412 and 414, as described above. In an embodiment, relevance function tester 706 may receive a second portion 722b of training information 118, having test queries, retrieved documents, and relevance scores (e.g., generated by human subjects 110). If relevance function tester 706 determines that the performance of determined relevance function 716 is not sufficient, relevance function tester 706 generates an indication 718 that is received by relevance function tuner 704. If relevance function tuner 704 receives indication 718, relevance function tuner 704 is configured to adjust one or more parameters of the defined query error. Relevance function tuner 704 may then re-determine values of the coefficients of the relevance function, and generate a re-determined version of relevance function 716 for test by relevance function tester 706.

[0117] If relevance function tester 706 determines that the performance of determined relevance function 716 is sufficient, relevance function tester 706 outputs relevance function 116. Relevance function 116 may be received by search engine 106 and/or other entity for usage, as described above.

[0118] The embodiments described herein, including systems, methods/processes, and/or apparatuses, may be implemented using well known servers/computers, such as computer 802 shown in FIG. 8. For example, relevance function determining system 108 shown in FIG. 1 and in FIG. 7, flowchart 400 shown in FIG. 4, and flowchart 600 shown in FIG. 6, can be implemented using one or more computers 802.

[0119] Computer 802 can be any commercially available and well known computer capable of performing the functions described herein, such as computers available from International Business Machines, Apple, Sun, HP, Dell, Cray, etc.

[0120] Computer 802 includes one or more processors (also called central processing units, or CPUs), such as a processor 806. Processor 806 is connected to a communication bus 804. In some embodiments, processor 806 can simultaneously operate multiple computing threads.

[0121] Computer 802 also includes a main or primary memory 808, such as random access memory (RAM). Primary memory 808 has stored therein control logic 828A (computer software), and data.

[0122] Computer 802 also includes one or more secondary storage devices 810. Secondary storage devices 810 include, for example, a hard disk drive 812 and/or a removable storage device or drive 814, as well as other types of storage devices, such as memory cards and memory sticks. Removable storage drive 814 represents a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, tape backup, etc.

[0123] Removable storage drive 814 interacts with a removable storage unit 816. Removable storage unit 816 includes a computer useable or readable storage medium 824 having stored therein computer software 828B (control logic) and/or data. Removable storage unit 816 represents a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, or any other computer data storage device. Removable storage drive 814 reads from and/or writes to removable storage unit 816 in a well known manner.

[0124] Computer 802 also includes input/output/display devices 822, such as monitors, keyboards, pointing devices, etc.

[0125] Computer 802 further includes a communication or network interface 818. The network interface 818 enables the computer 802 to communicate with remote devices. For example, network interface 818 allows computer 802 to communicate over communication networks or mediums 824B (representing a form of a computer useable or readable medium), such as LANs, WANs, the Internet, etc. Network interface 818 may interface with remote sites or networks via wired or wireless connections.

[0126] Control logic 828C may be transmitted to and from computer 802 via the communication medium 824B. More particularly, computer 802 may receive and transmit carrier waves (electromagnetic signals) modulated with control logic 830 via communication medium 824B.

[0127] Any apparatus or manufacture comprising a computer useable or readable medium having control logic (software) stored therein is referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer 802, main memory 808, secondary storage devices 810, removable storage unit 816 and carrier waves modulated with control logic 830. Such computer program products, having control logic stored therein that, when executed by one or more data processing devices, cause such data processing devices to operate as described herein, represent embodiments of the invention.

[0128] The invention can work with software, hardware, and/or operating system implementations other than those described herein. Any software, hardware, and operating system implementations suitable for performing the functions described herein can be used.

[0129] While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made therein without departing from the spirit and scope of the invention. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.