Method and apparatus for database retrieval utilizing vector optimization

[0002] The present invention relates generally to the field of information retrieval. In particular, the present invention relates to a method and apparatus for selecting the optimal number of terms for retrieving documents using a vector space analysis technique.

[0003] Advances in electronic storage technology have resulted in the creation of vast databases of documents stored in electronic form. These databases can be accessed from remote locations around the world. As a result, vast amounts of information are available to a wide variety of individuals. Moreover, information is not only stored in electronic form but it is created in electronic form and disseminated throughout the world. Sources for the electronic creation of such information include news, periodicals, as well as radio, television and Internet services. All of this information is also made available to the world through computer networks, such as the worldwide web, on a real time basis. The problem with this proliferation of electronic information, however, is how any one individual may access useful information in a timely manner.

[0004] When a user wants to search for information, she may provide a computer system with a query or description of her interest. For example, a user interested in sport may type the query “basketball from Olympics '96” (the query is a phrase) or may just type the terms “basketball”, and “Olympic '96”. Using grammar rules and a lexicon, a search engine may extract the terms from the query and construct its internal representation of the query, called a profile. In the above examples, the profile will contain the terms “basketball” and “Olympics '96”.

[0005] Profile training is the process of improving the formulation of a profile using a set of documents that the user considers representative for her interest (training data). The search engine extracts new terms from the training data and adds them to the initial profile. For example, after entering the query “basketball from Olympics '96”, the user may point the system to an article that describes a basketball game from two days ago. From this article, the system extracts the terms “basketball”, “game”, “ball”, and “score”. Then the profile will contain the terms “basketball”, “Olympics '96”, “game”, “ball”, and “score”. The user may even not provide any initial description of her interest (in which case, the initial profile is empty), but just give the system some training data to extract terms from. In the above example, without an initial description, the profile will contain only the terms extracted from the article, “basketball”, “game”, “ball”, and “score”.

[0006] The main input components of profile training are the initial description, the training database, the reference database, and the terms extraction algorithm. The training database contains articles that match the user's interest (training data). The terms extraction algorithm extracts terms from the training data and adds them to the profile. The reference database contains information that helps the extraction algorithm to decide whether or not to include in the profile a term from the training data. This is because the training data may contain terms that are not related to the user's interest and if included in the profile, may return non-relevant documents. In the above example, if the training article mentions that a basketball player likes piano, then adding the term “piano” to the profile will make the search engine retrieve articles related to music, which do not correspond to the user's interest in basketball. The assumption in using a reference database is that the terms extraction algorithm differentiate between the terms in the training data that are linked to the user's interest and the terms that are not.

[0007] Typically, the training documents contain a large number of terms. Selecting only the most representative terms from this set can improve efficiency and effectiveness of the retrieval process. To make use of training documents, a terms extraction algorithm creates a list of all the terms from the training data. To every term it attaches a weight based on the information in the reference database. The terms are then sorted in decreasing order of their weights, such that the term with the highest weight is the first. If the search engine wants to add to the profile n terms from the training data, then the first n terms from the sorted list of terms are added to the profile. Therefore, to train a profile we need two important elements: (1) a method to assign weights to the terms, and (2) a cut-off method to determine the number of terms to be added in a profile.

[0008] There have been may term selection methods proposed in literature based on the vector space and probabilistic models. Regardless of the method, the number of terms in a profile is generally the value for which experiments show a reasonable behavior (e.g., the first 30 or 50 terms) and it is a constant for all the profiles. There are also methods that associate a different number of terms to each profile. One example is to compute the number of terms with the formula 10+10 log(T), where (T) is the number of training documents per profile. However, the number of terms chosen according to such a formula is generally too large and there are many cases when more flexibility is needed. For example, there are document collections in which many profiles achieve best average precision with just one term. Another method is to compute the sum of the weights for all the terms and add terms in a profile until a specified fraction of the sum is achieved. This approach, again, may not detect the situations when profiles need very few terms.

[0009] It is an object of the present invention to provide a method and apparatus to effect improved information extraction from a variety of data sources.

[0010] It is a further object of the present invention to improve information extraction by selecting the appropriate number of terms in creating a profile.

[0011] It is a further object of the present invention to improve information extraction by minimizing the number of terms in a profile.

[0012] FIG. 1 is a block diagram that illustrates a computer system for performing information extraction according to one embodiment of the present invention.

[0013] FIG. 2 illustrates the curve of terms' weights for three profiles according to one embodiment of the present invention.

[0014] FIG. 3 illustrates a block diagram of a general profile training architecture according to one embodiment of the present invention.

[0015] FIG. 4 illustrates a curve of term weights used for selecting the optimal number of terms according to one embodiment of the present invention.

[0016] FIG. 5 illustrates a curve of term weights used for selecting the optimal number of terms according to one embodiment of the present invention.

[0017] FIG. 6 illustrates the average precision and the average number of terms/profile for multiple databases calculated according to one embodiment of the present invention.

[0018] FIG. 7 illustrates the average precision and the average number of terms/profile for multiple databases calculated according to another embodiment of the present invention.

[0019] The present invention is a technique for optimizing the number of terms in a profile used for information extraction. This optimization is performed by estimating the number of terms which will substantively affect the information extraction process. That is, the technique estimates the point in a term weight curve where that curve becomes flat. A term generally is important and remains part of the profile as long as its weight and the weight of the next term may be differentiated. When terms' weights are not differentiable, then they are not significant and may be cut off. Reducing the number of terms used in a profile increases the efficiency and effectiveness of the information retrieval process.

[0020] FIG. 1 is a block diagram of a computer system used for retrieving information from a database. Computer 20 comprises a central processing unit (CPU) 30 and main memory 40. Computer 20 is connected to an Input/Output (I/O) system 10 and disk storage unit 50. The I/O system 10 includes a display 5, a keyboard 7 and a mouse 9. Furthermore, this computer system is connected to a variety of networks for communicating with other computers and obtaining access to remote databases. Among the networks connected to this computer system is the worldwide web 55, an intranet 57, private external network 59. In general, the disk storage unit 50 stores the program for operating the computer system and it stores the documents of the database. The computer 20 interacts with the I/O system 10 and the disk storage unit 50. The computer 20 executes operations according to instructions contained in a program that is retrieved from the disk storage unit 50. This program causes the computer 20 to retrieve the text of documents, or parts thereof, which are stored in a database located either in disk storage 50 or in a storage location accessible over a network. These instructions also cause information received over a network to be distributed to specific individuals over a network based on the content of the information.

[0021] According to the present invention, the program that performs the information extraction incorporates instructions to modify the profile based on the number of terms in the profile. Specifically, in the present invention, the initial number of terms in a profile is computed. This number of terms is then reduced by using the observation that there is a correlation between the curve of the terms' weights and the average precision achieved by a profile. More precisely, the average precision of a profile increases as long as the curve of weights is sharp. When the curve becomes flat, the average precision either remains constant or decreases.

[0022] FIGS. 2(a)-(b) show the curve of terms' weights for two profiles of the Reuters collection: oat and housing, when Rocchio was used for term selection. Looking at FIGS. 2(a) and 2(b), we can see that the curve of terms' weights for profile oat becomes flat at around 10 terms, while the curve of terms' weights for profile housing is sharp for 80 terms. Adding more terms for profile oat hurts its average precision, whereas for profile housing, not adding more terms hurts its performance. From the examples above, if the number of terms for which the curve of terms' weights of a profile becomes flat was estimated, the average precision of the whole profile set (as the average precision of every profile increases) could be increased and the number of terms per profile could be reduced, which increases the efficiency of the system.

[0023] FIG. 3 illustrates a block diagram for a system which uses training data to modify and/or generate profiles and to extract information based on those profiles. In particular, the inputs to the system consist of a user description 120, reference database 130 and training documents 110. The inputs are combined in a term extraction algorithm 140. This term extraction algorithm may be one of several conventional techniques such as Rocchio or RocchioFQ. As is well known, the term extraction algorithm determines a term weight based on, among other things, the term frequency in a document, the inverse document frequency of a term and the total number of training documents. The output of the term extraction algorithm 140 is one or more profiles 150 (e.g., sports, music, art, etc.). These profiles are then used by the information extraction process 160 to extract documents from a data source (e.g., a database or data stream) and create a result dataset 170.

[0024] The present invention modifies one or more of the profiles 150 by estimating the number of terms for which a profile achieves its maximum average precision in returning relevant documents and then reducing the profile to that number of terms. This estimation method uses the curve of terms' weights to determine the number of terms for which the curve becomes flat. The main concept behind the estimation method is the notion of a term's importance. A term is important as long as the difference between its weight and the weight of the next term is “significant”, therefore as long as there is a differentiating characteristic between this term and the next one. When the differences between the term weights become very small or are the same, there is no differentiating characteristic among the terms, and they become unimportant.

[0025] Two techniques are described herein to carry out the method of the present invention. The first technique is used when the number of terms in the training documents is low (e.g., less than 80 terms). In the first technique, the first N terms are examined and are divided into their weight range in L levels or bins (see FIG. 4, where L=3). The values that divide two bins are called thresholds. In particular, level i is bounded by thresholds i−1, i. For a set of weights, w(0), w(1), w(2), . . . , w(N), the thresholds are computed as follows. The first threshold, T[0] equals w(0), the highest weight. The last threshold, T[L], equals the smallest weight. In general, threshold T[i] is the average of the previous threshold and the last threshold. $T\left[i\right]=\frac{T\left[i-1\right]+T\left[L\right]}{2}$

[0026] For each level, we compute the number of weights that lay between the upper and lower thresholds. After that, in each level, we add terms to the profile until one of the next two conditions holds:

[0027] (1) αN_i<N_i+1, where α is a coefficient that controls how many terms to add, N_iis the number of weights at level i, and N_i+1is the number of weights at level i+1,

[0028] or

[0029] (2) we have reached the last level.

[0030] The condition αN_i<N_i+1, means that when the number of weights in a bin is smaller than the number of weights in the next bin, the process stops. This is under the assumption that when there is not much difference between the weights of the terms, the curve is flat and the number of the points in that level increases.

[0031] The second technique is used when the number of terms in the training documents is large (greater than approximately 80 terms). This method is very similar to the previous one, but simpler. For the first N weights, we define “min” to be the value of the smallest weight and “max” to be the value of the largest weight (by analogy with the previous method, min is T[L] and max is T[0]). With these considerations, we add terms in a profile as long as the term weight w(t) satisfies the following formula:

[0032] w(t)>=min+perc*(max−min),

[0033] where perc (percentage) is a parameter. With this formula, as long as perc>0, we never take all N terms (see FIG. 5), whereas in the case of the first method, if condition (1) is not satisfied for any bin, we end up taking N terms.

[0034] To study the above described technique, the following corpora were analyzed: Reuters (approximately 30 terms per document), Associated Press (approximately 80 terms per document), Wall Street Journal (approximately 132 terms per document) and FBIS (approximately 220 terms per document). Ideally, for each term selection method (Rocchio, RocchioFQ, Prob2), the profile modification techniques will reduce the number of terms per profile while increasing or preserving the highest average precision obtained by that method with a fixed number of terms/profile. The first algorithm has three parameters: α, Levels and N, the maximum number of terms on the curve of terms' weights. In the experiments we run, we varied α and Levels within the range [3,8], while N, we choose two values, 80 and 300. In general, as expected, the more levels we have and the larger α is, the more terms we choose.

[0035] FIG. 6 summarizes the results for each corpus. The first line in each table shows the baseline maximum average precision and number of terms, obtained where profiles have a fixed number of terms. The second line shows the maximum average precision and the average number of terms/profile obtained when using the levels cut-off algorithm and the corresponding (α, Levels) values. The third and fourth lines show the values obtained with α=8 and Levels=3 (settings that gave a very good behavior on Reuters) and respectively α=8 and Levels=6 (settings that gave a reasonable behavior on AP, WSJ and FBIS). The values in bold letters correspond to the case when our algorithm achieves a better/very close average precision than/to the best average precision obtained with the corresponding term selection method, when all the profiles have the same number of terms. The notion “very close average precision” stands for the situations when the average precision obtained with our method is smaller than the maximum average precision with less than 0.005.

[0036] A rough look at the density of values in bold letters in FIG. 6 shows us that the first algorithm works very well on Reuters and AP, where it generally not only reduces the number of terms/profile, but it also increases the maximum average precision. On Reuters, the maximum average precision is 0.5317 and it is obtained with Prob2 and 50 terms. With Rocchio, the new algorithm achieves an average precision of 0.5459 with 29 terms, increasing the average precision by 2.67%, while decreasing the number of terms by 64%. With an upper bound of 300 terms, the new method achieves with Rocchio an average precision of 0.5513 with 33 terms, a 3.68% increase in the maximum average precision obtained on this collection. For RocchioFQ, it achieves the same average precision obtained by this method with a fixed number of terms, but with a 18.3% decrease in the number of terms/profile.

[0037] On AP, the maximum average precision is 0.2820 obtained with Rocchio and 180 terms. The new method achieves with RocchioFQ an average precision of 0.2861 with just 64 terms, an increase of 3.88% in the average precision and a reduction of 65% in the number of terms. The behavior is reasonable on WSJ, where in the case of an upper bound of 80 terms, the values are very close to the baseline ones, for Rocchio and Prob2. On FBIS, excepting RocchioFQ, it achieves a lower performance than the one obtained with a fixed number of terms. As we can see from FIG. 5, a setting of α=8 and Levels=3 achieves a very good behavior on Reuters, while the values α=8 and Levels=6 achieve a good behavior on the rest of the collections. α=8 and Levels=6 is not a combination that works well on Reuters and similarly, a setting of α=8 and Levels=3 would not work well on AP, WSJ and FBIS.

[0038] Comparing now the behavior of each method, in the case of an upper bound of 80 terms/profile, we note the followings. The new method works very well with Rocchio on Reuters, AP and WSJ. The worse behavior on FBIS is due to the fact that most of the times the number of selected terms is too large. With RocchioFQ, the behavior is very good on Reuters and AP, reasonable on FBIS, but not that good on WSJ. This is because on WSJ there are both profiles with too few terms and profiles with too many terms. With Prob2, the behavior is very good on Reuters, AP and WSJ, and reasonable on AP and FBIS, where again we take a too large number of terms. The values obtained when the curve of terms' weights has 300 terms are generally lower (excepting Reuters collection) than the ones obtained with a fixed number of terms. This is because we mostly select too many terms.

[0039] The second profile optimization technique has only two parameters: the percentage of the difference between the highest and smallest weights (perc) and the upper bound of the number of terms per profile, N. For these experiments, perc had the values 0.25, 0.1, 0.05, 0.01, while N had the values 80 and 300. As expected, the larger the value of perc, the more terms we choose.

[0040] FIG. 7 summarizes the results for each corpus. Similarly with the results reported for the first method, the first line in each table shows the maximum average precision and the corresponding number of terms, when all the profiles have the same number of terms. The second line shows the maximum average precision and the average number of terms/profile obtained with the second algorithm and the corresponding percentage. The third line (called Rec. from recommended) shows the values obtained with perc=0.05 for Rocchio and 0.01 for RocchioFQ and Prob2, settings that generally gave a good behavior on all the collections. Again, the values in bold letters correspond to the case when our algorithm achieves a better/very close average precision than the best average precision obtained with the corresponding term selection method, when all the profiles have the same number of terms.

[0041] On Reuters, the algorithm behaves very well, as the first algorithm also does. With Rocchio, we achieve an average precision of 0.5403 with 30 terms, an increase of 1.61% from the maximum average precision obtained with a fixed number of terms (Prob2, 0.5317, with 50 terms) and a 40% reduction in the number of terms. With an upper bound of 300 terms/profile, Rocchio achieves an average precision of 0.5428, with only 29 terms. For Rocchio, the new approach generally decreases the number of terms, while increasing/keeping the highest average precision obtained with a fixed number of terms per profile. Exception to this is Prob2 with an upper bound of 300 terms, where we take too many terms.

[0042] On AP and WSJ the behavior is again very good. On AP, the maximum average precision is 0.2820 obtained with Rocchio 180 terms. The algorithm achieves an average precision of 0.2802 with 131 terms, resulting in a 21% reduction in the number of terms. For all the methods, the new approach generally decreases the number of terms while keeping the same the maximum average precision obtained with a fixed number of terms. The cases when the maximum average precision is slightly lower, are due to the fact that we generally take too many terms.

[0043] On WSJ with an upper bound of 88 terms, the maximum average precision obtained with Rocchio is 0.2487 with 80 terms. The second algorithm achieves 0.2489 with just 49 terms, a 30% reduction in the number of terms. Similarly, for RocchioFQ, the maximum average precision is 0.2571, obtained with 80 terms. The new algorithm achieves 0.2540 with just 64 terms, a 20% reduction in the number of terms. For Prob2 with 300 terms, it achieves a slightly higher average precision, with a 25.3% reduction in the number of terms. On FBIS the behavior is very good for Prob2 with an upper bound of 300 terms, where we reach the maximum average precision with a 25.3% reduction in the number of terms and good for the other methods with both 80 and 300 terms. The situations where the average precision is slightly lower are again due to the fact that we take too many terms.

[0044] While this invention has been particularly described and illustrated with reference to a preferred embodiment, it will be understood by one of skill in the art that changes in the above description or illustrations may be made with respect to form or detail without departing from the spirit and scope of the invention.