Search quality detection

[0001] This invention relates to systems and methods of presenting search results. Specifically, this invention uses relevancy metrics associated with search results to categorize the search, and the presentation of the search results is altered accordingly.

[0002] Search engines are tools used to perform searches based on a query and return results or hits either as electronic documents or links to electronic documents that match the query. Due to the proliferation of information available in electronic sources, it has become increasingly important to avoid overwhelming a user with information likely to be irrelevant. Moreover, it is increasingly important to aid the user in navigating the relevant information more efficiently.

[0003] In service of these goals, most search engines contain an internal algorithm to determine which documents are most relevant to the search query. For example, some search engine algorithms consider both the frequency and the positioning of keywords in document to determine relevancy. If a keyword only appears once in a footnote to a document, there is a relatively small chance that the document contains the information a user is seeking by searching for the keyword. In contrast, if keywords appear repeatedly in a document and appear in the document title or headings, there is a relatively greater likelihood that the document contains information about the keywords that would be relevant, and hence useful, to the user. Whereas the first would result in a low relevancy number associated with the document, the latter would result in a higher relevancy number. Although general principles such as these are known in the art of search engine software engineering, the exact algorithms used by a search engine to determine a relevancy metric are usually proprietary information protected as trade secrets.

[0004] Once a search is run and relevancy is calculated for each result, the results are then sorted according to the relevancy metric, and the results list can be displayed in this order to the user. Since the most relevant results appear at the top of the list, users who begin scanning results encounter first those results most likely to contain useful information.

[0005] Some conventional search engine systems display a given number of search results, such as the top 100 results for a search, regardless of what search query is entered. If a search is too broad, the user may end up wasting time reading too many results in an attempt to find a real result of interest, or worse, the real result of interest might not have made the top 100 list. If a search query contains a spelling mistake, typographical error, or a terminology difference, the results returned may be entirely irrelevant to the user. With conventional search engine systems, the user must figure out how to change in the query in order to improve results. A higher quality search (i.e., one that is not too broad, too narrow, containing a spelling mistake, or the like) would help them locate desired information more efficiently.

[0006] Some search engines display the relevancy metric, which savvy users may use to interpret the quality of their results and modify their queries accordingly. An example of a search engine that displays relevancy metrics associated with each result is AnswerWorks® by Vantage Software Technologies. However, either from lack of familiarity with the relevancy metric or a failure to understand the significance of relevancy patterns in results, users may still not know how to improve their results or even that it would benefit them to try to do so.

[0007] Accordingly, systems and methods are needed to improve users' search experiences by helping them to more quickly locate desired results. There is a need for methods and systems that recognize certain indications of the nature or quality of the search query. There is also a need for systems and methods that aid users in improving their searches based on these determinations.

[0008] In various embodiments, the present invention provides methods and systems for categorizing the quality of the search by analyzing the relevancy numbers associated with the search results. Based on this categorization, the system alters how the search results are displayed, such as, for example, by adjusting the total number of search results to display or by providing feedback to the user regarding the quality of the search query or suggestions for altering it to improve the relevancy profile of the results.

[0009] In some embodiments, the system samples the relevancy numbers associated with the results. Then, the relevancy numbers are compared to established patterns to categorize the quality of the search query. Based on the determined quality category, the results display is changed accordingly. For example, the number of results to display, the message to display to the user, or in some embodiments a subsequent action the system executes, can all be based on the quality category. The display parameters and subsequent actions are fully customizable. For example, based on a determined category, the system may run a spell check on the query, open a new search dialog box, or the like.

[0017] The figures depict a preferred embodiment of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.

[0018] According to one embodiment, the present invention provides methods for categorizing the quality of a search. The present invention will be described in the context of an indexed help file included in a financial management software package such as QuickBooks® by Intuit, Inc. of Mountain View, Calif., that has been loaded onto a user's computer. The specific application discussed herein uses AnswerWorks® by Vantage Software Technologies, Newtown, Pa., a third party search engine that by default returns the top 100 results listed in order of relevancy, each with a relevancy number, in response to a user entered search string. It will be readily understood by those ordinarily skilled in the art that the software products, methods and systems described could be applied in other contexts, including web-based help files and general purpose databases accessible over the internet without departing from the teachings disclosed herein. As those skilled in the art will also appreciate, the invention can be applied in conjunction with other search engines configured to calculate and output any scoring system for assessing results in response any type of query, also without departing from the teachings of this disclosure.

Examples of Relevancy Curves for Searches

[0019] In this section the derivation of relevancy curves will be described. Referring to FIG. 1, there is shown four examples of queries and the top 10 of the top 100 results by relevancy. In these examples, the relevancy metric is an integer between 0 and 100, where 100 is the maximum obtainable relevancy. These examples have been chosen to illustrate common patterns in the curve or profile of the relevancy numbers that occur for each of the given situations. These patterns are the basis for differentiating between categories of searches described in the next section.

[0020] In Example 1, the search string “check” has been entered into the search engine. The top ten results and associated relevancy numbers for this example demonstrate a common pattern that emerges when the query is too broad. Because many topics in the help file of a financial management software application discuss checks, the relevancy numbers of all top ten results are high. Because the query is so broad, the context for the use of the word “check” is missing. Note that some results, such as “Checking the Status of your entire inventory” and “Checking your inventory status” are likely to be irrelevant to “check” in the sense of a written order directing a bank to pay money. Thus, the user may waste time reviewing them all without realizing that a more specific query would be helpful to find the desired material.

[0021] In Example 2, the search string “printing a check” has been entered. This time, the top ten results and associated relevancy numbers demonstrate a pattern that emerges for good searches. The relevancy numbers of the top ten results starts high and tapers off. With the added words “printing a” to “check”, the search engine no longer returns the “Checking the Status of your entire inventory” and “Checking your inventory status” results within the top ten. Thus, the search engine has more successfully identified a small number of results with a high likelihood of having useful information.

[0022] In Example 3, the search string “printing a chekc” has been entered. Because the last word in the search string contains a typographical error, the top ten results and associated relevancy numbers demonstrate a pattern that emerges for searches containing a spelling mistake or terminology difference. The relevancy numbers of the top ten results are uniformly quite low. None of these results are likely to contain the information the user wanted on printing a check.

[0023] In Example 4, the search string “how do I void a check?” has been entered. The top ten results and associated relevancy numbers demonstrate a pattern that emerges for excellent searches. The relevancy number for the top result is very high, followed by a sharp, tapering drop off for the rest of the top ten results. The search engine has identified an even smaller number of results that have a high probability of containing relevant information, in this case, one result.

[0024] Referring to FIG. 2, the relevancy number is plotted for each of the first ten search results (i.e., the top ten search results in order of relevancy) for each of the Examples 1-4 from FIG. 1. On FIG. 2, line segments connect the points of these plots in order to more clearly demonstrate the change in relevancy over the first ten results in each series. Thus, the patterns discussed above from the tables in Examples 1-4 are shown in graphical form in FIG. 2.

Categorizing Searches

[0025] In one implementation, the present invention analyzes the relevancy profile of the search results to categorize the search query, for example as being excellent, good, likely to contain a spelling mistake or terminology difference, or too broad/general. Examples 1-4 were provided to show the types of patterns that appear in the relevancy numbers, thus showing the cases to differentiate by using appropriate criteria in the categorization algorithm. The categorization algorithm determines which category is most appropriate for the search based on the relevancy numbers returned for the results.

[0026] Categories for searches are defined by criteria that the search results must satisfy. In this implementation, a criterion is a range of relevancy numbers for a given result in the sample. Categories are determined by evaluating the relevancy numbers of the relevant search results against the category criteria. For instance, for one example category, the criterion could be that the top result must have a relevancy number of at least 95 out of 100. Thus, the relevancy number of the top result of a search would be evaluated using this criterion to determine whether the search belongs in this category. Categories can also be defined by multiple criteria. An additional criterion to the example category could be that the second result has a relevancy number less than 95. In some embodiments, criteria such as these are established to evaluate the relevancy number of each result in the entire result set returned for the search. In others, criteria are only established for a sample of results. The advantage of limiting the number of results for which category criteria are applied is a reduction in processing time needed to categorize the search according to the algorithm.

[0027] Referring to FIG. 6 for illustration purposes, suppose 4 categories are defined. Category 1 searches return excellent results. Category 2 searches return good results. Category 3 searches return too many results with high relevancy values, indicating an imprecise search. Category 4 searches return results with relevancy numbers that are too low, indicating a terminology difference or spelling mistake in the search query.

[0028] An example of a categorization algorithm that would divide searches into the above categories contains the following conditions. In this example, because the criteria for the categories are not mutually exclusive, the relevancy numbers of the search results are evaluated against the set of conditions for one category before moving to the next set of conditions for the next category. Once a category is assigned for the search, the algorithm ends.

• • [0029] If first result's relevancy is equal to 100 and the third result's relevancy is less than 95, then this is a Category 1 search string.
  • [0030] Otherwise, if the first result's relevancy is greater than 95 and the tenth result's relevancy is less than 60, then this is a Category 2 search string.
  • [0031] Otherwise, if the first result's relevancy is greater than 95 and the tenth result's relevancy is greater than 85 then this is a Category 3 search string.
  • [0032] Otherwise, if the first result's relevancy is less than 60 then this is a Category 4 search string.

[0033] In some embodiments, a catch-all or default category is assigned at the end of the algorithm such that the search is assigned to that category if the search has not been assigned up to that point.

[0034] Categories are fully customizable through altering the criteria used to evaluate to the relevancy numbers of the results to make the categorization. Criteria are selected for categories that make the categories meaningful in the particular application of the invention.

[0035] FIG. 3 illustrates an example wherein a sample size of only two results, in this case, the top two results, is used to categorize a search. Using an appropriate categorization algorithm, categories corresponding to the same four categories as in the above example can be defined: Category 1 search strings return excellent results. Category 2 search strings return good results. Category 3 search strings return too many results with high relevancy values, indicating an imprecise search. Category 4 search strings return results with relevancy numbers that are too low, indicating a terminology difference or spelling mistake in the search string. An example algorithm that makes this categorization is as follows. Again, because the criteria for the categories are not mutually exclusive, the relevancy numbers of the search results are evaluated against the set of conditions for one category before moving to the next set of conditions for the next category. Once a category is assigned for the search, the algorithm ends.

• • [0036] If the second result's relevancy is equal to 100, then this is a Category 3 search string.
  • [0037] Otherwise, if the second result's relevancy is greater than 95, then this is a Category 2 search string.
  • [0038] Otherwise, if the first result's relevancy is greater than 95 and the second result's relevancy is less than 60, then this is a, Category 1 search string.
  • [0039] Otherwise, if the first result's relevancy is less than 60 then this is a Category 4 search string.

[0040] As in the previous example, in some embodiments, a catch-all or default category is assigned at the end of the algorithm such that the search is assigned to the default category if the search has not been assigned up to that point.

[0041] In an alternative embodiment, categories are determined by comparing the relevancy plot for a search to one or more archetypical patterns for relevancy plots. Each archetypical pattern corresponds to a category. FIG. 4 illustrates a set of archetypical patterns, one showing the pattern for a search that is too broad, one with a spelling mistake or terminology difference, one good search, and one excellent search. In some implementations, the category for a search is determined by calculating which of the archetypical patterns is the best fit to the data points of the relevancy plot for the search. The best-fit calculation may be accomplished by finding the sum of the absolute differences between the archetype curve and the relevancy curve for each archetype curve. In other implementations, the best fit is determined by assigning proportionally more weight in the best fit analysis to a subset of the data points, such as the relevancy numbers of the first 5 results. One of ordinary skill in the art will recognize many other methods of calculating which archetype is the closest match are possible. Referring to FIG. 5, there is shown the relevancy plot of one of the example search strings with the four archetypical patterns plotted on the same graph. As can be seen from FIG. 5, the plot of the example search string “how do I void a check?” most closely matches the archetypical curve for an excellent search. Thus, the example search is categorized as a Category 1 (Excellent) search.

Using the Categorization

[0042] Once a search has been categorized, the category can be used to determine outputs such as display characteristics of the search results or subsequent actions. FIG. 6 is a table summarizing an example set of display characteristics for the four categories of the above examples including the number of results displayed and the suggestions displayed to the user. In some embodiments, the number of results displayed and the suggestions displayed are tailored to the quality of the search in some embodiments. The following describes the example in FIG. 6:

[0043] For category 1 (Excellent Results) searches, 7 results are displayed. For category 1 searches, there is a high degree of probability that the information a user desires is contained in one of the first few hits because the relevancy numbers start high and taper off quickly.

[0044] For category 2 (Good Results) searches, a few more results (9 results) are displayed. The relevancy numbers do not taper off as quickly, so to maintain the confidence that the information a user desires is contained in one of the displayed results, the number of displayed results is slightly increased.

[0045] For category 3 (Too Many Hits) searches, slightly more results (10 results) are displayed along with suggestions to make the search more specific or rephrase the request as a simple question. Because the relevancy numbers for category 3 searches are high throughout the first ten results, the information a user desires may or may not be contained in any of the first ten results. Only displaying 10 results encourages the user to follow the suggestion of making the search more specific or rephrasing instead of potentially wasting time reviewing many results that have a lower probability of containing the desired information.

[0046] Lastly, for category 4 (Relevancy Too Low) searches, only 3 results are displayed along with suggestions to check spelling or use different terminology. Because the relevancy numbers for the search results are low throughout, there is a low probability that the top results returned for the search contain the information the user desires. Thus, only displaying the top 3 results encourages the user to follow the suggestion of checking spelling or using different terminology to obtain better results, rather than wasting time reviewing many results with low relevancy scores. FIG. 7 illustrates how the search results and suggestions are displayed to the user for a category 4 search in one embodiment.

[0047] In other embodiments, the search category determination may trigger other actions based on the determination, such as displaying or otherwise outputting the category; generating feedback for tracking purposes; displaying a message associated with the category, such as suggestions for altering the search string, or technical support information or phone numbers; formatting the output such as by including graphics, altering the font, or the like; playing a category-specific sound to alert the user; instigating a spelling check on the entered search string if it is determined that a search string is likely to be misspelled; opening a new search window or a help window; performing searches on different data sources; or the like. In addition, the search category determination may trigger the display of targeted advertisements or content. For example, advertisements for instructional manuals or classes may be displayed if it is determined from the search category that the user may not be using the correct terminology or is otherwise receiving low quality results.

[0048] The present invention has been described in particular detail with respect to one possible embodiment. Those of skill in the art will appreciate that the invention may be practiced in other embodiments. First, the particular naming of the components, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. Further, the system may be implemented via a combination of hardware and software, as described, or entirely in hardware elements. Also, the particular division of functionality between the various system components described herein is merely exemplary, and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead performed by a single component.

[0049] Some portions of above description present the features of the present invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described functionally or logically, are understood to be implemented by computer programs. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules or by functional names, without loss of generality.

[0050] Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0051] Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.

[0052] The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

[0053] The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent to those of skill in the, along with equivalent variations. In addition, the present invention is not described with reference to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references to specific languages are provided for invention of enablement and best mode of the present invention.

[0054] The present invention is well suited to a wide variety of computer network systems over numerous topologies. Within this field, the configuration and management of large networks comprise storage devices and computers that are communicatively coupled to dissimilar computers and storage devices over a network, such as the Internet.

[0055] Finally, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.