System and method for multi-lingual information retrieval

[0001] This application is a continuation of International Application No. PCT/US07/11292, filed May 9, 2007, which claims the priority of U.S. Provisional Application No. 60/800,059, filed May 12, 2006, the entire contents of each of which are incorporated herein by reference.

[0002] This disclosure relates to the area of information retrieval and/or search. More specifically, this disclosure relates to multi-lingual information retrieval which is a subfield of information retrieval (IR).

[0003] Cross-language information retrieval (CLIR) deals with providing a query in one language, and searching document collections in one or more different languages. For example, a user may pose a query in Chinese but retrieve relevant documents originally written in English. Cross language information retrieval is also called multi-lingual or trans-lingual information retrieval.

[0004] In this era of information explosion, especially with the advent of the World Wide Web (WWW or Web) in which anyone can create his or her own website (for example, blogs), how to find the information a user seeks from vast amounts of available information remains a challenging task, and how to find the information the user seeks even if it is written in a different language is even more daunting. In many instances, the most relevant information is written in a foreign language. However, a language barrier may prevent the user from retrieving such documents using conventional IR tools. For example, if the user sends a query “Iraq war” in English, the conventional IR system does not obtain articles about “Iraq war” written in Chinese, such as are available at http://141.155.90.70:88/files/articles/Iraq.htm, and therefore does not present the specific views communicated in Chinese regarding the Iraq war. On the other hand, if the user sends a query in Chinese  the system does not obtain articles written in English, such as at http://www time.com/time/time100/leaders/profile/mao.html, and therefore does not present the specific views communicated in English regarding Mao Zedong.

[0005] In conventional techniques of performing mono-lingual search (information retrieval), the user specifies a set of words, phrases or sentence (individually and/or collectively hereinafter “term”) which convey the semantics of the information sought, in a frame, which is also called a query, and the query is submitted by, for example, pressing a “Search” button nearby. The conventional system searches for documents related to this query in the target document set (for example, all or a subset of documents available on the Web), in as complete a manner as possible, then may rank them according to the degree of relevance to the query, and displays the results of the search in ranked order. The primary goal of a conventional IR system is to find as many of the documents that are relevant to the user query (recall) as possible while retrieving as few non-relevant documents as possible (precision). A conventional IR system is shown in FIG. 1.

[0006] There are many types of documents on the Web, and the documents are represented in different languages. There are documents of different formats (for example, Html, Doc, PDF), and there are images with captions in different languages. A single query preferably triggers search of all such resources.

[0007] Queries are typically processed so that the IR system can utilize them to perform appropriate searches. If an IR system can translate a query to another language, the system can search the document set for relevant documents in the other language. Similarly, if the IR system can translate the query to another form, then the IR system can augment the search. Often, interactions between the user and the machine are needed to ensure a comprehensive search.

[0008] A number of technologies have been proposed for dealing with how to represent the documents inside the computer. In addition, there are many other IR techniques dealing with query processing, indexing, ranking, etc. For example, in one conventional technology, documents in a collection are represented through a set of index terms or keywords. Such keywords might be extracted directly from the text of the document or might be specified by a human operator such as frequently performed in the library science. An example of an indexing approach is shown in FIG. 2.

[0009] In cross-lingual search, one or more language translations are performed, for example, translate a query from source language into target language and then perform mono-lingual search using the translated query, or translate the documents from target language into source language and then perform mono-lingual search using the original query. It has also been proposed to translate both query and documents to a certain intermediate representation, so that the two can be compared. The table in FIG. 3 shows a brief summary of current approaches for CLIR.

[0010] U.S. Pat. No. 5,301,109 entitled “Computerized cross-language document retrieval using latent semantic indexing” proposes a corpus-based, intermediate representation approach to CLIR. U.S. Pat. No. 5,867,811 entitled “Method, an apparatus, a system, a storage device, and a computer readable medium using a bilingual database including aligned corpora” also proposes a corpus-based approach.

[0011] U.S. Pat. No. 6,321,191 entitled “Related sentence retrieval system having a plurality of cross-lingual retrieving units that pairs similar sentences based on extracted independent words”, proposes a technique for retrieving related sentences out of n cross-lingual retrieval systems. Each of the n systems comprises a pair data storing unit that stores a multiple of pair data (two languages) having the same meaning.

[0012] Both monolingual information retrieval and cross-lingual information retrieval face the difficulty of understanding the user's true intention when the user queries in natural language. Information retrieval is not data retrieval, which consists mainly of determining which documents in a collection contain the keywords in the user query. The user of an IR system is concerned more with retrieving information about a subject than with retrieving data which matches a specified query. The user simply expresses the information sought in natural language. So an IR system ideally embodies some understanding of the natural language. For example, if the user queries  documents containing  but not containing  or  may not be displayed to the user.

[0013] To ensure the completeness of the search result, queries can be processed into a group of synonyms, for example, (cell phone, mobile phone, cellular phone,   Bush), etc.

[0014] The technique of using a thesaurus (where groups of synonyms are stored) in CLIR was proposed by G. Salton, “Automatic Processing of Foreign Language Documents,” (1970) Journal of American Society for Information Sciences. Salton reported experimenting with a method for automatic retrieval of documents in one language in response to queries in another using a vector representation and search technique in conjunction with a manually created dual-language thesaurus. The results for test samples of abstracts and queries were promising. However, creating an adequate multi-language thesaurus is difficult and requires considerable intellectual labor.

[0015] The following example explains why it is desirable to expand one query into a group of synonyms. To search for documents regarding Cross-lingual Information Retrieval, multiple synonyms, such as “Trans-lingual Information Retrieval”, “Multi-lingual Information Retrieval”, may be substituted in the search, and “Information Retrieval” may be substituted with “Search” or “communication”, and “-lingual” maybe substitute with “Language”. The phrase in this example can be expanded to at least 12 synonyms or related terms. In addition, the search may be guided by specification of the relevant field of technology such as “search engine”, “machine translation”, etc.

[0016] For example, a user may query the Spanish term “conjeturar sin fundamento” in a search engine, and may retrieve results containing or related to the exact query terms. However, in order to search for documents regarding “conjeturar sin fundamento” and find as many of the documents that are relevant to the user query as possible, multiple synonyms, such as “adivinar a ciegas”, “hacer suposiciones gratuitas”, may be substituted in the search. But how to automatically generate a set of synonyms to trigger multiple searches based on one query term remains a challenging task.

[0017] The Software Department of the Institute of Computing Technology in China developed a Question and Answering Search Engine System about Tourism in China, and used a thesaurus to expand the user query into a multiple of synonyms or related words. However, the thesaurus was manually developed and maintained by human information specialists.

[0018] Dictionary based approaches generally have a major problem of OOVs (Out of Vocabulary words), such as person names, company/organization names and place names, brand names, etc. Conventional CLIR approaches based on static dictionary cannot overcome this difficulty. In addition, a thesaurus may expand the user query into a multiple of synonyms or related words but cannot retrieve as few non-relevant documents as possible, because it cannot specify the relevant context for the search to narrow the scope of the retrieved results.

[0019] U.S. Pat. No. 6,604,101, entitled “Method and system for translingual translation of query and search and retrieval of multilingual information on a computer network”, proposes a “restricted/controlled query” method in which after the user inputs a query in the source language, it is standardized or regulated in a “dialectal controller”, and if no standardized form of the user's query is found, the user is prompted to describe the information sought in another way, and then the standardized query words are translated into target language terms, which are used to search the target-language document set. U.S. Pat. No. 6,604,101 does not disclose or suggest a multilingual, dynamically evolving dictionary which stores synonyms or related words, or similar sayings.

[0020] U.S. Patent Application Publication No. 20040139107A1, entitled “Dynamically updating a search engine's knowledge and process database by tracking and saving user interactions”, proposes supplementing a query with information from tracking user interactions and saving the information. However, U.S. Patent Application Publication No. 20040139107 A1 does not disclose or suggest updating a multi-lingual knowledge base according to voting by multi-lingual Web users.

[0021] In U.S. Patent Application Publication No. 20040139106 A1, entitled “Search engine with natural language-based robust parsing of user query and relevance feedback learning”, proposes an approach to accommodate the user through interaction and feedback to and from the user. However, U.S. Patent Application Publication No. 20040139106 A1 does not disclose or suggest a multi-lingual knowledge base according to voting by multi-lingual Web users.

[0022] U.S. Pat. No. 5,384,701, entitled “Language translation system”, proposes a system for translating phrases from a first language into a second language. The system includes a store holding a collection of phrases in the second language. The phrases in the second language are prepared in advance and held in the store. For example,  was stored as “How do you do?” However, U.S. Pat. No. 5,384,701 does not disclose or suggest that the knowledge base can be dynamically updated through contribution of Web users.

[0023] There remains a need for improvements to cross-language information retrieval techniques.

[0024] An approach for automatically expanding queries and increasing probabilities of retrieving relevant documents through multiple versions of the queried documents and/or the corresponding translated results is introduced in this disclosure.

[0025] In an aspect of this disclosure, a search query in a source language is translated into multiple search queries in a target language (for example, specified by language code and/or tag). In addition, the retrieved target language documents can be translated back into the source language, with multiple translation results. Thus not only the relevant documents that match the search queries can be retrieved and displayed to the user, but also the documents that match the synonyms, translation options and/or related terms of the queries can be retrieved and displayed to the user.

[0026] This disclosure also provides an approach for dynamically upgrading a thesaurus and/or a dictionary in an information retrieval system, via the user's vote.

[0027] In an aspect of this disclosure, a vote interface is provided to allow a user to edit, store and/or vote in favor of or against certain translation options in connection with relevant contexts. By accumulating such votes, a multilingual vote store is formed that evolves with time, and is dynamically updated in accordance with user interaction, contribution and/or editing. The vote store can be public and accessible by anyone who is interested. In addition, a voter can enter related terms in order to specify domain, area and/or context, such related terms can also be stored in the vote store (for example, as part of the translation option).

[0028] This disclosure also provides a self-learning translation engine which translates a document upon request, and dynamically updates itself by reference to evolving multi-lingual knowledge stored in the vote store. For example, the document can be parsed into sentences and/or other natural language units, such as phrases, keywords, colloquial expressions, etc., and the source language units can be saved or registered along with their translations, to further enhance the multi-lingual information base. Thus the translation engine can evolve with time and natural language trend.

[0029] Further, this disclosure also provides an approach for organizing and maintaining a multi-lingual document base according to an evolving multi-lingual knowledge base and automatically generating multiple versions of a document in connection with relevant contexts. Thus one document becomes a plurality of similar documents to enhance the probabilities of retrieving relevant results in the multilingual search as well as monolingual search.

[0030] Accordingly, another aspect of this disclosure is that the translation engine can perform context-sensitive translation of documents.

[0037] The subject matter of this disclosure embodies the recognition that natural language is constantly evolving with time, and varies with geographical region. For example, while “mobile phone” or “cellular phone” were the standard terms used when the use of such phones was more limited, such phones are now widely used and are commonly referred to as “cell phones” or “smart phones” today. In another example, a young girl may have been called “miss” or “young lady” years ago, but now is more often called “chick” in online chatting environment. On the other hand, in the Chinese language young girls were called  in the last century with a political tone, but now in China they are referred to in more colorful terms such as  etc. As another example, when the mainland Chinese say  the Taiwanese say  whereas the Hongkongers say  New words, new entity names, person/organization names, etc., are emerging every day.

[0038] So, the ability to dynamically update the knowledge stored in a dictionary or thesaurus for use in the IR system is one of the aspects of this disclosure. In an exemplary embodiment, a dynamic multi-lingual database is formed which stores words, phrases, and/or sentences, and is updated by, for example, public voting from the world-wide multilingual users (that is, one who speaks, or otherwise communicates using, more than one languages). An example of a user interface screen showing a view of the general vote store is shown in FIG. 5B.

[0039] For example, any multilingual user can opine as to how a certain word, phrase or sentence is to be translated into a target language according to his/her own interpretation, as one of the many translation options. In a specific context, a phrase in Spanish “conjeturar sin fundamento” may be translated from the source language Spanish into the target language Chinese as  Another user may vote that the same phrase “conjeturar sin fundamento” should be translated into the target language Chinese as   In another context, the same phrase “conjeturar sin fundamento” may be translated into the target language Chinese as  In another context, the same phrase “conjeturar sin fundamento” may be translated into the target language Chinese as

[0040] The users may enter votes through a vote interface (see example of user interface screen in FIG. 5A), and may edit (see example of user interface screen in FIG. 5F), store and/or vote in favor of, or against, certain translation options in connection with relevant contexts, forming a multilingual vote store that evolves with time, and that is dynamically updated by users' interaction, contribution and/or editing. The user may also specify a context for the Web page where he/she inserted the vote by adding related word, phrase and/or sentence (individually and/or collectively hereinafter “related term”) to describe certain translation options, and thus let the machine register information indicative of relevant context of the Web page. For example, the user may add “Hongkongers” as a related term to describe the translation option “Bush” to  in certain contexts. The user may further specify a language code, in connection with a document address and/or a Uniform Resource Identifier (URI) on the World Wide Web, to retrieve a corresponding document and/or Web page in the specified language as indicated by language code or tag associated with the document or Web page. While a country code, such as .us, .uk, .cn, etc., can be specified in connection with a document address to indicate a country-specific resource address, the language code or tag specified by the user in connection with the document address and/or the URI can specify the language of a document to be retrieved from the multi-lingual base saved in the local server, and such aspect introduces a convenient way of searching, retrieving and displaying a document and/or a Web page in the specified language.

[0041] Each user can opine as to how a certain word/phrase/sentence is to be translated, including specifying a translation of a certain word/phrase/sentence and/or selecting one of a plurality of presented choices of translations. Further, the user can vote in favor of, or against, an existing translation. In addition, when ranking and displaying matched documents returned by a search, the user can choose which term used is more comfortable to him/her, and/or specify a relevant context and/or a language of documents to be retrieved.

[0042] The translation of a term can vary from person to person. For example, Taiwanese typically feels comfortable with translation of “President Bush” to  whereas the mainland Chinese are used to seeing “President Bush” translated to   and for a person living in Hong Kong “President Bush” is generally translated to  

[0043] Using the tools of this disclosure, each user can access documents regardless whether the document, as originally stored, is in English, Spanish, Chinese, Japanese, German, French, etc. The user can express his/her view (such as via the user's web page, a blog, etc.) in specific language and/or dialect, and can share knowledge, vocabulary, way of saying things, etc., with the rest of the world. On the other hand, the user can retrieve documents and resources from an information/content sources, such as the World Wide Web, in a specified language, even when the document, as originally stored, was in another language. The tools of this disclosure can constitute a channel through which anyone can post his views, vocabulary, etc., on the Internet, and retrieve the documents in a specified language.

[0044] For example, if a user works for the non-profit organization  the user can specify that the organization's name be translated into “Asian-American Coalition for Education”. The search engine can use “Asian-American Coalition for Education” to query English documents, or the translation engine can use the Chinese-English pair to perform document translation, and the Chinese name is not translated into poor English, for example, “Asia Mode school”.

[0045] The interpretation of a document may vary from person to person. A person may interpret “Can you give me a hand?” as “Can you assist me to do this?”, whereas another person may interpret it as “Can you help me?”. Using multi-lingual knowledge base, the user may automatically generate multiple versions of one document by translating the document into different versions and translating them back. Thus the user may query via a search engine and retrieve multiple versions derived from one document.

[0046] For example, a document containing “cheap airplane tickets sold at the lowest price” may be translated and interpreted in different ways. According to translation options added to a multi-lingual vote store, the term may be translated into various Chinese terms         etc., and may be translated into even more diversified versions in other Web pages when additional translation options are inserted into the multi-lingual vote store. At the same time, the users may also insert translation options for translating Chinese to English, voting  as “discounted” or “inexpensive”,  as “airline ticket” or “plane ticket”, and  as “dumping” or “on sale” in various contexts. Thus using multi-lingual knowledge base and combining different translation options in dynamically evolving multi-lingual vote store, a document containing “cheap airplane tickets sold at the lowest price” may automatically be translated back into the source language English and generate multiple results such as “cheap airplane tickets on sale”, “inexpensive plane tickets sold at the lowest price”, “discounted airplane tickets on sale”, or “dumping airline tickets” and maybe more versions according to the current votes. The multiple versions derived from the document may be stored and then retrieved by users who query in search engines using various search queries, such as “discounted tickets”, “inexpensive tickets”, “plane tickets lowest price”, “airplane tickets on sale”, “dumping plane tickets”, etc., and thus to find as many of the documents that are relevant to the user query as possible.

[0047] Further, a user may retrieve documents through specification of query terms and/or related terms to narrow the scope of the search results. A user may select one or more synonyms in translation options to specify the search query, and/or select one or more related terms to specify the relevant context, and then display the search results according to the specification of query term and/or relevant context. For example, the user may select the translation option  and/or the related term “Hongkongers”, and then display relevant search results in connection with the specified search query and/or related term.

[0048] Another aspect is providing multiple translation results for search queries and retrieved documents. A search query in the source language is translated into a set of multiple search queries in the target language, with which to query the target language document set. The retrieved target language documents are translated back into the source language, with multiple translation results. Meanwhile, a search query in the source language can be used to query the source language document set, then the retrieved source language documents can be translated into a plurality of translated documents in the target language, with multiple translation results. During the translation procedure, when multiple translation options are available for words, phrases, sentences or expressions, the option with the most votes in the vote store is selected, whereas the other options are kept in the translated documents in the form of meta-data and/or hidden fields. For example, a search query in Spanish “conjeturar sin fundamento” may be translated into a set of multiple search queries in the target language Chinese such as    and/or  The retrieved target language documents then are translated back into the source language Spanish, with multiple translation results such as “conjeturar sin fundamento”, “adivinar a ciegas”, “hacer suposiciones gratlitas”, etc. During the translation procedure, the option with the most votes “conjeturar sin fundamento” may be selected, whereas the other options “hacer suposiciones gratuitas” and “adivinar a ciegas” are kept in the translated documents in the form of meta-data and/or hidden fields.

[0049] An additional aspect is allowing public voting in the general vote store. An example of a user interface screen for such purpose is shown in FIG. 5C. The general vote store is preferably accessible to anyone through the Internet (see, for example, user interface screen shown in FIG. 5B). Each user can view and update it. When a person suggests that a word, phrase, sentence or expression X in the source language be translated into the word, phrase, sentence or expression Y in the target language, the vote for the relationship X→Y (called translation option) is incremented by 1. The more people voted for the relation X→Y, the higher the possibility that X is translated into Y in the resultant documents. When providing the suggestion, the person can also enter some related terms to indicate the domain, area and/or context of this relationship, and the related words, phrases or sentences are also stored in the vote store, in connection with a particular translation option. For instance, “Jerry Yang” may be voted to be translated into  with the related term voted as Yahoo and/or Co-founder. “Bush” may be voted to be translated into  with the related term voted as president and/or “Hongkongers” in connection with relevant contexts.

[0050] Another aspect is a general translation engine which is provided to translate any document upon a request. FIG. 5D shows a user interface screen for such purpose. The translation engine can be updated dynamically through self-learning of multi-lingual knowledge stored in the general vote store. Based on the public voting in the general vote store, the general translation engine can self-learn constantly evolving natural language knowledge, absorb new translation options into its database, dictionary or thesaurus, and thus dynamically update itself to keep up-to-date with respect to current changes in the natural language trend. For example, according to a new Chinese to English translation option in the general vote store, translation of  to “hot chick” may be absorbed into the general translation engine's dictionary, and used to translate any document upon a request.

[0051] In another aspect, public voting is allowed in the special vote store for a specific context. The special vote store is related to a specific context and accessible to anyone through the Internet, (see, for example, user interface screen shown in FIG. 5A). Anyone can view and update the special vote store in connection with a specific context (for example, a Web page, a cluster of related Web pages, a category of document, a URI and/or document address, a Web domain name, etc.; see, for example, user interface screen shown in FIG. 5D). The user can be prompted to select special vote store or general vote store (see, for example, user interface screen shown in FIG. 5E). Alternatively, the selection can be automated based on whether a context is presented (for example, when term to be translated is presented in a document, a specified sentence, paragraph, which is related to a Web page, a cluster of related Web pages, a category of document, a URI and/or document address, a Web domain name, etc.).

[0052] A user may add new translation options into the special vote store, and may edit, store and/or vote in favor of, or against, existing translation options in the special vote store in connection with a specific context. Thus when a person suggests that a word, phrase, sentence or expression X in the source language be translated into the word, phrase, sentence or expression Y in the target language in a specific context, the vote for the relationship X→Y (called translation option) in the special vote store in connection with the specific context is incremented by 1. In the translation procedure, when multiple translation options are available for words, phrases, sentences or expressions in the general vote store as well as in the special vote store, the option with the most votes in the special vote store is automatically selected as the default translation option, whereas the other options are kept in the translated documents in the form of meta-data and/or hidden fields. For example, a Hong Kong Web page or a group of Hong Kong Web sites containing the term “president Bush”, the translation option “Bush” to  with the most votes in the special vote store in connection with the Hong Kong Web sites is automatically selected as the default option during the translation procedure, while the other options “Bush” to   and “Bush” to  in the general vote store are kept in the translated documents in the form of meta-data and/or hidden fields.

[0053] Another aspect is the special translation engine which is the translation engine for translating documents in a specific context upon a request. Such translation engine can be trained to improve itself dynamically by performing self-learning of multi-lingual knowledge stored in the special vote store. Based on the public voting in the special vote store, the special translation engine can self-learn constantly evolving natural language knowledge, absorb new translation options into its database, dictionary or thesaurus, and thus dynamically update itself to keep up-to-date with respect to the current changes in the natural language trend in a specific context. For example, in a new Chinese to English translation option in the special vote store,  to “Tung Chee-hwa” may be absorbed into the special translation engine's dictionary, and used to translate documents in a group of Hong Kong Web sites upon a request.

[0054] The translation is preferably context sensitive. The context can be represented in a multitude of forms, including but not limited to Web based environments, documents categories, Web links, etc. For example, the Web link and/or a group of Web links (such as Web links under the same domain name, same sub-domain name, etc.) may form a “context” for the processing of natural language by a computer. When a vote is received through a vote interface in a specific Web link, a special vote store can be automatically formed based on the Web link or context in connection with the related terms used to describe the vote, and at the same time the vote is inserted into the general vote store regardless of the Web link or context. The votes in the special vote store are counted independently by the computer. In the translation procedure in a specific context, in a case that the number of votes for a translation (for example “Jerry Yang” to  stored in a special vote store differs from that in the general vote store, the votes in the special vote store prevail, and when multiple translation options are available for words, phrases, sentences or expressions in the general vote store as well as in the special vote store, the option with the most votes in the special vote store is automatically selected as the default translation option for the specific context.

[0055] The computer can record and store the Web based environments, documents categories and/or the Web links through which votes are inserted in connection with related terms that describe the votes. Thus the context of a new document can be determined based on the Web-based environments, document categories, Web link, etc., through which the document is submitted for the translation procedure.

[0056] In the translation procedure, the special vote store and the special translation engine are applied corresponding to a group of similar Web links determined by a domain name, a sub-domain name and/or a cluster of domain names. For example, when Web link 1 and Web link 2 are under the same domain name, the special vote store and the special translation engine created based on Web link 1 can be applied to a translation request for a new document submitted through Web link 2. Further, the related terms used to describe the votes can be used to describe the Web links or context in which the votes are inserted. Thus, a context can be represented in a computer by the related terms.

[0057] In addition, when the user specifies a language code in connection with the document address and/or the URI to retrieve the relevant document and/or Web page with corresponding language tag, the translation applied to the document and/or Web page is performed in the translation procedure in accordance with the corresponding contexts (for example, category of document, URI and/or document address, Web domain name, cluster of related Web pages, etc.; see, for example, user interface screen shown in FIG. 5D) to achieve specific translation results. The user may search, retrieve and display the translated documents and/or Web pages in a specific language in corresponding contexts, when the user specifies a language code in connection with the document address and/or the URI, to specify which directs to the translated documents and/or Web pages with corresponding language tag (see, for example, user interface screen shown in FIG. 5H).

[0058] Another aspect of this disclosure is a dynamic translation database. The vote store is updated constantly by users, and thus it is a dynamic translation database which contains the most up-to-date words, phrases, sentences or expressions in natural languages. The vote store differs from and also complements traditional dictionaries in various ways. For example, the database is interactive, and the translation database interacts with users to allow users to update the database in real time and search and view the updates in real time. In addition, the database is more comprehensive than traditional dictionary, and it collects unique expressions, product names, names of places, names of people, etc, which are not included in the traditional dictionary but are often used by searchers in a search query. Further, the database is dynamic, and changes and evolves with user's options and popularity of the translation of words, phrases, sentences, expressions, etc., whereas traditional dictionaries are fixed once they are published and are only changed after certain period of time (for example, several months or even years), and thus multi-lingual search with such dictionaries is not comprehensive. On the other hand, the multi-lingual base of this disclosure can contain up-to-date information, without being delayed by compiling a traditional dictionary, and votes are accumulated from the users, and thus it is able, in a translation, to take into account a single user's vote on up-to-date words, phrases, sentences, expressions, etc. In contrast, traditional dictionaries used by machine translation engines lack this ability, and the translation engines utilizing such dictionaries cannot recognize such words, phrases, sentences, expressions, etc., in order to translate properly.

[0059] In addition, automatic bi-directional translation option in public voting is another aspect of this disclosure. The vote for any translation option is translated back automatically. For example, the relationship between X and Y is stored as a bi-directional translation option, i.e. when the vote for the relationship X→Y is incremented by 1, the vote for the reverse relationship Y→X is also incremented by 1 automatically. Accordingly, the vote store can be used for translation from the source language to the target language and for translation from the target language to the source language, thus increasing the richness of translation option. For example, a user may register, or vote for, the translation option from English “hot chick” to Chinese  in the multi-lingual vote store, and the vote for the reverse relationship from Chinese  to English “hot chick” is incremented by 1 automatically.

[0060] In another aspect, the translated documents in the target language are downloaded from the Internet and saved or registered locally in a set of local servers. The documents in the source language are often stored in remote servers located in other places (and even other continents) and they can be difficult (when considered on a large scale) to retrieve due to long-distance transmission through a multitude of routers. Therefore, before retrieval, these documents are already translated into the target language and saved or registered in the local servers to ensure the fastest retrieval by search engines, and facilitate multi-lingual search. In one exemplary embodiment, a document or Web page in the appropriate language can be obtained by specifying language code in connection with the document address and/or the URI with corresponding language tag (see, for example, user interface screen shown in FIG. 5H).

[0061] As an example, the Web site www.youtube.com and its Web pages may be translated into Chinese and stored locally in a set of local servers. A user in China may enter a query in Chinese  in a search engine. A search based on such query will be applied to assorted information on the Web, including content that was originally in English such as on the Web site www.youtube.com, but when such content is selected, translated documents of the English-language content is retrieved and presented to the user. See also the example illustrated in FIGS. 5G and 5H which show a Web page in Chinese and the translated page in English, respectively, which would be made available to users based on user's language preference, or selection of language codes and/or tags.

[0062] The user may specify language codes to tag the translated documents and/or Web pages saved or registered in the multi-lingual base to indicate the source language and the target languages, and thus when the user specifies the language code in connection with the document address and/or the URI with corresponding language tag, the translated documents and/or Web pages with corresponding language can be retrieved and displayed. For example, the user may specify the language tag “cn” which represents Chinese language code in connection with the URI http://www.youtube.com to retrieve and display the corresponding translated Web pages tagged with Chinese language code “cn” in the multi-lingual base saved in a set of local servers.

[0063] As already mentioned above, source documents as well as translated documents can be saved in a set of local servers to form a multi-lingual base. The registered information base can contain parallel pages, texts, words, phrases, sentences, expressions and documents between source and target languages, and is updated dynamically by Web users adding new documents and votes into the base. When the translation engine receives a translation request from the Internet for any Web page or any information based on the Web, the translation engine performs the translation procedure automatically, and then both the source documents and the translated documents are added to the multi-lingual base and stored in a set of local servers. Meanwhile the translated documents can be translated back into the source language, and also be added to the multi-lingual base and stored in a set of local servers, with multiple translated versions according to the current vote status in the vote store. Thus the parallel pages, texts, words, phrases, sentences, expressions and documents between source and target languages are downloaded from the Internet and organized pair by pair in the form of viewable Web page and/or in the form of meta-data and/or hidden fields. The pairs can be further extended to include more language pairs, and organized as a chain of viewable Web pages displayed on the Internet and/or in the form of meta-data and/or hidden fields, with certain buttons or links such as “English”, “Spanish”, “Chinese”, etc., to guide the navigation. The user may further specify a target language code in connection with the document address and/or the URI on the World Wide Web to retrieve the corresponding translated document and/or Web page tagged with the corresponding target language code. For example, a Web page containing the term “cheap airplane tickets sold at the lowest price” may be sent to the translation engine and automatically translated into multiple Chinese versions according to the current votes and saved or registered in the local servers as a group of Web pages containing various Chinese terms such as        etc. Then both the source Web page and the translated Web page are downloaded from the Internet and stored in a set of local servers, and are organized pair by pair in the form of viewable Web pages containing the term “cheap airplane tickets sold at the lowest price” and the translated term    whereas the other options such as       etc., are saved or registered in the form of meta-data and/or hidden fields. The Chinese-English pair can be further extended to include more language pairs, for example, English-Spanish pair containing the term “se venden billetes de avión más baratos”, and thus a set of Web pages containing the term “cheap airplane tickets sold at the lowest price”,   and “se venden billetes de avión más baratos” are organized as a chain of viewable Web pages displayed on the Internet and/or in the form of meta-data and/or hidden fields, with certain buttons or links such as “English”, “Spanish”, “Chinese”, etc., to guide the navigation. The user may also add the language tags such as “en”, “cn”, “es”, “ru” etc., which represent target languages English, Chinese, Spanish, Russian, etc., respectively, in connection with the URI which directs to the corresponding Web page to retrieve the Web page translated into target language tagged with the corresponding target language code in the multi-lingual base saved in the local server, and thus to offer a convenient way of searching, retrieving and displaying a document and/or a Web page in corresponding languages.

[0064] Automatically parsing documents saved in the multi-lingual base into sentences and/or other forms of natural language units such as phrases, keywords, colloquial expressions, etc., to form a parallel language information base accessible to the public: when the documents downloaded from the Internet are translated and saved or registered in the local servers, both the source documents and the translated documents are automatically parsed into sentences and/or other forms of natural language units such as phrases, keywords, colloquial expressions, etc., to form a parallel language information base. The parsed units are automatically organized and displayed pair by pair and/or chain by chain in a public directory, which can be accessible to anyone through the Internet. In the public directory of the parsed units, a word, phrase, sentence or expression with multiple translation options is highlighted. Anyone is able to view these translation options and their related terms, and insert their votes by adding new votes into the vote store and/or by selecting any one of the existing translation options as a suggestion. For example, a source document and its translated result may be parsed into a pair of terms “cheap airplane tickets sold at the lowest price” and   and/or a chain of terms “cheap airplane tickets sold at the lowest price”,    and “se venden billetes de avión más baratos”, and then displayed in a public directory for public voting. Viewing the pair of terms and/or the chain of terms displayed in the directory, a user may insert a new vote suggesting that “cheap airplane tickets sold at the lowest price” be translated into  Certain users may also upload the modified units in the public directory onto the saved or registered Web page in the local servers and automatically replace the older versions of translation displayed on the Internet.

[0065] Multi-Lingual Search Bar: both the source documents and the translated documents are saved or registered in a set of local servers to form a multi-lingual base. By providing a multi-lingual search bar to designate the queries to the documents stored in the multi-lingual base, the multi-lingual search functionality is accessible to non-cross-lingual search engines. See the examples of user interface screens illustrated in FIGS. 5I through 5L. In FIG. 5I, the search portal is presented in English but the user can alternatively select Chinese (FIG. 5J). Search specified through the English search portal returns pages that are originally in English as well as English translated pages that were originally in another language such as Chinese (FIG. 5K). Likewise, a search specified through the English search portal returns pages that are originally in Chinese as well as Chinese translated pages that were originally in another language such as English (FIG. 5K).

[0066] Such information base contains parallel pages, texts, words, phrases, sentences, expressions and documents between source and target languages, and thus enables search engines to retrieve documents translated from other languages and implement their cross-language functionality by adding a multi-lingual search bar to their existing search page. For example, the document containing the term “cheap airplane tickets sold at the lowest price” may be translated into multiple Chinese versions according to the current votes and saved in the local servers as a group of documents containing various Chinese terms such as        etc. A user may send a search query  in a non-cross-lingual search engine connected to multi-lingual search bar, the search engine then designates the query to the translated documents stored in the multi-lingual base and retrieve the documents containing the Chinese term  Thus, the multi-lingual search functionality is accessible to non-cross-lingual search engines.

[0067] Retrieving a document through the multiple versions derived from the document: a user may also send search queries in the source language in search engines, and retrieve a document through the multiple versions derived from the document translated back into the source language which are stored in the local servers in the form of meta-data and/or hidden fields. For example, in the Chinese to English translation direction, the current votes may translate  into “discounted” or “inexpensive”,  into “airline ticket”, “plane tickets” or “airfare”, and  into “dumping” or “on sale”. Thus a document containing “cheap airplane tickets sold at the lowest price” may automatically be translated back into the source language English from the documents in Chinese containing   and  and generate multiple translated documents containing terms such as “cheap airplane tickets on sale”, “inexpensive plane tickets sold at the lowest price”, “discounted airfare on sale”, “dumping plane tickets”, etc., according to the current votes. The resultant multiple versions derived from the document are also saved or registered in the local servers in the form of meta-data and/or hidden fields, and a user may send search queries in the source language such as “discounted airfare”, “inexpensive plane tickets”, “plane tickets lowest price”, “airplane tickets on sale”, “dumping plane tickets”, etc., and retrieve the document in the source language that is relevant to the user query through the multiple versions derived from the document.

[0068] Multiple Facets of translated documents: In any target document translated from source language, although the document is translated by selecting one of the translation options for any word, phrase, sentence and expression in the document, the other translation options are still kept in the document in the form of meta-data and/or hidden fields. Thus the translated document is accessible for search engines not only by its viewable words, but also by its hidden words. For example the word  has the translation options as follows: cell phone, cellular phone, cell, mobile phone, handset, etc., although “cell phone” might be used in the visible translation as the most voted translation option or as the default translation option in a specific context, the set of translation options are kept in the translated document in the form of meta-data and or hidden fields. Therefore not only the translated document can be retrieved by search engines by the viewable word “cell phone” contained in the target document, but also by the other translation options “cellular phone”, “cell”, “mobile phone”, etc., stored in the meta-data and/or hidden field of the target document.

[0069] Multiple Results: one document in the target language is translated into a group of documents in the source language, based on the translation options for words, phrases, sentences and expressions in the document. Meanwhile one document in the source language can be translated into a group of documents in the target language in the same fashion. All of these translated documents have the similar meaning (hereinafter “similar documents”) but different wordings. Thus one document becomes a group of documents to be stored in the multi-lingual base, which enhances the possibility of hits by search engines and improves its ranking in the search result list. For example, to search for documents containing “conjeturar sin fundamento” and to find as many of the documents that are relevant to the user query as possible, multiple similar documents containing synonyms such as “adivinar a ciegas”, “hacer suposiciones gratuitas” may be retrieved in the search results, and users may rank, display and narrow the scope of the search results by specifying one of the synonyms as search query, and/or one of the related terms as relevant contexts.

[0070] Dynamic View: Due to multi-facets of the translated documents, when one document (source) is translated into a group of documents (targets), one of the resultant documents only shows the translation option with the most votes for its words, phrases, sentences and expressions, or shows the default translation option in a specific context, for instance in a specific Web page or a cluster of related Web pages, while the other translation options are kept as hidden words in the meta-data. This displayed and viewable target document is the document when querying the translation of the source document. But when the vote for any word, phrase, sentence and expression in the source document is changed, the source document dynamically gets a new target document that is displayed and viewable for searchers. So the displayed and viewable target document is always kept up to date with the vote changes. Therefore, the translation of any source document into target document is dynamic and up-to-date with respect to the current votes. Further, the documents stored in the multi-lingual base can be modified by certain users such as authorized Webmaster of the translated Web page, and the users with power to modify the saved documents can upload the modified terms and/or other kinds of content including but not limited to graphics, music, subtitles, lyrics, etc., onto the saved Web page in the local servers and automatically replace the older versions of translation and/or other kinds of content displayed on the Internet. For example, a translated Web page which displays the term “mobile phone” may dynamically get a new target document which contains the term “cell phone” according to the changes of the current votes in the general vote store or in the special vote store, and/or according to the modification by a Webmaster of the translated Web page, and thus the most up-to-date term “cell phone” may be displayed in the translated Web page, replacing the term “mobile phone” as the viewable translation displayed on the Internet.

[0071] Automatic Synonym Database: With the help of the vote store and translation database, an automatic synonym database in any language X is automatically generated as follows: for any word, phrase, sentence or expression E in the language X, find all of E's translation options in the language Y, then for each of these translation options in the language Y, find all of its translation options in the language X. All of these translation options in the language X for all of these translation options in the language Y constitute the set of synonym database of the word, phrase, sentence or expression E in the language X. With the help of the resultant automatic synonym database, the performance of matching with the source language documents is greatly improved. Automatic Synonym Database differs from traditional thesaurus in that it is interactive, dynamic, up-to-date and more comprehensive, and thus increases the richness of search queries and enables more comprehensive search results. For example, the word  has the translation options as follows: cell phone, cellular phone, cell, mobile phone, etc., and consequently these translation options have the following translation options:       etc. The later set of words, phrases and expressions are stored in the automatic synonym database to multiply search queries and to enable more comprehensive search results.

[0072] Multi-optional Display: When a user gets the retrieved results using a search query, the results are displayed with the documents that match the user's search query. Meanwhile the synonyms of the query automatically generated and the related terms in connection with the synonyms are also displayed to the user at the search result page. When the user selects any of the synonyms and/or related terms, the documents that match the selected synonym are rendered to the user in the result page. So not only the documents matching the entered query, but also the documents matching the synonyms of the query are available to the user in a multi-optional display interface. For example, the user may select the term “discounted airfare” as one of the synonyms of the search query “cheap tickets” from a set of synonyms automatically generated, and then display relevant documents matching the specified synonym in the search results. Further, a user may also select one of the related terms in connection with the synonyms to specify a context for the search query, and display the resultant documents according to specification of the related terms, to narrow the scope of the search results. For example, the user may select the related term “Yahoo Co-founder” displayed at the search result page from a set of related terms in connection with person's name “Jerry Yang”, and then only the documents that match the name “Jerry Yang” and relevant to the context of “Yahoo Co-founder” are displayed to the user, hiding other irrelevant search results from the current result page.

[0073] One Stop Cross-Lingual Query: when a user queries in the source language, the query, as well as the retrieved results, are translated to/from target languages automatically and internally, without any intervention by the user. The user can enter the query in his/her own language (source language) and the multi-lingual search results are presented in the source language. No need for the user to supply more inputs to enter the query, translate the query, retrieve the documents in target language, translate the documents, etc. For example, the user may enter “discounted airfare” and search results are presented in English and are not only retrieved from documents written in English matching the search query “discounted airfare” and/or its automatically generated synonyms, but also from documents written in Chinese, Spanish, French, German, Russian, etc., translated into English automatically and internally, and matching the search query and/or its automatically generated synonyms.

[0074] Monolingual Navigation and/or Multilingual Navigation: If a user enters a query in the source language, the retrieved results can be presented in the source language only, although the results contain both the documents that are originally in the source language and the documents that are originally in the target language. The documents which are originally in the target language are previously translated into the source language, stored in the multi-lingual base, and then are matched with the input query and/or its automatically generated synonyms. From the user's point of view, there is not any discrepancy between these two kinds of documents and the user feels that he/she is navigating through monolingual world in the source language with monolingual interface. Meanwhile, if a user enters a query in the source language, the retrieved results can also be presented in both the source language and the corresponding target language to enable multilingual navigation. In the search result page, an option can be provided to the user to display both the documents in the source language and corresponding translated documents in the target language, and/or to display both the documents in the target language and corresponding documents translated back into the source language. To guide the navigation between the documents in the source language and the documents in the target languages, the user may further specify language codes in connection with the document address and/or the URI on the World Wide Web to retrieve the relevant document and/or Web page tagged with corresponding language code.

[0075] Multi-Lingual Marks and Language Tags or Codes: To identify cross-lingual search results and its original language, the translated documents retrieved are marked by original language bar with indication of its original language. Otherwise, there is no such bar presented in the search results. Further, the language codes or tags such us “en”, “cn”, “es”, “ru” etc., can be added to the multi-lingual base in connection with the corresponding document address and/or the URI. The language codes or tags represent the target languages into which the document was translated, thus the user can retrieve and display the document and/or the Web page by specifying a language code in connection with the document address and/or the URI on the World Wide Web.

[0076] Hybrid Search: when a user submits a query, the search results are obtained from the translated documents stored in the local server as well as from the Web. By translating the search queries into target language, and then using the translated queries to search on the Web, the new documents that have not been previously translated and saved or registered in the multi-lingual base can be retrieved from the Web, listed in the search results, and translated back into source language. Thus hybrid search method not only allows increasing the multi-lingual base with user's queries, but also enhances the multi-lingual search quality to be more up-to-date and comprehensive.

[0077] Ambiguity Resolve: in any translated document, a word, phrase, sentence or expression with multiple translation options is highlighted. A user can view these translation options and their related terms, and select any one of them as a suggestion. The system then takes the suggestion as a vote. The user may also specify a context where he/she inserted the vote by adding related terms to describe certain translation options, and thus let machine record the relevant context of translation. For example, the user may add “Hongkongers” as a related term to describe the translation option “Bush” to  in a context, and add “Yahoo Co-founder” as a related term to describe the translation option “Jerry Yang” to  in another context. In the search result page, both the synonyms of the search query and their related terms are displayed, thus the user can select one of them to specify the search query, and/or select one of the related terms to specify the relevant context, and then display the search results according to the specification of query term and/or relevant context. For example, the user may select the term “Jerry Yang” and/or the related term “Yahoo Co-founder”, and then display the search results relevant to the specified search query and/or the related term, to resolve the ambiguity inherent in the name “Jerry Yang” due to the fact that lots of people have similar or even same names.

[0078] Query Ambiguity Resolve: when a user enters a query in his/her own language, the system shows him/her the related words in translation options in his/her own language in real time and the user is allowed to select any one of them to fine tune the query.

[0079] Phrase and Expression Detection: New phrases and expressions are found by calculating the probability of two or more words occurring consecutively in one document, paragraph or sentence. These new phrases or expressions may or may not have any translation options. But they are stored in the translation database as phrases or expressions. When translating a document, if the document contains such kind of phrases or expressions that have no translation options, the phrases or expressions are highlighted and the user is prompted to vote for these phrases or expressions.

[0080] An example (FIG. 4) of a cross-language information retrieval system includes a translation engine 1, search engine 2, Web spider 3, vote store 4, thesaurus 5, new phrase store 6, new phrase finder 7, document base 8, search interface 9 and vote interface 10. The translation engine 1 translates documents and queries from source languages to target languages. The search engine 2 searches the document base 8 and Web with users' query. The Web spider 3 navigates the Web and downloads files. The Vote Store 4 stores vote information. The thesaurus 5 stores synonyms. The new phrase store 6 stores new phrases which have no votes. The new phrase finder 7 finds new phrases. The document base 8 stores downloaded documents and translated documents. A user uses the search interface 9 to enter search queries and view the retrieved results. A user uses the vote interface 10 to view and update the vote store.

[0081] The components work and interact with each other as follows.

[0082] The Web spider 3 crawls along the Web, retrieves and downloads interested Web pages to form a growing information base. The downloaded Web pages are saved or registered in the Document Base 8. The Translation Engine 1 translates the downloaded documents and saves or registers the translated documents in the Document Base 8. During translation of the documents, the translation engine 1 looks up the Vote Store 4, retrieves available translation options for the term (word/sentence/expression), and translates it into the translation option with the most votes. During translation of the documents, the translation engine 1 saves or registers other translation options with less votes in the document as meta-data and/or hidden fields. If the word has no translation options but in the meantime appears in the New Phrase Store 6, the translation engine 1 highlights the word to indicate it as a new phrase with a special mark. The Vote Interface 10 accepts users' votes, updates the Vote Store 4 bi-directionally in real time, and removes the term (word/sentence/expression) from the New Phrase Store 6 if it appears in the Vote Store 4. The Vote Interface 10 invokes the Translation Engine 1 to update those documents which contain the term (word/sentence/expression) with the new translation options, and thus make the translated document updated dynamically with the newly input information.

[0083] The synonyms of a query term are generated as follows: for any term (word/phrase/sentence/expression) E in the language X, find all of E's translation options in the language Y, then for each of these translation options in the language Y, find all of its translation options in the language X. All of these translation options in the language X for all of these translation options in the language Y constitute the set of synonym database of the term E in the language X. The generated synonyms are saved or registered in the Thesaurus 5.

[0084] The Search Interface 9 accepts a user's query and invokes the Search Engine 2. The Search Engine 2 searches the Document Base 8 and the Web with the user's query and returns the Search Interface 9 with a ranked list of documents in the user's native language. The Search Interface 9 not only displays this list of documents to the user, but also displays a set of synonyms of the user's query by searching the Thesaurus 5. If the user clicks on a query term's synonym, a new search process starts with the selected synonym, and the system displays a ranked list of documents related to the synonym's search results. If the user clicks on a document link displayed, the document content are shown on the screen, with each term (word/sentence/expression) accompanied with its vote status in the Vote Store 4 as well as in the New Phrase Store 6. The user can view each word/sentence/expression's vote information by clicking its vote status. The user is presented with the Vote Interface 10, through which the user can vote for the word/sentence/expression, and for new phrases. The New Phrase Finder 7 runs in the background as follows: it finds new phrases by calculating the probability of two or more words occurring consecutively in one document, paragraph or sentence in the Document Base 8, and once new phrases are found, they are saved or registered in the New Phrase Store 6.

[0085] The Search Interface 9 accepts a user's query and invokes the Search Engine 2. The Search Engine 2 searches the Document Base 8 and the Web with the user's query and returns the Search Interface 9 with a ranked list of documents in the user's native language. To guide the navigation between the documents in the source language and the documents in the target languages, the user may further specify a language code in connection with the document address and/or the URI on the World Wide Web to retrieve and display the relevant document and/or Web page tagged with corresponding language code in the multi-lingual base. The Search Interface 9 not only displays this list of documents to the user, but also displays a set of synonyms of the user's query by searching the Thesaurus 5. If the user clicks on a query term's synonym, a new search process starts with the selected synonym, and the system displays a ranked list of documents related to the synonym's search results. If the user clicks on a document link displayed, the document content are shown on the screen, with each term (word/sentence/expression) accompanied with its vote status in the Vote Store 4 as well as in the New Phrase Store 6. The user can view each word/sentence/expression's vote information by clicking its vote status. The user is presented with the Vote Interface 10, through which the user can vote for the word/sentence/expression, and for new phrases. The New Phrase Finder 7 runs in the background as follows: it finds new phrases by calculating the probability of two or more words occurring consecutively in one document, paragraph or sentence in the Document Base 8, and once new phrases are found, they are saved or registered in the New Phrase Store 6.

[0086] It should be apparent that the cross-language information retrieval system discussed exemplarily above can operate as an independent system or a backend coupled to a conventional search engine or information retrieval system to provide the search engine or system with the additional functionalities disclosed in this disclosure.

[0087] The above specific examples and embodiments are illustrative, and many variations can be introduced on these examples and embodiments without departing from the spirit of the disclosure or from the scope of the appended claims. For example, elements and/or features of different illustrative embodiments may be combined with each other and/or substituted for each other within the scope of this disclosure and appended claims.