Document Retrieval System Involving Ranking of Documents in Accordance With a Degree to Which the Documents Fulfill a Retrieval Condition Corresponding to a User Entry

1. Field of the Invention

The present invention generally relates to a document retrieval system, and more particularly to a document retrieval system in which documents retrieved are given in an order starting from the document closest to the user's requirement so that it is possible to flexibly and quickly retrieve documents.

2. Related Art

In a conventional document retrieval system, at the time of registering documents in the system, an operator selects keywords fitting each document, then, each document for which keywords have been selected is classified on the basis of the thesaurus. When the keywords are added or eliminated, the thesaurus must be renewed. The renewal operation of the thesaurus is quite complex. In addition, when documents which nearly (not exactly) fulfill the user's requirement are retrieved, it is necessary to examine the requirement in detail. To eliminate these disadvantages, the document retrieval system in which a keyword connection and a relevance value are used has been proposed. The keyword connection represents the degree of relationship between each two keywords. The relevance value represents a document's a degree of relevance as a document corresponding to the user's requirement. The greater relevance value is given to the document which is closer to the user's requirement. For example, copending U.S. application Ser. No. 296,769, now U.S. Pat. No. 5,168,565 issued Dec. 1, 1992 proposes a document retrieval system in which the keyword connection and the relevance value are used so that it is possible to flexibly retrieve documents which are required by the user. In addition, Japanease Patent Application No. 63-187988 proposes a document retrieval system having a learning function. In this system, the weight of each relationship in the keyword connection is changed in accordance with the user's judgement and designation so that the user's judgement is reflected in the next retrieval result.

In the proposed document retrieval system, it is possible to retrieve documents only in a case where the user's query is represented by a keyword formula in which keywords are connected by only logical OR. However, a keyword formula which represents the user's arbitrary query generally contains keywords connected by logical AND, OR and NOT. A document retrieval process in which documents are retrieved on the basis of the keyword formula including logical AND, OR and NOT has not yet been developed. That is, in the document retrieval system in which the keyword connection and the relevance value are used, it is impossible to retrieve the document "not having a keyword X", and to retrieve the document "having both keywords X and Y".

Accordingly, a general object of the present invention is to provide a novel and useful document retrieval system in which the disadvantages of the aforementioned prior art are eliminated.

A more specific object of the present invention is to provide an improved document retrieval system in which the keyword connection and the relevance value are used.

Another specific object of the present invention is to provide a document retrieval system in which the relevance value is calculated in accordance with the keyword formula containing keywords connected by the logical OR, AND and NOT, and learning is performed, which changes the keyword connection so that the retrieval result fits the user's requirement.

The above mentioned objects of the present invention can be achieved by a document retrieval system for retrieving one or a plurality of registered documents from a document database responsive to retrieval conditions designated by a user, the document retrieval system comprising, query converting means for converting the retrieval condition designated by the user into a query, the query having a predetermined normal form in which keywords and at least one type of logical operation out of logical operations AND, OR and NOT are connected, bibliographical information indicating means for indicating the relation between each of the registered documents and keywords, a keyword connection table having relationship values, each of the relationship values representing the degree of relationship between each two keywords, selecting means for referring the bibliographical information indicating means and the keyword connection so as to select one or a plurality of registered documents which satisfy the query from the query converting means, and outputting means for outputting one or a plurality of registered documents selected by the selecting means. The selecting means comprises calculation means for calculating a relevance value for each of the registered documents corresponding the the query from the query converting means and one or plurality of relationship values corresponding to keywords given to each of the documents in the bibliographical information indicating means, one or a plurality of relationship values being obtained from the keyword connection table, the relevance value indicating the degree of relationship between the query and each of the registered documents, and storing means fof storing one of a plurality of registered documents in downward sequential order of the relevance value calculated by the calculation means.

The document retrieval system further comprises inputting means for inputting an evaluation value which indicates the degree of requirement of the user for each of the retrieved documents selected by the selecting means, and learning means for using the evaluation value input by the inputting means so as to modify one or a plurality of the relationship values in the keyword connection table.

Additional objects, features and advantages of the present invention will become apparent from the following detailed description when read in conjunction with the accompanying drawings.

FIG. 1 is a basic block diagram illustrating a document retrieval system of a preferred embodiment according to the present invention;

FIG. 2 is a view illustrating by example of a keyword connection table;

FIG. 3 is a flowchart illustrating a document retrieval process in the system shown in FIG. 1;

FIG. 4 is a flow chart illustrating a process executed according to the present invention.

FIG. 5 is a flow chart illustrating a process executed by the present invention according to a first case of FIG. 4.

FIG. 6 is a flow chart illustrating a process executed according to the present invention, in accordance with a second case of FIG. 4.

FIG. 7 illustrates a screen on which retrieval results are displayed.

FIG. 8 illustrates a screen on which titles of documents, bibliographical information, and the like, are displayed.

FIG. 9 illustrates a screen used to allow input of evaluation information for the learning function of the present invention.

A description will now be given of a preferred embodiment of the present invention.

In a document retrieval system, which will be described below, the keyword connection and the relevance value are used. Thus, the user flexibly selects documents, which fulfill the user's requirement, on the basis of the relevance values. The keyword connection represents relationships among keywords. The keyword connection is made up of relationship values. Each of the relationship values represents the degree of a relationship between two keywords. The relationship value is considered to be in a range between "0" and "1". When the relationship value is equal to "0", there is no relationship between the corresponding two keywords. When the relationship value is greater than "0", there is some relationship between the corresponding two keywords. In addition, when the relationship value is equal to "1", the corresponding two keywords have the closest relationship with each other. The keyword connection is also expressed as a two dimensional arrangement of the relationship values. The relationship value between the i-th keyword Ki and the j-th keyword Kj is denoted as Wi[Kj], more simply as Wij.

The relevance value represents a document's degree of relevance as a document satisfying to the user's requirement. The greater relevance value is given to the document which is closer to the user's requirement.

FIG. 1 shows a basic structure of a document retrieval system.

A keyword connection control section 13 receives registered document 11 which is to be registered and bibliographical information 12 of the registered document 11. Keywords given to each document, which is input from the key board 23 by the user, is supplied to the keyword connection control section 13. Then, the keyword connection control section 13 generates the keyword connection table and stores it into a file 15. The keyword connection control section 13 manages the keyword connection table stored in the file 15. A document information control section 14 receives the registered document 11 and the bibliographical information 12 thereof and the keywords of each registered document. Then, the document information control section 14 stores the registered document 11 and the corresponding bibliographical information 12 as a document information database into a file 16. The document information control section 14 manages the document information database stored in the file 16. In addition, the keyword entered from the key board 23 is coupled to the corresponding registered document 11 and the document information control section 14 makes an inverted file which represents a relation between the input keywords and each document. The document information control section 14 stores the inverted file. The inverted file is one type of a bibliographical information indicating means which indicates the relation between the input keywords and each document. In the present invention, the bibliographical information indicating means is not limited to the inverted file. It is possible to provide another type of the bibliographical information indicating means such as an index file.

A document selection section 17 includes a relevance value calculating part 18, a document sorting part 19, a request processor part 20, a display control part 21 and a learning control part 22. The relevance value calculating part 18 uses the required information in the keyword connection table stored in the file 15 and calculates the relevance value of each document on the basis of a query transferred from the request processing part 20. A procedure for calculating the relevance value in more detail will be described later.

The document sorting part 19 sorts documents in which the relevance value is calculated in downward sequential order of the relevance value. The results of the sorted documents are transferred to the display control part 21. The display control part 21 permits the retrieval results to be displayed on a display device 24 in accordance with the user's designations which are input from the key board 24 and supplied through the request processing part 20. The learning control part 22 carries out a learning proces of the keyword connection in accordance with the user's designations from the request processing part 20. In the learning control part 22, a change of each relationship value in the keyword connection is calculated in accordance with the user's designation from the request processing part 20. The change of each relationship value is transferred to the keyword connection control section 13, and then the keyword connection control section 13 rectifys the keyword connection table stored in the file 15. A more detailed procedure for the learning will be described later.

The request processing part 20 transfers the query given by the user from the key board 23 to the relevance calculating part 18 for achieving a retrieval of documents fulfilling the user requests. The request processing part 20 also supplies an instruction relating to the learning to the learning control part 22.

Due to the document retrieval, appropriate documents which are required by the user are found, and the retrieval result is displayed on the display device 24.

The query which is the retrieval condition is explained by the keywords specified by the user and the bibliographical information in which the conditions for retrieving are determined. First, it is determined whether the conditions other than keywords, which are given to each document, fulfill the user's requests or not. Regarding the documents which have the conditions fulfilling the user's request, the relevance value thereof is calculated. On the other hand, regarding the documents which have no condition fulfilling the user's request, the relevance value is not calculated and "0" is determined as the relevance value thereof.

The user inputs the query expressing information which is requested by the user from the key board 23. The query is represented by one keyword or a plurality of keywords connected by the logical AND, OR and NOT. In a case where the information requested by the user is not expressed by one keyword, a plurality of the keywords are used.

Next, descriptions will now be given of (1) the query input to this document retrieval system, (2) the procedure for calculating the relevance value and, (3) the procedure for the learning of keyword connection.

The document selection part 17 retrieves documents in accordance with the flowchart shown in FIG. 3. In the flowchart, step 100 is performed in the request processing part 20. Step 200 is performed in the relevance value calculating part 18. Step 300 is performed in the document sorting part 19.

(1) The query from the user:

The query input from the key board 23 is expressed by the plurality of the keywords connected by the logical AND, OR and NOT. That is, it is possible for the user to order the document retrieval system "a document which does not have a keyword A", "a document which fulfills a condition where there are both keywords A and B", and the like.

The query can be converted into a conjunctive normal form. The conjunctive normal form includes subqueries. The subqueries are connected by only the logical AND with each other. Then, each of the subqueries is expressed by keywords connected by the logical OR and NOT. The request processing part 20 converts the query from the key board 23 into the conjunctive normal form step 100). The query in the conjunctive normal form is written as:

where "*" represents the logical AND, and N is equal to or greater than 1 (N.gtoreq.1). SQ(h) (h=1, . . . , N) is the subquery, and is written as:

where Kq.sub.i represents the i-th keywords, and "+" and "!" respectively denote the logical OR and the logical NOT. Then, "n" is equal to or greater than 0 (n.gtoreq.0), "m" is equal to or greater than 0 (m.gtoreq.0) and "n+m" is equal to or greater than 1 (n+m.gtoreq.1). When Q.sup.+ (h) is a set including the keywords without NOT, and Q.sup.- (h) is a set including the keywords with NOT, Q.sup.+ (h) and Q.sup.- (h) are respectively represented as:

Thus, no keyword is included in both sets Q.sup.+ (h) and Q.sup.- (h). The request processing part 20 checks whether or not there is a keyword which is included in both the sets Q.sup.+ (h) and Q.sup.- (h) in each of the subqueries. Then, if the request processing part 20 detects a keyword included in both the sets Q.sup.+ (h) and Q.sup.- (h), this checking result is displayed on the display device 24, and thus the user can be made aware of errors.

(2) The calculation of the relevance value:

The relevance value represents the degree of relationship between the retrieval condition (the query) and each document. The relevance value calculating part 18 calculates the relevance value of each document in accordance with step 200 shown in FIG.3.

The relevance value is calculated as follows.

First, a relevance value of the subquery (termed a sub-relevance value) is calculated. The sub-relevance value is calculated by step 213 shown in FIG. 3.

Second, a product of sub-relevance values of all subqueries is calculated so that the relevance value of one document is obtained. Steps 213 through 216 are repeated so that the relevance value of one document is calculated.

The relevance values of all documents are calculated, and then appropriate documents are selected by a threshold process or the like depending on the user's designation. Finally, the appropriate documents are retrieved.

In the flowchart shown in FIG. 3, document number i is initially set to "1" in step 201, the initial sub-relevance value ri is set to "1.0" in step 211, and then the sub-condition number h which is a number given to each sub-query is initially set to "1" in step 212. After that, step 213 calculates the h-th sub-relevance value ri(h) on the basis of the following formula: ##EQU1##

In the formula (3), when Ai is a set of keywords of i-th document, Rij and Sij are respectively defined as follows: ##EQU2##

In these formulas (4) and (5), Wjk denotes the degree of the relationship between the k-th keyword and the j-th keyword. That is, Wjk is the relationship value in the keyword connection matrix.

When the set Q.sup.+ (h) or the set Q.sup.- (h) is the empty set (.phi.), the sub-relevance value is calculated on the basis of the following formulas, instead of the formula (3) described above: ##EQU3##

After calculating in step 213 as has been described above, the sub-relevance value ri(h) which is obtained by the calculation in step 213 is multiplied by the ri value which was obtained last time. Then the product of ri.times.ri(h) becomes a new ri value. This process is performed in step 214. While the sub-condition number h is successively incremented to N (steps 215 and 216), the calculations in steps 213 and 214 are performed. Then finally, step 213 calculates all sub-relevance values of N subqueries, and the final ri value in step 214 becomes as follows. ##EQU4##

The final ri value is the relevance value of the document including a set Ai of keywords.

While the document number i is successively incremented (steps 202 and 203), the relevance value is calculated as has been described above. Finally, the relevance values of all registered documents are sorted in downward sequential order thereof by step 300 which is a process performed by the document sorting part 19. Then, the appropriate document selected by the threshold process or the like is displayed on the display device 24.

A description will now be given of an example of the calculation of the relevance value.

the relevance value r1 of the document, including keywords {K1, K2}, is calculated. The keyword connection matrix of keywords K1, K2, K3 and K4 is given as shown in FIG. 2. In the relationship values in the keyword connection matrix, Wij is equal to Wji (Wij=Wji) so that the keyword connection is indicated as a triangular form.

The query is converted into the conjunctive normal form by De Morgan's rules. That is,

In this case,

Then, ##EQU5## Accordingly,

That is, in this case, the relevance value of the document is equal to 0.867.

On the other hand, the query can be also converted into a disjunctive normal form. The disjunctive normal form includes subqueries. The subqueries are connected by only the logical OR with each other. Each of the subqueries is expressed by keywords connected by the logical AND and NOT. Then, after converting the query into the disjunctive normal form, the relevance value of each document can be calculated by using the disjunctive normal form.

The query in the disjunctive normal form is written as:

where N is equal to or greater than 1 (N.gtoreq.1). SQ(h) (h=1, . . . , N) is the subquery, and is written as:

where Kq.sub.i represents the i-th keywords. Then, "n" is equal to or greater than 0 (n.gtoreq.0), "m" is equal to or greater than 0 (m.gtoreq.0) and "n+m" is equal to or greater than 1 (n+m.gtoreq.1). When Q.sup.+ (h) is a set including the keywords without NOT, and Q.sup.- (h) is a set including the keywords with NOT, Q.sup.+ (h) and Q.sup.- (h) are respectively represented as:

Thus, no keyword is included in both sets Q.sup.+ (h) and Q.sup.- (h).

The relevance value of one document is calculated as follows.

When the sub-relevance value corresponding to the h-th subquery is ri(h), the sub-relevance value is calculated on the basis of the following formula. ##EQU6##

In the formula (13), Rij and Sij are the same ones difined by formulas (4) and (5).

After the sub-relevance values are calculated by the formulas (11) (4) and (5), the relevance value ri of the i-th document is finally calculated as follows: ##EQU7##

Thus, the relevance value of the document can be calculated on the basis of formulas (11) (4) (5) and (12). In the case where the query is converted into the disjunctive normal form, the relevance value of the document is calculated in accordance with substantially the same flowchart as shown in FIG. 3. In this case, the calculations of the formulas (11) (4) and (5) are performed in step 213, and the calculation of the formula (12) is performed in step 214.

A description will now be given of an example of the calculation of the relevance value.

the relevance value r1 of the document, including keywords {K3, K4}, is calculated. The keyword connection of keywords K1, K2, K3 and K4 is given as shown in FIG. 2.

In this case, the query can be converted into the conjunctive normal form as follows:

However, when the relevance value is calculated in accordance with the query converted into the disjunctive normal form, it is not necessary for the query to convert into the conjunctive normal form, as the query given above is represented by the disjunctive normal form.

That is, the relevance value of the i-th document is calculated as follows:

Then, the sub-relevance values are calculated as follows:

Accordingly,

That is, in this case, the relevance value of the document is equal to 0.433.

(3) The procedure for the learning of keyword connection:

(3-1) In the case where the query is expressed by the conjunctive normal form;

The user selects a document (i-th document), for which it is necessary to correct the relevance value, from a document list in which the retrieved documents are arranged in downward sequential order of relevance value. The user inputs, from the key board 23, a evaluation value t which indicates the degree of the requirement of the user for the i-th retrieved document. In the learning of the keyword connection, when the evaluation value t is greater than the relevance value ri of the i-th document (t>ri), the relationship values in the keyword connection matrix are changed so that the relevance value ri of the i-th document increases. Then, when the evaluation value t is less than the relevance value ri (t<ri), the relationship values in the keyword connection are changed so that the relevance value ri of the i-th document decreases. In addition, when the evaluation value t is equal to the relevance value ri, the learning of the keyword connection is not performed.

The learning of the keyword connection is carried out by the learning control part 22. The basic procedure for the learning of the keyword is, for example, proposed in Japanese Patent Application No. 1-132696. That is, at the time of learning, when the user judges that the retrieved document is completely irrelevant, the evaluation value of the document is equal to "0", and when the user judges that the retrieved document is completely relevant, the evaluation value of the document is equal to "1". Then, when the user is vague about whether the retrieved document is relevant or not, the evaluation value is an arbitrary value between "0" and "1".

When the evaluation value expressing the user's judgement for the document having the relevance value X is "t", an evaluation function is defined as follows.

In one learning process, all relationship values in the keyword connection matrix are modified so as to reduce the value of the evaluation function E(ri) for the retrieved i-th document having the relevance value ri. That is, the relationship value Wmn is modified as follows:

In this formula, g() is a normalizing function to ensure that the modified relationship value remains within a range between "0" and "1". ##EQU8##

The changing value .DELTA.Wmn is calculated as follows by using the gradient descent method: ##EQU9##

In the formula (16), .alpha. is a learning coefficient of a positive real number. Then, according to the formula (8) described above, the following formula is obtained: ##EQU10##

Thus, the changing value .DELTA.Wnm is calculated in accordance with the following formula: ##EQU11##

In the formula for which the changing value .DELTA.Wnm is calculated, (.differential. ri(h)/.differential. Wmn) is calculated in accordance with formulas (19) (20) and (21) as will be described later. In a case other than the case where the set (m,n) fulfills the condition where Km Query and Kn Ai, the changing value .DELTA.Wmn=0, so that the changing value .DELTA.Wmn is calculated with regard to only a set (m,n) which fulfills the condition where Km Query and Kn Ai.

The partial differential(.differential. ri(h)/.differential. Wmn) must be calculated to obtain the changing value .DELTA.Wmn. Due to the assumption that no keywords are included in both the set Q.sup.+ (h) and the set Q.sup.- (h), the partial differential(.differential. ri(h)/.differential. Wmn) can be calculated in each of three cases as follows. ##EQU12##

The result calculated in accordance with one of the formulas (19) (20) and (21) is substituted for the formula (18) so that the changing value .DELTA.Wmn is obtained.

Furthermore, the formula (18) to calculate the changing value .DELTA.Wmn can be simplified. The calculating formula in the case where ri is not equal to "0"(ri.noteq.0) differs from the calculating formula in the case where ri is equal to "0"(ri=0).

In the case where ri.noteq.0:

ri(h) is not equal to "0" for all h (1.ltoreq.h.ltoreq.N), so that, from the formula (8), the following formula is obtained: ##EQU13##

Therefore, the formula (17) is converted as follows: ##EQU14##

Thus, ##EQU15##

In the case where ri=0:

In this case, formulas differ from each other in accordance with the number of the subquery in which the sub-relevance value ri(h) is equal to "0". When the number of the subquery in which ri(h)=0 is equal to "1" the formula (17) is converted as follows: ##EQU16##

In the formula (25), h* makes ri(h*)=0. Thus, ##EQU17##

When the number of the subquery in which ri(h)=0 is equal to or greater than "2" the formula (11) is converted as follows: ##EQU18##

Thus,

In the learning of the keyword connection as has been described above, the user inputs the evaluation value t for the i-th retrieved document having the relevance value ri. Then, the learning part 22 carries out the process in accordance with the flowchart shown in FIG. 4.

Referring to FIG. 4, step 400 determines whether the relevance value ri of the i-th retrieved document is equal to "0" or not. When the relevance value ri is equal to "0", step 410 carries out the process of "Case 1". Step 410 carries out the process of "Case 1" in accordance with a flowchart shown in FIG. 5. In FIG. 5, step 411 selects a keyword Km from keywords included in the query, and then step 412 selects a keyword Kn from keywords (Ai) given to the i-th document. Step 413 calculates the changing value .DELTA.Wmn on the basis of the formula (30) and calculates a new relationship value Wmn which represents the degree of the relationship between the keywords Km and Kn.

Step 413 includes steps 421 through 425. The sub-condition number h which is a number given to each sub-query is initially set to "1" in step 421. Then, "tmp" is initially set to "0" in step 422, where, finaly, "tmp" becomes as follows: ##EQU19##

After that, while the sub-condition number h is successively incremented to N (step 425), a new "tmp" value is calculated by using the last "tmp" in step 423, as follows: ##EQU20## When the sub-condition number reaches to N and the process for all sub-queries is finished (step 424 finally, the changing value .DELTA.Wmn is calculated. Then, step 426 calculates the new relationship value Wmn on the basis of formula (14). That is, the calculation where

is performed. The process which is carried out in steps 412 and 413 is repeated for all keywords given to the i-th document (step 414). Then, when the process for all keywords given to the i-th document is finished, the process which is carried out in steps 411, 412, 413 and 414 is repeated for all keywords included in the query (step 415). As a result, new relationship values Wmn between every keyword given to the i-th document and every keyword included in the query are obtained.

On the other hand, when step 400, shown in FIG. 4, determines that the relevance value ri is not equal to "0", step 500 calculates new relationship values in the keyword connection, as follows.

The sub-condition number h, a count value which represents a number of h fulfilling the condition where ri(h) is equal to "0" and the tmp value are respectively initially set (steps 501, 502 and 503) so that:

where, finally, "tmp" becomes as follows: ##EQU21##

After that, step 504 determines whether the sub-relevance value ri(h) is equal to "0.0" or not. Then, when the sub-relevance value ri(h) is not equal to "0.0", the tmp value is multiplied by the sub-relevance value ri(h) in step 505. The product of tmp.times.ri(h) becomes a new tmp value. When sub-relevance value ri(h) is equal to "0" in step 504, step 508 determines whether the count value is greater than "1" or not. When the count value is not greater than "1", the count value is incremented in step 509, and then sub condition number h is replaced with h * in step 510. While the sub condition number h is successively incremented, the process in step 505 or in steps 509 and 510 is repeated (steps 506 and 507). Then, if the count value becomes greater than "1" in step 508, the process is finished at this time. In this case, on the basis of the formula (28) described above, the relationship values in the keyword connection are not changed (.DELTA.Wmn=0).

When the sub condition number h reaches to N, that is, when the process with regard to all subqueries is finished, step 520 carries out the process of "Case 2" in accordance with a flowchart shown in FIG. 6. In FIG. 6, step 521 selects a keyword Km from keywords included in the query, and step 522 selects a keyword Kn from keywords (Ai) given to the i-th document. Step 523 calculates the changing value .DELTA.Wmn on the basis of the formula (26) and calculates a new relationship value Wmn which represents the degree of the relationship between the keywords Km and Kn on the basis of the formula (14). That is, the calculation where

is performed. In this formula, tmp value and h * have been calculated in step 500 shown in FIG. 4, and F(i, h * , Km, Kn) is the partial differential in the formula (26), as follows: ##EQU22##

The process which is carried out in steps 522 and 523 is repeated for all keywords given to the i-th document (step 524). Then, when the process for all keywords given to the i-th document is finished, the process which is carried out in steps 521, 522, 523 and 524 is repeated for all keywords included in the query (step 525 ). As a result, new relationship values Wmn between every keyword given to the i-th document and every keyword included in the query are obtained.

When the learning process described above is finished, the learning control part 22 supplies new relationship values through the request processor part 20 to the keyword connection control section 13. Then the keyword connection control section 13 replaces the corresponding relationship values in the keyword connection matrix stored in the file 15 with the new relationship values.

(3-2) In the case where the query is expressed by the disjunctive normal form;

The following formula is obtained according to the formula (12): ##EQU23##

This formula (29) can be converted into simple calculation formulas as follows.

In the case where ri(h).noteq.1;

ri(h) is not equal to "0" for all h (1.ltoreq.h.ltoreq.N) so that the following formula is obtained: ##EQU24## Thus, the formula (29) is converted as follows. ##EQU25##

In the case where ri(h)=1 ;

formulas differ from each other in accordance with the number of the subquery in which the sub-relevance value ri(h) is equal to "1". When the number of the subquery in which ri(h)=1 is equal to "1", the formula (29) is converted as follows: ##EQU26## In the formula (32), h* makes ri(h*)=1.

On the other hand, when the number of the subquery in which the sub-relevance value ri(h) is equal to "1" is equal to or greater than "2", the formula (29) is converted into as follows. ##EQU27##

A partial differential (.differential. ri(h)/.differential. Wmn) must be calculated to obtain the partial differential(.differential. ri/.differential. Wmn) in accordance with formula (31) or (32).

A description will be given of a procedure for calculating the partial differential .differential. ri(h)/.differential. Wmn).

Due to the assumption that no keywords are included in both the set Q.sup.+ (h) and the set Q.sup.- (h), the partial differential(.differential. ri(h)/.differential. Wmn) can be calculated in each case of three cases as follows.

1 First case: Km Q.sup.+ (h)

The following calculation formula of the partial differential(.differential. ri(h)/.differential. Wmn) is obtained. ##EQU28##

2 Second case: Km Q.sup.- (h)

The following calculation formula of the partial differential(.differential. ri(h)/.differential. Wmn) is obtained. ##EQU29##

3 Third case: Km Q.sup.+ (h) & Km Q.sup.- (h)

In this case, the calculation formula is simple. That, is ##EQU30##

As has been described above, the partial differetial(.differential. ri(h)/.differential. Wmn) can be calculated by using the formulas (31) (32) (33) (34) (35) and (36). When Km Query or Kn Ai where ##EQU31##

.DELTA.Wmn is equal to "0" (.DELTA.Wmn=0). Thus, the partial fifferential(.differential. ri(h)/.differential. Wmn) is calculated with regard to only a set(m,n) which fulfills the condition where Km Query and Kn Ai.

In the process of the learning of the keyword connection as has been described above, the user inputs the evaluation value t with regard to the retrieved document, in which the relevance value is given, to the system. The evaluation value t is a real number in a range between 0 and 1, and represents the suitability of the retrieved document. The evaluation value is "1" when the retrieved document is desired by the user, and the evaluation value is "0" when the retrieved document is not desired by the user. In addition, when the user's evaluation is vague, for example, in a case where the document is not desired very much by the user, the evaluation value is an arbitrary value between 0 and 1.

It is difficult for the user to represent the evaluation of the retrieved document by a real number in the range between 0 and 1.

Accordingly, in this document retrieval system, the user can give the evaluation of N ranks (N.gtoreq.2). The greater the evaluation rank is, the greater the corresponding evaluation value that is set. When the user gives the n-th rank of the evaluation, the evaluation value t is calculated in accordance with t=f(n). The function f(n) must fulfill the following conditions:

For example, the function f(n) is determined as follows: ##EQU32##

In this function f(n) in which N is determined equal to 5 (N=5), when the user gives the fifth rank of evaluation (n=5), the evaluation value t is equal to 1.0 (t=1.0), and when the user gives the second rank of evaluation (n=2), the evaluation value t is equal to 0.25 (t=0.25).

A description will now be given of an interface for the user to input the ranks of the user's evaluation of the retrieved document with reference to FIGS. 7 through 9.

The retrieval result is displayed on the display divice 24 as shown in FIG. 7. In FIG. 7, the display is divided into a plurality of windows. A window 50 indicates a title of "RETRIEVAL RESULT" and the process after retreiving such as "EDIT", "DISPLAY" and the like. A window 54 indicates retrieval information. The retrieval information inculudes the rank number of the document, the relevance value of the document and the title of the document. The retrieved documents are desplayed in downward sequential order of the relevance value. If the user wants to see the content of the retrieved document, "DISPLAY" in the window 50 is selected by a mouse or the key board 23. Then a rank number corresponding to the document is selected so that the content and the bibliographical information of the selected document and the like are displayed as shown in FIG. 8. That is, windows on the display device 24 are changed. In FIG. 8, a window 55 indicates "BIBLIOGRAPHICAL INFORMATION" and the next process such as "LEARNING", and the like. A window 56 indicates information with regard to the selected document. A window 57 indicates a title of "TEXT", and a window 58 indicates the content of the selected document. When the "LEARNING" in the window 55 is selected by the mouse or the key board 23, a new window 60 which overlaps the windows 56 and 57 is generated on the display device 24. In the window 60, five marks 62(1) through 62(5) are indicated. The marks 62(1) through 62(5) correspond to the five ranks of the evaluation of the retrieved document. The first mark 62(1) corresponds to the greatest evaluation degree, and the fifth mark 62(5) corresponds to the least evaluation degree. Then, each of marks 62(2) through 62(4) corresponds to an evaluation degree less than the greatest one and greater than the least one, and they are arranged in downward sequential order of the evaluation degree. For example, when N is equal to 5 (N=5), the mark 62(1) corresponds to the condition where n=5, the mark 62(2) corresponds to the condition where n=4, the mark 62(3) corresponds to the condition where n=3, the mark 63(4) corresponds to the condition where n=2 and the mark 62(5) corresponds to the condition where n=1. Then, when the user selects one of the marks 62(1) through 62(5), the evaluation value t is calculated in accordance with the formula (51) by using the corresponding n. For example, when the user selects the mark 62(3), n=3 so that the evaluation value t becomes to 0.5 (t=0.5). This calculated evaluation value t is supplied to the learning control part 22, and the learning control part 22 performs the learning process on the basis of the suplied evaluation value t, as has been described above.

Thus, at the time of the learning of the keyword connection, it is not necessary for the user to directly input the evaluation value t which is the exact real number. The user simply selects one of the marks indicated on the display device 24 so that the learning of the keyword connection is carried out.

The processes for the learning of the keyword connection described with reference to FIGS.7 through 9 are performed in the request processor part 20, the display control part 21 and the learning control part 22.

In the embodiments described above, keywords for documents are input by the user from the key board 23, and the document control section 14 generates and stores the inverted file which indicates the relation between the keywords and each of the documents. However, it is possible to provide a keyword extracting part which extracts keywords from documents when registering the document, and to provide an inverted file generator which is supplied with registered documents and keywords and generates the inverted file. The keyword extracting part and the inverted file generator described above are further disclosed in co-pending U.S. patent application Ser. No. 296,769, the disclosure of which is hereby incorporated by reference.

According to the present invention, it is possible to provide a document retrieval system in which the relevance value is calculated in regard to the keyword formula containing keywords connected by the logical OR, AND and NOT, and the learning is performed, which changes the keyword connection so that the retrieval result fits the user's query.

The present invention is not limited to the aforementioned embodiments, and variations and modifications may be made thereof without departing from the scope of the claimed invention.