Evaluating distinctiveness of document

[0001] 1. Field of the Invention

[0002] The present invention relates to natural language processing which includes document summarization. More particularly, the present invention relates to quantitatively evaluating the degree of distinctiveness of a constituent element (such as a sentence, term or phrase) of one of two documents or document sets that have been compared, thereby enhancing the performance of the natural language processing.

[0003] 2. Description of the Related Art

[0004] A process in which two documents or document sets are compared so as to extract the different parts between them is important in multi-document summarization. With regard to the following discussion, the document from which the different parts are extracted shall be called the “target document”, while the other document with which the target document is compared shall be called the “comparison document”. It has heretofore been a common practice to divide both the target document and the comparison document into small elements, and to collate the resulting elements and to identify the elements having no correspondence, as the different parts. The element can be a sentence, a paragraph, and each individual domain in the case where the document has been divided at the change points of topics extracted automatically. In such a case, vector space models are often employed for the collation of the elements. In a case where each element is represented by a vector space model, the components of the vector correspond to individual terms occurring in the document, and the frequency of the corresponding term in the element, or a quantity associated therewith is given as the value of each of the vector components.

[0005] The cosine similarity between the vectors can be employed for judging whether the correspondence between the elements is good or bad. The elements are judged to correspond to each other when the cosine similarity is higher than a predetermined threshold. Accordingly, an element of the target document whose similarities to all of the elements of the comparison document are less than the threshold is regarded as the different part. In another known method, after both documents have been represented by graphs, the corresponding relationships of graph elements are found so as to obtain the different parts from the graph elements having no correspondence.

[0006] There are two techniques for the extraction of the different parts:

• • [0007] (A) Extracting any part in which expressed information differs.
  • [0008] (B) Extracting any part that reflects a difference of concepts expressed in the documents by both documents.

[0009] Many prior-art methods of multi-document summarization are based on technique (A). The different parts between both the documents are extracted, and the importance of each different part in the target document is not evaluated. Consequently, a part that is not very important as information can be extracted as the different part merely because the part differs from the comparison document. From technique (B), the present invention makes possible the extraction of any different part that satisfies the following conditions:

[0010] The different part extracted from the target document is also an important part in the target document. That is, the difference and importance balance. The different part satisfying this condition is more appropriately expressed as a “distinctive part” in the target document, rather than merely the different part. Therefore, a different part satisfying this condition shall be hereinafter called the “distinctive part”.

[0011] An evaluation value can be calculated as to the extent of distinctiveness for each sentence of the target document.

[0012] An evaluation value can be calculated as to the degrees of distinctiveness of terms or term series for the extracted distinctive part so as to identify what term or term series forms a main factor.

[0013] One method of evaluating a degree of distinctiveness of a target document that satisfies the above-noted conditions is as follows. One embodiment in accordance with the present invention is a method of extracting document segments having high degrees of distinctiveness from within the target document. First, both the target document and a comparison document are divided into document segments, and a vector of each document segment whose components are the frequencies of terms occurring in the document segment is obtained. Since the most natural document segment is a sentence, the document segment is assumed to be a sentence hereinafter. Thus, both the documents are represented as sets of sentence vectors. Subsequently, there is found a projection axis which maximizes a ratio (squared sum of projected values originating from the target document)/(squared sum of projected values originating from the comparison document) when all the sentence vectors of both the documents are projected on a certain projection axis. Regarding such a projection axis, the squared sum of the projected values of the sentence vectors of the target document becomes large, whereas the squared sum of the projected values of the sentence vectors of the comparison document becomes small. Therefore, information that exists to a great extent in the target document and that barely exists in the comparison document is reflected. As a result, when the sentence vectors are projected on the projection axis, the absolute values of the projected values become large in the target document for sentences that differ in contents from the comparison document, and they can be adopted as the base of the calculation of the degrees of distinctiveness of the individual sentences of the target document.

[0014] Another embodiment in accordance with the present invention is a method of selecting terms having a high degree of distinctiveness. Regarding such terms, the correlations between the frequencies of a target term in individual sentences and the degrees of distinctiveness of the respective sentences are calculated, and any term having a high correlative value is selected. Since such a term should occur only in sentences having a high degree of distinctiveness, such a term can be regarded as a distinctive term. Accordingly, the degree of distinctiveness of the term is calculated on the basis of the correlative value.

[0015] The degrees of distinctiveness of term series, such as phrases or patterns, occurring in the target document can be evaluated by a method similar to the method used for sentences or terms. By way of example, a vector in which its components corresponding to terms contained in a target term series take a value “1”, whereas its other components take a value “0” is obtained for each term series, and the degree of distinctiveness of each term series can be calculated by the method employed for calculating the sentence degree of distinctiveness. Alternatively, when the frequencies of each term series in individual sentences are calculated beforehand, the degree of distinctiveness of the term series can be evaluated by replacing the frequencies of each term with those of each term series in the method employed for calculating the term degree of distinctiveness.

[0016] Furthermore, a method of evaluating the degree of distinctiveness of a target document in accordance with another embodiment of the present invention is as follows. A sentence is assumed to be a document segment. The processing steps up to the obtaining of the vectors of the document segments are the same as the above-noted embodiment. Thereafter, a similarity to the whole target document and a similarity to the whole comparison document are calculated for each sentence of the target document. Any important sentence in the target document has a high degree of similarity to the entire target document, and any sentence having a content of which differs from that of the comparison document has a low degree of similarity to the entire comparison document. Therefore, a degree of distinctiveness in which difference and importance are well balanced can be defined by employing a ratio (similarity to the entire target document)/(similarity to the entire comparison document). Furthermore, the degree of distinctiveness of a term can be calculated by calculating the correlations between the degrees of distinctiveness of individual sentences and the frequencies of the term in the respective sentences. Also, the degree of distinctiveness of each term series can be calculated by obtaining a vector from the term series and then calculating the similarities of the vector to the entire target document and to the entire comparison document as in the above-noted embodiment. Alternatively, the degree of distinctiveness of each term series can be calculated from the correlations between the frequencies of the term series in individual sentences and the degrees of distinctiveness of the respective sentences.

[0017] According to the present invention, when two documents are compared, the degrees of distinctiveness can be calculated for individual sentences, phrases or words that constitute a target document that is one of the two documents. In a case where a comparison document is the other document and both the comparison document and the target document are, for example, newspaper accounts stating an identical case, sentences having high degrees of distinctiveness are selected from the target document, and accordingly, sentences stating topics different from those of the comparison document can be identified. For example, in a case where, regarding a certain traffic accident, the “outline of the accident” and the “assailant and victim” are stated in the comparison document, while the “view of the police”, etc. are stated in addition to the “outline of the accident” in the target document, the degrees of distinctiveness of sentences concerning the “view of the police” are high in the target document, and a part associated with the “view of the police” can be identified. If a user has already looked over the comparison document, he/she is permitted to identify and read only the part of the “view of the police” unknown to him/her. Thus, the efficiency of the acquisition of information can be enhanced. When the target document and the comparison document are the respective sets of replies from a certain population and from another population in a survey, the tendency of distinctive replies in the population of the target document can be grasped by applying a method in accordance with an embodiment of the present invention. In this manner, owing to the application of the present invention, information from the target document can be easily identified and analyzed.

[0023] FIG. 1 is a block diagram of a computer including a document reader 110 for reading both a comparison document and a target document. A data processor 120 receives an output from the document reader 10 and performs term detection, morphological analysis, document segmentation, etc. of the inputted documents, etc. Processor 120 supplies the detected, analyzed, and segmented documents to a selection data processing block to a selection engine 130 and, which selects a document segment having a high degree of distinctiveness or a term having a high degree of distinctiveness from within the target document. The engine 130 replies the document segment selected to a device 140, for example, a computer display or printer, which outputs the selected distinctive document segment or distinctive term. The computer of FIG. 1 can be a general-purpose computer or a computer dedicated to perform the operations described in connection with FIGS. 2–5.

[0024] The following is a description of a method the computer of FIG. 1 performs in response to execution of a program which the computer stores to extract any document segment having a high degree of distinctiveness from within a target document. FIG. 2 is a flowchart of the operations the computer of FIG. 1 performs in sequence for evaluating the degree of distinctiveness of a document segment. This method can be practiced in such a way tat a program having the embodiment incorporated therein is run on a general-purpose computer. Referring to FIG. 2, numeral 11 designates a comparison/target document input step performed by the document reader 110, numeral 12 designates a term detection step performed by the data processor 120, numeral 13 designates a morphological analysis step, and numeral 14 designates a document segmentation step also performed byte data processor 120. Furthermore, numeral 15 designates a document segment vector construction step, numeral 16 designates a topic difference factor analysis step, numeral 17 designates a document segment vector projection step, numeral 18 designates a document segment distinctiveness calculation for each order of topic difference factor step, numeral 19 designates a total document segment distinctiveness degree calculation step. and numeral 20 designates a distinctive document segment selection step. Selection engine 130 performs each of steps 15–20. The following description of how the computer of FIG. 1 operates to perform the steps of Fig 2 uses English language documents as an example.

[0025] First, a target document and a comparison document are inputted in the comparison/target document input step 11. In the term detection step 12, processor 120 detects words, formulas, a series of symbols, etc. from both documents. Hereinafter, all of the words, the series of symbols, etc. are generically referred to as “terms”. In the case of English language writing, the notation method wherein the terms are spaced apart has been established, so that the detection of the terms is easy. Subsequently, both comparison and target documents are morphologically analyzed in the morphological analysis step 13, for example, a tag indicating the part-of-speech is added to each term. In the document segmentation step 14, both documents are divided into document segments. The most fundamental unit of the document segments is a sentence. In case of the English language writing, a sentence is easily extracted since a sentence ends in a period, followed by a space. Other document segmentation methods include (1) separating a complex sentence into a principal clause and a subordinate clause, (2) collectively dividing a plurality of sentences into document segments so as to have a substantially equal number of terms, and (3) dividing a document, from the beginning thereof, into segments having the same number of terms, irrespective of sentences, and so forth.

[0026] In the document segment vector construction step 15,selection engine 130 determines (1) the number of components of a vector that is to be constructed from the terms occurring in the whole document and (2) the correspondence between the respective components and the individual terms. The components of the vector need not correspond to all the types of occurring terms, but rather the vector is constructed from only selected terms, for example, nouns and verbs, by utilizing the part-of-speech tagging results of step 13. In step 15, selection engine 130 constructs the document segment vector by obtaining the types of terms occurring in each document segment, and the frequencies thereof, and determines the values of the corresponding vector components by multiplying the frequencies by weights. The weights can be selected using known prior-art techniques.

[0027] In the topic difference factor analysis step 16, selection engine 130 finds a projection axis that maximizes the ratio of the comparison and target documents by using the squared sums of the projected values of all the document segment vectors. Hereinafter, a sentence is assumed to be the document segment. Consider documents D and T for which the set of the occurring terms is given by {w₁, . . . , w_J}, and which are respectively composed of M and N sentences. The document D is the target document, while the document T is the comparison document. The respective documents are represented by sets of sentence vectors, and the sentence vectors of the kth sentences of the respective documents are indicated by d_k=(d_kl, . . . , d_kJ)^T and t_k=(t_kJ, . . . , t_kl)^T, where superscript T denotes a transposition of the vector. FIGS. 5A–5D show concept diagrams when the document segment is a sentence. The target document D is composed of M sentences (FIG. 5A), and the sentence vector d_k is constructed from the kth sentence (FIG. 5B). The component of vector d_k corresponding to the term w_j is indicated as d_kj. Since the component d_kj expresses the frequency of the term w_j in the kth sentence, it takes each exemplified value. FIGS. 5C and 5D refer to the comparison document. Let symbol α denote a projection axis to be found. Assume that ∥α∥=1, where ∥α∥ denotes the norm of α. Letting P_D and P_T denote the squared sums of the projected values when respectively projecting all of the sentence vectors of the documents D and T on the axis α, the projection axis to be found is given as α which maximizes the ratio or criterion J(α)=P_D/P_T. The squared sums P_D and P_T are expressed follows:

[0028] $\begin{array}{cc}{P}_D=\sum _{k=1}^M{\left(d_k^T\alpha \right)}^2={\alpha }^T{S}_D\alpha ,& \mathrm{Equation}1\\ S_D=\sum _{k=1}^M{d}_k{d}_k^{T}x,& \mathrm{Equation}2\\ P_T=\sum _{k=1}^N{\left(t_k^T\alpha \right)}^2={\alpha }^T{S}_T\alpha ,& \mathrm{Equation}3\\ S_T=\sum _{k=1}^N{t}_k{t}_k^T.& \mathrm{Equation}4\end{array}$
Therefore, the criterion J(α) can be written as:

[0029] $\begin{array}{cc}J\left(\alpha \right)=\frac{P_D}{P_T}=\frac{{\alpha }^T{S}_D\alpha }{{\alpha }^T{S}_T\alpha }.& \mathrm{Equation}5\end{array}$

[0030] The axis α that maximizes the criterion J(α) given by Equation 5 can be obtained such that a value obtained by differentiating J(α) with respect to α is made equal to 0 (zero). This axis is given as the eigenvector of a generalized eigenvalue problem indicated by:
S_Dα=λS_Tα.  Equation 6

[0031] These operations correspond to obtaining a projection axis which maximizes the ratio (squared sum of the projected values originating from the target document)/(squared sum of the projected values originating from the comparison document) when all of the sentence vectors of both documents are projected on a certain projection axis. Regarding such a projection axis, the squared sum of the projected values of the sentence vectors of the target document becomes large, whereas the squared sum of the projected values of the sentence vectors of the comparison document becomes small. Therefore, information that is prevalent in the target document and is not prevalent in the comparison document is reflected. In general, a plurality of values can be calculated as each of the eigenvalue and eigenvector of equation 6. The eigenvalue and eigenvector of ith order are respectively denoted by λ_i and α_i. The eigenvector of the ith order can be regarded as expressing an ith factor which reflects information existing in the target document D and not existing in the comparison target T. Accordingly, the eigenvector α_i of the ith order shall be called the “ith-order topic difference factor vector” of the target document D. Selection engine 130 calculates this topic difference factor vector in step 16 (topic difference factor analysis). Since λ_i=α_i^TS_Dα_i/α_i^TS_Tα_i, λ_i is the value of the criterion when using α_i.

[0032] Accordingly, the extents of the differences between both documents as reflected on the topic difference factor vectors of respective orders differ from one another. Therefore, the topic difference factor vectors of the respective orders are preferably weighted in accordance with the extents of the differences. The weighting is performed by determining the norm of the ith-order eigenvector α_i so as to become:
α_i^TS_Tα_i=1  Equation 7
Then,
α_i^TS_Dα_i=λ_i  Equation 8

[0033] The squared sum of the inner products between the eigenvectors α_i and the respective sentence vectors of the target document D becomes equal to λ_i. In the case of Equation 6, a matrix S_T must be a regular matrix in order for the eigenvector to be calculated. In actuality, however, the matrix S_T cannot be obtained as a regular matrix when the number of sentences is smaller than the number of terms in the comparison document, or where specified terms co-occur in pairs at all times. In such a case, the eigenvector can be calculated by regularizing the matrix S_T in conformity with the following equation:
Ŝ_T=S_T+β²1  Equation 9

• • [0034] wherein β² denotes a parameter, and 1 denotes the identity matrix.

[0035] In the case of Equation 7, the criterion J(α) is equivalently reduced as follows:
j(α)=P_D/(P_T+β²).  Equation 10

[0036] In step 17, during which the selection engine 130 determines the document segment vector projection, each sentence vector of the target document is projected on the topic difference factor vector of each order to calculate a corresponding projected value. The projected value of the sentence vector d_k of the sentence k of the target document on the ith-order topic difference factor vector α_i is denoted by y_ki, and is calculated by the engine 130 using the following equation:
y_ki=α_i^Td_k.  Equation 11

[0037] Since, however, the projected value thus defined tends to become larger for a longer sentence, normalization of the projected value based on ∥d_k∥ can be performed to make the projected value independent of the length of the sentence. In this case, the projected value y_ki is given by:
y_ki=α_i^Td_k/∥d_k∥.  Equation 12

[0038] In step 18, engine 130 calculates the document segment distinctiveness for each order of the topic difference factor, the ith-order degree of distinctiveness distinc(d_k, i) of the sentence vector d_k is calculated on the basis of the projected value y_k. In general, the projected value y_ki takes a plus or minus value. The absolute value of the projected value y_ki increases as the content of the sentence k gets closer to that of the target document D and is more different from that of the comparison document T. Therefore, the distinctiveness degree distinc(d_k, i) can be defined as:
distinc(d_k,i)=y_ki²  Equation 13
or
distinc(d_k,i)=|y_ki|.  Equation 14

[0039] Engine 130 calculates the ith-order distinctiveness degree, indicated by Equation 13, using only the ith factor. Engine 130 calculates the degrees of distinctiveness based on a plurality of factors to accurately express the distinctiveness of the sentence k. For this purpose, engine 130 calculates, in step 19, the total document segment degree of distinctiveness of the sentence k in conformity with:

[0040] $\begin{array}{cc}\mathrm{distinc}\left(d_k\right)=\sum _{i=1}^L\mathrm{distinc}\left(d_k,i\right).& \mathrm{Equation}15\end{array}$

[0041] The variable L indicates the number of topic difference factor vectors used in the calculation of the sentence distinctiveness, and a suitable value must be experimentally determined. The maximum value of L is the number of eigenvalues whose values become equal to or greater than 1 (one).

[0042] In step 20 of the distinctive document segment selection, engine 130 selects any distinctive sentence in the target document on the basis of the degrees of distinctiveness of the respective orders and the total degree of distinctiveness as previously calculated. The selection proceeds as described below. In the simple method engine 130 selects any sentence having a total degree of distinctiveness equal to or above a predetermined value.

[0043] A more complex method employs the topic difference factor vector of a specified order. Engine 130 initially classifies individual sentences into a list group in which the projected values of the respective sentence vectors onto the topic difference factor vector become positive, and a second group in which they become negative. Subsequently, any sentences whose degrees of distinctiveness of each order are equal to or greater than the predetermined value are selected from the respective groups. These operations are executed for all the topic difference factor vectors up to the predetermined order L, and duplicate sentences are excluded, whereby the distinctive sentences are selected. The Engine 130 can select distinctive sentences by either the simple or complex method.

[0044] Moreover, according to the program of FIG. 2, not only the degrees of distinctiveness of the document segments, but also those of the combinations of terms, such as phrases, term groups under dependency relationships, or term series patterns, are evaluated as stated below. In, for example, an expression “game of soccer played at Yokohama”, “of soccer” modifies the noun “game”, and hence, “game of soccer” becomes a noun phrase. Further, “played at Yokohama” modifies the noun phrase “game of soccer”, and hence, the above expression becomes a noun phrase as a whole. In more detail, “at Yokohama” modifies the verb “played”, and hence, “played at Yokohama” becomes a term group under a dependency relationship. Besides, in a case where expressions “game of soccer played at xx” occur repeatedly as various place names are included as xx, “game of soccer played at” becomes a term series pattern.

[0045] In step 13, in addition to the execution of the morphological analysis, processor 120 extracts the combinations of terms to be evaluated. Phrases, or term groups under dependency relationships are extracted by executing a syntactic analysis. Various methods have already been devised for the extraction of term series patterns occurring frequently, and they can be employed without any problem. In step 15, a vector p=(p_i, . . . , p_j)^T is constructed for each combination of terms to-be-evaluated, in addition to the document segment vectors used in step 16. The vector p is a vector in which its components corresponding to terms contained in the combination of terms take a value “1”, whereas its other components take a value “0”. A practical example of the vector p is explained as follows. In the case of the expression “game of soccer played at Yokohama”, the vector p becomes a vector in which only the components corresponding to terms “Yokohama”, “played”, “soccer” and “game” take a value “1”, whereas the other components take a value “0”. Using such a vector p instead of the sentence vector d_k in steps 17, 18 and 19, engine 130 calculates the degrees of distinctiveness of the combination of terms to be evaluated. In step 20, accordingly, distinctive combinations of terms likewise selects the engine 130 in the distinctive sentences.

[0046] As a second embodiment, there will be described a method of selecting any term having a high degree of distinctiveness from within a target document. Regarding terms, the correlations between the frequencies of a target term in individual sentences and the degrees of distinctiveness of the respective sentences are calculated, and any term having a high correlative value is selected. The degree of distinctiveness of the term is calculated on the basis of the correlative value. FIG. 3 is a flowchart of a second embodiment in accordance with the present invention for evaluating the degree of distinctiveness of a term. This method can be practiced in such a way that a program having the embodiment of the present invention contained therein is run on a general-purpose computer. Referring to the flowchart of FIG. 3, numeral 11 designates the step of comparison/target document input, numeral 12 designates the step of term detection, numeral 13 designates the step of morphological analysis, and numeral 14 designates the step of document segmentation. Furthermore, numeral 15 designates the step of document segment vector construction, and numeral 16 designates the step of topic difference factor analysis. Still furthermore, numeral 27 designates the step of document segment vector projection, numeral 28 designates the step of term degree of distinctiveness calculation for each order of topic difference factor, numeral 29 designates the step of total term degree of distinctiveness calculation, and numeral 30 designates the step of distinctive term selection. Among the above steps, elements 11–16 are the same as those shown in FIG. 2.

[0047] An example in which a document segment is assumed to be a sentence will be described as in the case of FIG. 2. At the document segment vector projection step 27, all the sentence vectors of the comparison document T are projected in addition to the projection of the sentence vector of the target document D in step 17 in FIG. 2. The projected value of the sentence vector t_k of the comparison document T on the ith-order topic difference factor vector α_i, denoted by z_ki is calculated by:
z_ki=α_i^Tt_k  Equation 16
or
z_ki=α_i^Tt_k/∥t_k∥.  Equation 17

[0048] In step 28 of the term degree of distinctiveness calculation for each order of topic difference factor, the correlation between the projected values of individual sentences and term frequencies in the respective sentences is first calculated. Here, correl(w_j, i) denotes the correlation coefficient between the values of those components of sentence vectors in the target document and the comparison document which correspond to a jth term w_j and the projected values of the respective sentence vectors on the ith-order topic difference factor vector α_i. The jth components of the sentence vectors d_k and t_k are d_kj and t_kj, and the projected values thereof on the vector α_i are y_ki and z_ki, respectively, so that the correlation coefficient can be calculated by:

[0049] $\begin{array}{cc}\mathrm{correl}\left(w_j,i\right)=\frac{\left(\sum _{k=1}^M{y}_{\mathrm{ki}}{d}_{\mathrm{kj}}+\sum _{k=1}^N{z}_{\mathrm{ki}}{t}_{\mathrm{kj}}\right)}{(\sqrt{\sum _{k=1}^M{y}_{\mathrm{ki}}^2+\sum _{k=1}^N{z}_{\mathrm{ki}}^2}\sqrt{\sum _{k=1}^M{d}_{\mathrm{kj}}^2+\sum _{k=1}^N{t}_{\mathrm{kj}}^2}}.& \mathrm{Equation}18\end{array}$

[0050] The correlation coefficient increases for the term w_j when a proportional relationship holds between the value of the component corresponding to the term w_j in the sentence vector d_k or t_k and the projected value of the sentence vector on the topic difference factor vector α_i. That is, the correlation coefficient increases when the ith-order degree of distinctiveness of the sentence becomes large when the term w_j occurs and becomes small when the term w_j does not occur. In such a case, the term w_j can be regarded as a distinctive term that governs the ith-order degree of distinctiveness of each sentence. Accordingly, the term degree of distinctiveness of ith order denoted by distinc(w_j, i) can be calculated in step 28 by:
distinc(w_j,i)=correl(w_j,i)²  Equation 19
or
distinc(w_j,i)=|correl(w_j,i)|.  Equation 20

[0051] In step 29 of the total term degree of distinctiveness calculation, a total degree of distinctiveness for each term is calculated by combining a plurality of factors as in the case of FIG. 2. The total degree of distinctiveness of the term w_j denoted by distinc(w_j) can be calculated in step 29 by:
distinc(w_j)=Σ_i=1^L distinc(w_j,i).  Equation 21

[0052] In step 30 of the distinctive term selection, any term distinctive in the target document is selected on the basis of the degrees of distinctiveness of the respective orders and the total degree of distinctiveness as have been calculated. The selection can proceed as stated below. The simplest method is a method in which any term having a total degree of distinctiveness equal to or greater than a predetermined value is selected. Also, the ensuing method is possible. First, as to the topic difference factor vector of specified order, individual terms are classified into a group in which the correlation coefficients between the projected values of the respective sentence vectors onto the topic difference factor vector and the frequencies of the respective terms become positive, and a group in which they become negative. Subsequently, any terms whose degrees of distinctiveness of each order are equal to or greater than the predetermined value are selected from the respective groups. These operations are executed for all the topic difference factor vectors up to the predetermined order L, and duplicate terms are excluded, whereby the distinctive terms are selected. The distinctive terms can be selected by either of the methods.

[0053] Moreover, according to the second embodiment in accordance with the present invention, not only the degrees of distinctiveness of the terms, but also those of the combinations of terms, such as phrases, term groups under dependency relationships, or term series patterns, can be evaluated as stated below. As in the first embodiment, in step 13, in addition to the execution of the morphological analysis, the combinations of terms to be evaluated are extracted. Phrases, or term groups under dependency relationships can be extracted by executing a syntactic analysis. Various methods have already been devised for the extraction of term series patterns occurring frequently, and they can be employed without any problem. In step 15, frequencies at which each combination of terms to-be-evaluated occurs in the respective document segments are calculated, in addition to the construction of the document segment vectors for use in step 16. Here, P_Dk denotes the frequency in the sentence k of the target document D, and P_Tk denotes the frequency in the sentence k of the comparison document T. The degree of distinctiveness of the combination of terms to be evaluated can be calculated instead of that of the term w_j by replacing d_kj with P_Dk and by replacing t_ki with p_Tk in steps 28 and 29. As a result, in step 30, distinctive combinations of terms can be selected likewise to the distinctive terms.

[0054] Next, in order to evaluate the distinctiveness of a target document, a third embodiment in accordance with the present invention operates as follows. The steps of comparison/target document input 11 through document segment vector construction 15 of the third embodiment are common to those of the first and second embodiments. Thereafter, a similarity to the whole target document and a similarity to the whole comparison document are calculated for each sentence of the target document. FIG. 4 is a flowchart showing the third embodiment of the present invention for evaluating the degrees of distinctiveness of a document segment and a term. This method can be practiced in such a way that a program having the embodiment contained therein is run on a general-purpose computer.

[0055] Referring to the flowchart of FIG. 4, numeral 11 designates the step of comparison/target document input, numeral 12 designates the step of term detection, numeral 13 designates the step of morphological analysis, and numeral 14 designates the step of document segmentation. Furthermore, numeral 15 designates the step of document segment vector construction, numeral 36 designates the step of similarity calculation, numeral 37 designates the step of document segment degree of distinctiveness calculation, and numeral 38 designates the step of term degree of distinctiveness degree. Still furthermore, numeral 39 designates the step of distinctive document segment selection, and numeral 40 designates the step of distinctive term selection. Steps 11–15 are the same as those shown in FIG. 2.

[0056] Engine 130, in similarity calculation step 36, calculates the similarities between the individual sentence vectors of the target/comparison documents and the whole target/comparison documents. Letting sim(D, d_k) denote the similarity of the sentence vector d_k of the target document to the whole target document, and sim(T, d_k) denote the similarity thereof to the whole comparison document, sim(D, d_k) and sim(T, d_k) can be respectively calculated on the basis of the squared sums of the inner products between the sentence vector d_k and all the sentence vectors of the target document and the comparison document, as follows:

[0057] $\begin{array}{cc}\mathrm{sim}\left(D,d_k\right)={\left(\frac{\sum _{m=1}^M{\left(d_k^T{d}_m\right)}^2}{\sqrt{\sum _{m=1}^M\sum _{n=1}^M{\left(d_m^T{d}_n\right)}^2}{d_k}^2}\right)}^{1/2}& \mathrm{Equation}22\\ \mathrm{sim}\left(T,d_k\right)={\left(\frac{\sum _{m=1}^N{\left(d_k^T{t}_m\right)}^2}{\sqrt{\sum _{m=1}^N\sum _{n=1}^N{\left(t_m^T{t}_n\right)}^2}{d_k}^2}\right)}^{1/2}.& \mathrm{Equation}23\end{array}$

[0058] Alternatively, letting d and t denote the average sentence vectors of the target document and the comparison document, the above similarities can be respectively calculated as follows:

[0059] $\begin{array}{cc}\mathrm{sim}\left(D,d_k\right)=\frac{d_k^T\stackrel{\_}{d}}{\sqrt{\left({\stackrel{\_}{d}}^T\stackrel{\_}{d}\right)\left(d_k^T{d}_k\right)}},& \mathrm{Equation}24\\ \mathrm{sim}\left(T,d_k\right)=\frac{d_k^T\stackrel{\_}{t}}{\sqrt{\left({\stackrel{\_}{t}}^T\stackrel{\_}{t}\right)\left(d_k^T{d}_k\right)}}.& \mathrm{Equation}25\end{array}$

[0060] Incidentally, at the similarity calculation step 36, similarities to the whole target document and the whole comparison document are also calculated for all the sentence vectors of the comparison document in preparation for the term degree of distinctiveness calculation of step 38. In step 37 of document segment degree of distinctiveness calculation, engine 130 calculates degrees of distinctiveness for all of the sentence vectors of the target document. Any sentence that is important in the target document has a high similarity to the whole target document, and any sentence having a content that differs from that of the comparison document has a low similarity to the whole comparison document. Therefore, a degree of distinctiveness in which difference and importance are well balanced can be defined by employing a ratio (similarity to the whole target document)/(similarity to the whole comparison document). Accordingly, engine 130 in step 37 calculates the degree of distinctiveness distinc(d_k) of the sentence k of the target document D as follows:
distinc(d_k)=sim(D,d_k)/sim(T,d_k).  Equation 26

[0061] The degree of distinctiveness of the sentence k thus calculated becomes large in a case where the sentence k has a high similarity to the target document and a low similarity to the comparison document. Incidentally, in step 37 of the document segment degree of distinctiveness calculation, the sentence degrees of distinctiveness of the comparison document T are also calculated in preparation for the next step 38 of the term distinctiveness calculation. The degree of distinctiveness of the sentence k of the comparison document T shall be denoted by distinc(t_k).

[0062] In step 38, engine 130 performs the term degree of distinctiveness calculation from the correlation coefficients between the degrees of distinctiveness of individual sentences and term frequencies in the respective sentences. In step 38, engine 130 calculates the degree of distinctiveness of a term w_j denoted by distinc(w_j) in accordance with:

[0063] $\begin{array}{cc}\mathrm{distinc}\left(w_j\right)=\frac{\left(\sum _{k=1}^M{d}_{\mathrm{kj}}\mathrm{distinc}\left(d_k\right)+\sum _{k=1}^N{t}_{\mathrm{kj}}\mathrm{distinc}\left(t_k\right)\right)}{\sqrt{\sum _{k=1}^M{d}_{\mathrm{kj}}^2+\sum _{k=1}^N{t}_{\mathrm{kj}}^2}\sqrt{\sum _{k=1}^M{\mathrm{distinc}\left(d_k\right)}^2+\sum _{k=1}^N{\mathrm{distinc}\left(t_k\right)}^2}}.& \mathrm{Equation}27\end{array}$

[0064] The correlation coefficient becomes high for the term w_j when a proportional relationship holds between the value of the component corresponding to the term w_j in the sentence vector d_k or t_k and the degree of distinctiveness of the sentence. That is, the correlation coefficient increases when the degree of distinctiveness of the sentence becomes large when the term w_j occurs and the degree of distinctiveness of the sentence becomes small when the term w_j does not occur. In such a case, the term w_j can be regarded as a distinctive term that governs the degree of distinctiveness of each sentence.

[0065] In steps 39 and 40, engine 130 selects the distinctive document segment and distinctive term, respectively such that any sentence whose degree of sentence distinctiveness is equal to or greater than a predetermined value, and any term whose term degree of distinctiveness is equal to or greater than a predetermined value are selected. Thereby, distinctive sentences and terms can be obtained.

[0066] According to FIG. 4, both the degrees of distinctiveness the document segments and the terms, but the degrees of distinctiveness of the combinations of terms, such as phrases, term groups under dependency relationships, or term series patterns, can be evaluated as stated below. In step 13, in addition to the execution of the morphological analysis, the combinations of terms to be evaluated are extracted. Phrases, or term groups under dependency relationships can be extracted by executing a syntactic analysis. Various methods have already been devised for the extraction of term series patterns occurring frequently, and they can be employed without any problem. In step 15, a vector p=(p_i, . . . , p_i)^T is constructed for each combination of terms to-be-evaluated, in addition to the document segment vectors for use in step 16. The vector p is a vector having components corresponding to terms contained in the combination of terms to be evaluated take a value “1”, and having its other components take a value “0”. Subsequently, in steps 36 and 37, the similarity sim(D, p) between such a vector p and the target document D and the similarity sim(T, p) between the vector p and the comparison document T are calculated by replacing the sentence vector d_k with the vector p. As with equations 20 and 21, these similarities can be defined as follows:

[0067] $\begin{array}{cc}\mathrm{sim}\left(D,p\right)={\left(\frac{\sum _{m=1}^M{\left(p^T{d}_m\right)}^2}{\sqrt{\sum _{m=1}^M\sum _{n=1}^M{\left(d_m^T{d}_n\right)}^2}{p}^2}\right)}^{1/2},& \mathrm{Equation}28\\ \mathrm{sim}\left(T,p\right)={\left(\frac{\sum _{m=1}^N{\left(p^T{t}_m\right)}^2}{\sqrt{\sum _{m=1}^N\sum _{n=1}^N{\left(t_m^T{t}_n\right)}^2}{p}^2}\right)}^{1/2}.& \mathrm{Equation}29\end{array}$

[0068] Alternatively, as with equations 22 and 23, the similarities may well be defined as follows:

[0069] $\begin{array}{cc}\mathrm{sim}\left(D,p\right)=\frac{p^T\stackrel{\_}{d}}{\sqrt{\left({\stackrel{\_}{d}}^T\stackrel{\_}{d}\right)\left(p^{T}p\right)}},& \mathrm{Equation}30\\ \mathrm{sim}\left(T,p\right)=\frac{p^T\stackrel{\_}{t}}{\sqrt{\left({\stackrel{\_}{t}}^T\stackrel{\_}{t}\right)\left(p^{T}p\right)}}.& \mathrm{Equation}31\end{array}$

[0070] Using the similarities, the degree of distinctiveness of the combination of terms to be evaluated can be calculated as follows:
distinc(p)=sim(D, p)/sim(T,p).  Equation 32

[0071] In step 40, those combinations of terms whose degrees of distinctiveness are equal to or greater than a predetermined value are selected as distinctive combinations of terms.

[0072] Furthermore, in this embodiment, the degrees of distinctiveness of phrases, term groups under dependency relationships, or term series patterns each composed of a plurality of terms can be calculated as stated below. In step 15, frequencies at which each combination of terms to-be-evaluated occurs in the respective document segments are calculated, in addition to the construction of the document segment vectors for use in step 16. Here, P_Dk denotes the frequency in the sentence k of the target document D, and P_tk denotes the frequency in the sentence k of the comparison document T. The degree of distinctiveness of the combination of terms to be evaluated can be calculated instead of the degree of distinctiveness of the term w_j by replacing d_ki with P_Dk and by replacing t_ki with P_Tk in step 38. In step 39, those combinations of terms whose degrees of distinctiveness are equal to or greater than a predetermined value are selected as distinctive combinations of terms.

[0073] The results of an experiment utilizing equation 13 follow in order to set for the certain features of the present invention. As data used in the experiment, two documents were selected under the criteria of suitable lengths and a high similarity from the first category “acq” of a text-categorizing corpus “Reuters-21578”. The “id”s of these documents were 1836 and 2375. The cosine similarity between the documents was 0.955. The document 1836 was composed of 43 sentences, while the document 2375 was composed of 32 sentences. The documents were news items on an identical day. The document 2375 thought to have been sent later was set as the target document D, and the document 1836 as the comparison document T, so as to extract distinctive sentences, etc. from the target document D. The contents of the documents relate to the takeover of a US aviation company “USAir” by a US aviation company “TWA”. Sentences D-1 through D-4 summarize news, sentences D-5 through D-24 state the details of a takeover drama, and sentences D-25 et seq. state an analysis about the company “TWA”. Many information items which are not existent in the document T are contained in some of the sentences D-1 through D-4 and D-5 through D-24, and in the sentences D-25 et seq. The whole texts of the documents will be indicated as “experimental document data” at the end of this specification.

[0074] The experiment was conducted in accordance with the first embodiment of the present invention. As a result, eight sentences D-1, D-8, D-11, D-24, D-25, D-27, D-28 and D-30 were selected as sentences of high degrees of distinctiveness. Also in a human reading comparison experiment, these sentences were recognized as sentences that were weakly related to the comparison document and which were distinctive in the target document.

[0075] The results obtained by selecting words of high degrees of distinctiveness in accordance with equation 19 are listed below. For each of the ten words having the high degrees of distinctiveness, the degree of distinctiveness of the word, the occurring frequency thereof in the target document D, and the occurring frequency thereof in the comparison document T is indicated.

[0076]

			Occurring	Occurring
			Frequency	Frequency
	Selected	Degree of	In Target	In Comparison
	Word	Distinctiveness	Document D	Document T
	succeed	85.7	3	0
	work	85.3	2	0
	cost	85.2	2	0
	surviving	81.6	2	0
	clear	80.4	3	0
	company	71.0	5	1
	fall	67.3	2	0
	arbitrager	67.0	5	1
	bid	62.4	5	1
	merge	61.9	3	1

[0077] In view of these results, the words whose occurring frequencies were low in the comparison document T and high in the target document D could be selected.

[0078] As an application of such an experiment, the ensuing example is considered. When a certain news item was read earlier to grasp its content, any key word whose content is not stated in the earlier news item can be extracted from a news item that arrived later. Accordingly, a determination can be made as to whether the later news item needs to be read in detail. Different degrees of distinctiveness can be obtained even for two terms, such as the words “succeed” and “clear” mentioned above, which have quite the same frequencies in the target document and in the comparison document, and which is more distinctive can be judged as the merit of the present invention.

[0079] Experimental Document Data

[0080] The documents used in the present invention are stated below

[0081] Comparison Document T (Reuter-id 1836)

[0082] Trans World Airlines Inc complicated the bidding for Piedmont Aviation Inc by offering either to buy Piedmont suitor USAir Group or, alternatively, to merge with Piedmont and USAir. Piedmont's board was meeting today, and Wall Street speculated the board was discussing opposing bids from Norfolk Southern Corp and USAir. The TWA offer was announced shortly after the Piedmont board meeting was scheduled to begin. TWA offered to buy USAir for 52 dlrs cash per share. It also said it was the largest shareholder of USAir and threatened to go directly to USAir shareholders with an offer for 51 pct of the stock at a lower price. TWA also said it believed its offer was a better deal for USAir shareholders than an acquisition of Piedmont, but it said it alternatively would discuss a three way combination of the airlines. Market sources and analysts speculated that TWA chairman Carl Icahn made the offer in order to put his own airline into the takeover arena. We're just wondering if he's not just trying to get TWA into play.

[0083] There's speculation on the street he just wants to move onto something else, said one arbitrager. We think TWA might just be putting up a trial balloon. Analysts said the offer must be taken seriously by USAir, but that the airline will probably reject it because the price is relatively low compared to other airline deals. They also said Icahn must prove his offer credible by revealing financing arrangements. They need to show their commitment and their ability to finance. I think it's a credible offer, said Timothy Pettee, a Bear Stearns analyst. I think it's certainly on the low end of relative values of airline deals, said Pettee. Pettee estimated 58 dlrs would be in a more reasonable range based on other airline mergers. USAir stock soared after TWA made public its offer.

[0084] A spokesman for USAir declined comment, and said USAir had not changed its offer for Piedmont. USAir offered of buy 50 pct of that airline's stock for 71 dlrs cash per share and the balance for 73 dlrs per share in USAir stock. USAir closed up 5-⅜ at 49-⅛ on volume of 1.9 min shares. Piedmont, which slipped ½ to close at 69-⅝, also remained silent on the TWA action. Piedmont has an outstanding 65 dlr cash per share offer from Norfolk Southern Corp. Norfolk Southern declined comment, but said it stuck with its offer for Piedmont. Norfolk owns about 20 pct of Piedmont and opened the bidding when it said it would propose a takeover of Piedmont. Some analysts said Icahn may be trying to acquire USAir to make his own airline a more attractive takeover target. Icahn I think had wanted to sell his airline and there were no takers. I think the strategy might have called for making his investment more attractive.

[0085] One way to accomplish that specific objective is to go out and acquire other airlines, said Andrew Kin of Eberstadt Fleming. I don't know whose going to buy them, but at least this way it becomes a much more viable package, said Kim. But Icahn's financing ability for such a transaction remains in doubt, in part because of TWA's heavy debt load. Wall street sources said TWA has some cash with which to do the offer. The sources said Icahn has not lined up outside financial advisers and plans to make his own arrangements. Icahn earlier this year abandoned plans to buy USX Corp <X> and still retains 11 pct of that company's stock. Some Wall street sources said the financier's USX plan was impacted by the cloud hanging over his adviser, Drexel Burnham Lambert Inc, because of Wall Street's insider trading scandal. Industry sources also predicted USAir might reject the TWA offer on price and financing concerns. It's littered with contingencies and it doesn't even have a financing arrangement, said one executive at another major airline. But the executive conceded a merged TWA USAir would be a strong contender with USAir's east coast route system and planned west coast presence from PSA. USAir could feed the international flights of TWA, which has a midwest presence in its St. Louis hub. Adding Piedmont, dominant in the southeast, to the mix would develop an even stronger force. The combined entity would also have TWA's pars reservation system. Such a merger would be complex and analysts said it would result in an airline iwth an 18 pct market share.

[0086] Target Document D (Reuter-id2375)

[0087] D-1 Carl Icahn's bold takeover bid for USAir Group <U> has clouded the fate of Piedmont Aviation Inc, which was being courted by USAir.

[0088] D-2 Yesterday, Icahn's Transworld Airlines Inc <TWA> made a 1.4 billion dlr offer for USAir Group.

[0089] D-3 The move complicated a USAir takeover offer for Piedmont, which was believed to be close to accepting the bid.

[0090] D-4 Today, USAir rejected Icahn's 52 dlr per share offer and said the bid was a last minute effort to interfere in its takeover of Piedmont.

[0091] D-5 Icahn was unavailable for comment.

[0092] D-6 Piedmont fell one to 68-⅝ on volume of 963,000.

[0093] D-7 TWA was off ⅜ to 31-½.

[0094] D-8 USAir fell 1-⅜ to 47-¾ as doubt spread it would be taken over.

[0095] D-9 Analysts and market sources view the TWA bid as an attempt to either trigger a counter offer from USAir or to attract a suitor who might want both airlines once they merged.

[0096] D-10 The next move is either Icahn starts a tender offer or Piedmont and USAir announce a deal, speculated one arbitrager.

[0097] D-11 Some arbitragers said there is now some risk in the current price of Piedmont since it is not clear that USAir's bid will succeed.

[0098] D-12 Piedmont's largest shareholder and other suitor, Norfolk Southern Corp <NSC> has offered 65 dlrs per share for the company.

[0099] D-13 USAir offered 71 dlrs cash per share for half of Piedmont stock, and 73 dlrs per share in stock for the balance.

[0100] D-14 Some arbitragers, however, believe the depressed price of Piedmont offers a buying opportunity since the airline is destined to be acquired by someone.

[0101] D-15 USAir, they said, is the least likely to be bought.

[0102] D-16 Icahn, who has long talked about further consolidation in the airline industry, also offered USAir the alternative of a three way airline combination, including TWA and Piedmont.

[0103] D-17 But Wall Street has given little credibility to Icahn's offer, which lacked financing and was riddled with contingencies.

[0104] D-18 Still, he has succeeded in holding up a merger of two airlines both of which analysts said would fit well with TWA.

[0105] D-19 You can't discount him, said one arbitrager.

[0106] D-20 Analysts, however, said Icahn would have to prove he is serious by following through with his threats or making a new offer.

[0107] D-21 In making the offer for USAir, Icahn threatened to go directly to shareholders for 51 pct of the stock at a lower price if USAir rejected his offer.

[0108] D-22 It's clear Icahn wants to sell and he's bluffing, said one arbitrager.

[0109] D-23 Analysts said the 52 dlr per share offer was underpriced by about six dlrs per share.

[0110] D-24 Some analysts believe Icahn's proposed three way airline combination might face insurmountable regulatory hurdles, but others believe it could be cleared if the companies are acquired separately.

[0111] D-25 TWA would have to be the surviving company for the deal to work, said one analyst.

[0112] D-26 Analysts said such a merger would be costly and complicated.

[0113] D-27 TWA has the best cost structure, since Icahn succeeded in winning concessions from its unions.

[0114] D-28 In order for the other carriers to come down to TWA's wage scale in a merger, TWA would have to be the surviving entity, analysts said.

[0115] D-29 Such a move does not necessarily free Icahn of TWA, they said.

[0116] D-30 They said he showed skill in reducing Ozark Airlines' costs when he merged it into TWA last year, and he might be a necessary ingredient for a merger to work.

[0117] D-31 However, other analysts speculated the managements of Piedmont and USAir would not tolerate Icahn as head of a new company.

[0118] D-32 They said a USAir acquisition of TWA might be a way for him to exit the company if USAir's airline is then merged into TWA.