Document classification method and apparatus

[0001] 1. Field of the Invention

[0002] The present invention relates to natural language processing which includes the classification of documents. More particularly, the invention permits one to exactly extract the distinction between document sets, thereby improving the processing performance.

[0003] 2. Description of the Related Art

[0004] Document classification is a technology for classifying documents into predetermined groups, and has become more important with an increase in the circulation of information. Regarding the document classification, various methods, such as the vector space model, the k nearest neighbor method (kNN method), the naive Bayes method, the decision tree method, the support vector machines method, and the boosting method, have heretofore been studied and developed. A recent trend in document classification processing has been detailed by M. Nagata and H. Hira in TEXT CATEGORIZATIONSAMPLE FAIR OF LEARNING THEORIES contained in Proceedings of the Information Processing Society of Japan, Vol. 42, No. 1 (January 2001). In any of the classification methods, information on a document class is described in any form and is collated with an input document. It shall be called the class model below. The class model is expressed by, for example, the average vector of documents belonging to each class in the vector space model, the set of the vectors of documents belonging to each class in the kNN method, and a set of simple hypotheses in the boosting method. In order to achieve a precise classification, the class model must precisely describe each class. It may be said that, in the high-performance classification methods hitherto proposed, the class model describes each class more precisely.

[0005] In this regard, although many of the classification methods aim at the preciseness of the description of the class model, they do not consider class-model overlapping. In the vector space model or the kNN method, for example, the class model of a certain class also includes information matching with another class. If an overlap exists between the class models, there is a likelihood that it will exist between a certain input document and the class to which the input document does not belong, and can cause a misclassification. In order to eliminate the cause for the misclassification, the class model needs to be described by finding the distinctive information of each class so that class-model overlapping may be reduced.

[0006] In view of the above, according to the present invention, a technique is provided for extracting features which occur in each given class, but which rarely occur in any other class, and features which occur in any other class, but which rarely occur in the given class. A classification scheme including two stages, a main and a sub classifier is constructed so that such features can be effectively used. In the main classification scheme, an existing high performance classification method is adopted, while in the sub classification scheme, the features are used. The main classification scheme is assumed to classify an input document on the basis of the similarity between the input document and each class.

[0007] As described below, the features to be used in the sub classification scheme are extracted using all training documents with labels indicating the classes of each individual document. First, in the main classification scheme, for each of the training documents, the similarities are obtained for each class. Documents whose similarities to a pertinent class exceed a preset threshold are judged as belonging to the pertinent class. These documents are divided into a set in which the documents are correctly classified into their proper classes (hereinbelow, called a given class document set), and a set in which the documents are classified into the given class document set in spite of belonging to other classes (hereinbelow, called a competing document set). Each document is expressed by a set of sentence vectors. Each component of a sentence vector is the frequency of each term occurring in the pertinent sentence, or a quantity corresponding to the frequency, while the dimension thereof is the number of kinds of terms occurring in all of the training documents, or the number of kinds of selected terms. Assume that all of the sentence vectors of all of the documents are projected onto a certain projection axis. The ratio between the sum of the squared projection values from a given class document set and the sum of the squared projection values from the competing document set is preferably adopted as a criterion function that indicates the degree of differences between the sets. The features for use in the sub classification scheme are extracted using the projection axis that maximizes the criterion function.

[0008] A plurality of such projection axes can be represented as the eigenvectors of a generalized eigenvalue problem. More specifically, when the criterion function is expressed by the (sum of the squared projection values from the given class document set)/(sum of the squared projection values from the competing document set), the projection axis which maximizes the criterion function has a large value as the sum of the squared projection values from the given class document set and a small value as the sum of the squared projection values from the competing document set. Therefore, the projection axis reflects information that rarely occurs in any competing document and often occurs in the given class. Accordingly, such projection axes are referred to as positive topic difference factor vectors. In contrast, when the criterion function is expressed by the (sum of the squared projection values from the competing document set)/(sum of the squared projection values from the given class document set), the projection axis which maximizes the criterion function reflects information which rarely occurs in the given class and often occurs in any competing document. Accordingly, such projection axes are referred to as negative topic difference factor vectors.

[0009] In the sub classification scheme, the weighted sum of the squared inner products between the sentence vectors of the input document and a certain number of positive topic difference factor vectors of each class is added to the similarity of the pertinent class obtained in the main classification scheme. The weighted sum of the squared inner products between the sentence vectors of the input document and a certain number of negative topic difference factor vectors of each class is similarly subtracted from the similarity of the pertinent class. The thus corrected similarity is compared with a predetermined threshold for each class.

[0010] As stated before, in the present invention, the similarities calculated by the main classification scheme are corrected by the sub classification scheme. If the weighted sum of the squared inner products between the sentence vectors of the input document and the certain number of positive topic difference factor vectors is calculated in a certain class by the sub classification scheme, the positive topic difference factor vectors stipulate features which exist in the class. Therefore, if the input document belongs to the pertinent class, the above weighted sum usually has a large value, and the similarity is corrected into a large value. On the other hand, if the input document does not belong to the pertinent class, the above weighted sum often has a small value, and the similarity change is small. Further, if the weighted sum of the squared inner products between the sentence vectors of the input document and the certain number of negative topic difference factor vectors is calculated in the certain class, the negative topic difference factor vectors stipulate features which ought not to exist in the class. Therefore, if the input document belongs to the pertinent class, the above weighted sum often has a small value and the similarity change is small. However, when the input document does not belong to the pertinent class, the above weighted sum often has a large value and the similarity is corrected into a small value. Since the similarity is corrected in this manner, the corrections often result in enlarging the similarity to the class to which the input document belongs, and reducing the similarity to the class to which the input document does not belong. Accordingly, the classification precision is improved.

[0011] The foregoing and a better understanding of the present invention will become apparent from the following detailed description of the example embodiments and the claims when read in connection with the accompanying drawings, all forming a part of the disclosure of this invention. While the foregoing and following written and illustrated disclosure focuses on disclosing example embodiments of the invention, it should be clearly understood that the same is by way of illustration and example only and that the invention is not limited thereto. The spirit and scope of the present invention are limited only by the terms of the appended claims.

[0012] The following represents brief descriptions of the drawings, wherein:

[0013] FIG. 1 is a block diagram showing a document classification apparatus according to an embodiment of the present invention;

[0014] FIG. 2 is a flowchart of an embodiment of the present invention;

[0015] FIGS. 3A-3C are diagrams for explaining a document vector;

[0016] FIG. 4 is a flowchart of the steps for calculating the similarity of an input document (step 14 in FIG. 2) in conformity with the kNN method;

[0017] FIG. 5 is a flowchart of the steps for obtaining positive and negative topic difference factor vectors in order to correct the similarity, by using the document set of a given class and a document set which was misclassified into the given class or which was likely to be misclassified thereinto;

[0018] FIGS. 6A-6C are diagrams of the construction of documents belonging to a class 1; and

[0019] FIG. 7 is a flowchart of steps for classification (step 22 in FIG. 5).

[0020] Before beginning a detailed description of the subject invention, mention of the following is in order. When appropriate, like reference numerals and characters are used to designate identical, corresponding, or similar components in differing drawing figures. Furthermore, in the detailed description to follow, example sizes/models/values/ranges are given, although the present invention is not limited thereto. Arrangements are shown in block diagram form in order to avoid obscuring the invention and also in view of the fact that specifics with respect to implementation of such block diagram arrangements are highly dependent upon the platform within which the present invention is to be implemented, that is, such specifics are well within the purview of one skilled in the art. Where specific details, such as circuits or flowcharts, are set forth in order to describe example embodiments of the invention, it should be apparent to one skilled in the art that the invention can be practiced without, or with variations of, these specific details. Finally, it should be apparent that differing combinations of hard-wired circuitry and software instructions can be used to implement embodiments of the present invention, that is, the present invention is not limited to any specific combination of hardware and software.

[0021] FIG. 1 is a block diagram of a document classification apparatus according to an embodiment of the present invention. First, a document to be classified is inputted to a document input unit 110. In a data processing unit 120, the inputted document is subjected to data processing, such as term extraction and document segment extraction. In a classification engine 130, a similarity is calculated by a main classification scheme, and it is corrected by a sub classification scheme, with reference to a unit 140 that includes classification information for each class. A class to which the inputted document belongs, is determined using the corrected similarity and is outputted to a classification output unit 150.

[0022] FIG. 2 is a flowchart of the processing steps the apparatus of FIG. 1 performs, from document input to class decision. The document is supplied to unit 110 during input step 11. During step 12, unit 120 extracts and selects terms. During step 13, unit 120 extracts a document segment vector. During steps 14 and 15, engine 130 respectively performs similarity calculation and similarity correction. During step 16, unit 140 makes a class decision. The steps 11 to 14 correspond to the main classification scheme, while the steps 15 and 16 correspond to the sub classification scheme. An example is described below using an English language document.

[0023] First, the document to be classified is inputted at the document input step 11. At the term extraction and selection step 12, words, equations, a series of symbols, etc. are extracted from the document. All the words and series of symbols are hereinafter referred to as terms. In case of the written English language, a notation method in which the words are separately written has been established, and hence, the detection of the terms is easy. At the term extraction and selection step 12, terms that are included in a term list for use in classification are extracted from among the terms that occur in the input document. The selection of the terms for use in the classification can be effected using a large quantity of labeled training documents, and a tf-idf (term frequency-inverse document frequency) technique, a method employing ₂ statistics, a method employing mutual information, etc. are examples of known methods affording favorable results. The document segment vector extraction step 13 divides the document into document segments, and creates a vector for every document segment. The most fundamental processing in the division of the document into the document segments is division into sentence units. In the written English language, a sentence ends in a period and is followed by a space, so that the sentence can be easily extracted. Other methods of dividing a document into document segments include a method in which plural sentences are collected into the document segments so that the numbers of the terms of the document segments are substantially equal, and a method in which the document is divided from its head irrespective of sentences so that the numbers of terms included in the document segments are substantially equal, etc. It is also possible to use the whole document as one document segment. Subsequently, a vector is created for every document segment. The components of the vector represent the frequencies of the individual terms for use in the classification, in the pertinent document segment. Alternatively, the frequencies are multiplied by weights. There has been research on how to set the weights, and effective methods of setting the weights are known to those skilled in the art. A vector that is generated by adding up all of the document segment vectors is called the document vector. The ensuing description assumes that a sentence vector is the document segment vector. When an input document X consisting of K sentences (FIG. 3A) is inputted, the k-th sentence vector is denoted by x_k (FIG. 3B), and the document vector by x (FIG. 3C). Numerals on the bottom of FIG. 3B exemplify the components of the sentence vector. That is, the numerals indicate the frequencies of the terms corresponding to the respective components of the sentence vector x_k.

[0024] The similarity calculation step 14 (FIG. 2) calculates the similarity of the input document to each class. Various methods are also known for finding the similarity. In case of the vector space model, the mean document vector of each class is found using a training corpus and is saved. Letting the mean vector of class 1 be m₁, the similarity sim(X, 1) of the input document to the class 1 can be expressed by:

[0025] Here, x denotes the norm of x and superscript T represents vector transpose.

[0026] The kNN method performed by the apparatus of FIG. 1 will now be described with reference to the flowchart shown in FIG. 4. In the kNN method, letting Y_t denote the t-th document in a training document set, and letting y_t denote the document vector of the t-th document, the similarity sim(X, Y_t) of the input document X to the document Y_t is obtained by:

[0027] After the similarities of the input document X to all of the training documents have been obtained (step 142), k documents whose similarities to the input document X are the largest are selected (step 144). Thereafter, the k selected documents are sorted for every class according to labels attached to each document (step 146). Subsequently, the similarity sim(X, 1) of the input document to the class 1 is calculated (step 148). The similarity sim(X, 1) is defined as the summation of the similarities of the input document X to the documents sorted into the class 1. That is, it is calculated by: $\begin{array}{cc}\mathrm{sim}\left(X,l\right)=\underset{Y_t{}_l}{}\mathrm{sim}\left(X,Y_t\right)& \left(3\right)\end{array}$

[0028] Here, ₁ denotes the set of the training documents belonging to the class 1 among k documents.

[0029] At the similarity correction step 15 (FIG. 2), the similarity is corrected using positive topic difference factor vectors and negative topic difference factor vectors that have been saved for every class. The positive topic difference factor vectors of the class 1 for use in the similarity correction are denoted by _i(i1, . . , L_G), and the negative topic difference factor vectors are denoted by B_i (i1, . . , L_P). Then, a corrected similarity to the class 1 as denoted by sim_C(X, 1) is given by: $\begin{array}{cc}{\mathrm{sim}}_c\left(X,l\right)=\mathrm{sim}\left(X,l\right)+a\underset{i=1}{\overset{L_G}{}}\underset{k=1}{\overset{K}{}}{\left(x_k^T{}_i\right)}^2-b\underset{i=1}{\overset{L_p}{}}\underset{k=1}{\overset{K}{}}{\left(x_k^T{}_i\right)}^2& \left(4\right)\end{array}$

[0030] Note that a and b are parameters of positive values and have been previously determined together with L_P and L_G. The values of the parameters a, b, L_P and L_G can be determined such that performances for a document set not used for the calculations of the vectors _i and _i are found while the values of the respective parameters a, b, L_P and L_G are being changed in succession, and such that the combination of the values affording the maximum F measure is selected. The F measure is defined as follows: $\begin{array}{c}\mathrm{Precision}=\frac{\begin{array}{c}\begin{array}{c}(\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{classes}\mathrm{correctly}\\ \mathrm{assigned}\mathrm{to}\mathrm{each}\mathrm{document}\mathrm{as}\end{array}\\ \mathrm{the}\mathrm{result}\mathrm{of}\mathrm{classification})\end{array}}{\begin{array}{c}\begin{array}{c}(\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{classes}\mathrm{assigned}\mathrm{to}\\ \mathrm{each}\mathrm{document}\mathrm{as}\mathrm{the}\mathrm{result}\end{array}\\ \mathrm{of}\mathrm{classification})\end{array}}\\ \mathrm{Recall}=\frac{\begin{array}{c}\begin{array}{c}(\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{classes}\mathrm{correctly}\\ \mathrm{assigned}\mathrm{to}\mathrm{each}\mathrm{document}\mathrm{as}\end{array}\\ \mathrm{the}\mathrm{result}\mathrm{of}\mathrm{classification})\end{array}}{\begin{array}{c}(\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{classes}\mathrm{to}\mathrm{which}\\ \mathrm{each}\mathrm{document}\mathrm{ought}\mathrm{to}\mathrm{belong})\end{array}}\end{array}$

[0031] The corrected similarity sim_C( (X, 1) is calculated by the following equation: $\begin{array}{cc}{\mathrm{sim}}_c\left(X,l\right)=\mathrm{sim}\left(X,l\right)+\underset{i=1}{\overset{L_G}{}}\underset{k=1}{\overset{K}{}}{a_i\left(x_k^T{}_i\right)}^2-\underset{i=1}{\overset{L_p}{}}\underset{k=1}{\overset{k}{}}{b_i\left(x_k^T{}_i\right)}^2& \left(5\right)\end{array}$

[0032] In this case, a_i and b_i are weights for the i-th positive topic difference factor and the i-th negative one, respectively. When L_P and L_G are given, the optimum values of the weights a_i and b_i can be obtained by employing a linear discriminant analysis. More specifically, a vector of L_PL_G1 dimensions is prepared for every document not used for the calculations of the vectors _i and _i, and (x_k^T_i)² (i1, . . , L_G), (x_k^T_i)² (i1, . . , L_P) and sim(X, 1) are given as components. Subsequently, the linear discriminant analysis is conducted between the document set of the class 1 and a document set belonging to another class, and weights which optimally separate both the document sets are determined for the respective components. The document set belonging to another class signifies documents that belong to another class in which the similarity sim(X, 1) to the class 1 exceeds a certain threshold, as the result of classification at a classification step 22 (FIG. 5). It is generally said that, with the linear discriminant analysis, a projection axis that optimally separates two groups of vector sets can be found. The projection axis is calculated so that a difference vector between the mean vectors of the respective groups is multiplied by an inverse matrix of a matrix in which the covariance matrices of the respective groups have been added. Thereafter, the weights for (x_k^T_i)² (i1, . . , L_G) and (x_k^T_i)² (i1, . . , L_P) are divided by the weight for sim(X, 1), whereby a_i and b_i can be determined, respectively. Such processing is executed for all of the combinations of the values of L_P and L_G, and the values of the weights a_i and b_i affording the best classified results may be adopted.

[0033] At the class decision 16 (FIG. 2), the class to which the input document belongs is determined by a comparison between a predetermined threshold for each class and the corrected similarity. If the corrected similarity for the class 1 is larger than the threshold for the class 1, a determination is made that the input document belongs to the class 1.

[0034] FIG. 5 is a flowchart of the steps for determining the positive topic difference factor vector and the negative topic difference factor vector that serve to correct the similarity at the step 15 in FIG. 2. During step 21, training documents are prepared. During step 22, classification occurs. During step 23, document set editing is completed. During step 24, a topic difference factor analysis is effected.

[0035] At the training document preparation step 21, the set of training documents for determining the positive and negative topic difference factor vectors is prepared, and the document vector and document segment vectors are obtained for each of the documents. At the subsequent classification step 22, each training document is selected as the input document so as to calculate its similarities to all of the other training documents and thereby determine the class to which it belongs (steps 14 and 16 in FIG. 2). All of the training documents are sorted by executing such operations. In this case, however, the similarity correction at the step 15 in FIG. 2 is not executed.

[0036] The classification step 22 in FIG. 5 is described below with reference to the flowchart of FIG. 7.

[0037] Step 221: Data processing such as term extraction and document segment extraction is executed for all of the training documents.

[0038] Step 222: One of the training documents is selected as the input document.

[0039] Step 223: The similarities between the input document and the other training documents are calculated to obtain similarities to respective classes in accordance with Eq. (3).

[0040] Step 224: A determination is made as to whether the similarities to the respective classes have been obtained for all of the training documents.

[0041] Step 225: Documents whose similarities are larger than the threshold of a given class are divided into a document set that includes correctly classified documents and its competing document set that includes misclassified documents.

[0042] Now, the flowchart of FIG. 5 is described in detail. The set of M documents, correctly classified as belonging to the class 1, is denoted by D (FIG. 6A). Assuming that the m-th document D_m of the set D consists of K_D(m) sentences, the k-th sentence vector is denoted by d_mk (FIG. 6B). The competing document set editing step 23 (FIG. 5) creates the set of competing documents each of which was misclassified into class 1 or was likely to be misclassified thereinto, every class on the basis of the classified result at the classification step 22. Any competing document with the class 1 is extracted by selecting a document whose similarity sim(X, 1) to the class 1 exceeds a certain threshold. The threshold can be arbitrarily determined depending upon the number of competing documents to-be-selected. A competing document set T with the class 1 is assumed to consist of N documents. The n-th document T_n of the set T is assumed to consist of K_T(n) sentences, and the k-th sentence vector is denoted by t_nk (FIG. 6C). The topic difference factor analysis step 24 (FIG. 5) calculates positive and negative topic difference factor vectors using the document set belonging to each class and its competing document set. A projection axis that is to be found as the positive topic difference factor vector is denoted by . Letting P_D and P_T denote the sums of squared projection values in the case where all the sentence vectors of the document sets D and T are projected onto the axis , respectively, the positive topic difference factor vector is obtained as the which maximizes a criterion function J()P_D()/P_T(). The which maximizes J() reflects features which can exist in the document set D, but which rarely exist in the document set T, for the reason that it ought to have a large value as the sum of the squared projection values of the sentence vectors of the document set D and a small value as the sum of the squared projection values of the sentence vectors of the document set T. In this case, P_D() and P_T() are respectively represented as: $\begin{array}{cc}{P}_D\left(\right)=\underset{m=1}{\overset{M}{}}\underset{k=1}{\overset{K_D\left(m\right)}{}}{\left(d_{\mathrm{mk}}^T\right)}^2={}^T{S}_D& \left(6\right)\\ S_D=\underset{m=1}{\overset{M}{}}\underset{k=1}{\overset{K_D\left(m\right)}{}}{d}_{\mathrm{mk}}^{}{d}_{\mathrm{mk}}^T& \left(7\right)\\ P_T\left(\right)=\underset{n=1}{\overset{N}{}}\underset{k=1}{\overset{K_T\left(n\right)}{}}{\left(t_{\mathrm{nk}}^T\right)}^2={}^T{S}_T& \left(8\right)\\ S_T=\underset{n=1}{\overset{N}{}}\underset{k=1}{\overset{K_T\left(n\right)}{}}{t}_{\mathrm{nk}}{t}_{\mathrm{nk}}^T& \left(9\right)\end{array}$

[0043] Therefore, the criterion function J() can be written as: $\begin{array}{cc}J\left(\right)=\frac{P_D\left(\right)}{P_T\left(\right)}=\frac{{}^T{S}_D}{{}^T{S}_T}& \left(10\right)\end{array}$

[0044] The that maximizes the criterion function J() given by Eq. (10) can be obtained by differentiating Eq. (10) by and then setting the result equal to zero. That is, it is given as the eigenvector of the following generalized eigenvalue problem:

[0045] A plurality of eigenvectors can be usually obtained from Eq. (11), and the first to L_G-th eigenvectors selected from among them become the positive topic difference factor vectors _i(i1, . . . L_G) at the step 15 in FIG. 2. If denotes the other projection axis which is to be found, and if J()P_T()/P_D() denotes a criterion function, then the axis which maximizes the criterion function J() represents features which should exist in the document set T, but which rarely exist in the document set D. In this case, the which maximizes the criterion function J() is given as the eigenvector of the following generalized eigenvalue problem, likewise to Eq. (11):

[0046] The first to L_P-th eigenvectors selected from among a plurality of eigenvectors obtained from Eq. (12) become the negative topic difference factor vectors _i (i1, . . . L_P) at the step 15 in FIG. 2. In the case of Eq. (11), the matrix S_T must be a regular matrix for the eigenvectors to be obtained. In actuality, however, the matrix S_T cannot be obtained as a regular matrix in a case where the number of sentences in the training document set is smaller than the number of terms, or where a certain number of term pairs always co-occur. In such a case, it is permitted to obtain the eigenvectors by regularizing the matrix S_T in accordance with the following equation:

[0047] where ² denotes a parameter, and I denotes the identity matrix. In case of employing Eq. (13), the criterion function J() corresponds to the following:

[0048] In the above embodiment, the lengths of the document and sentences are not taken into consideration. Accordingly, even when the similarity of the input document to each class has been obtained without regard to the length of the document, there arises the problem that the correction magnitude of the similarity enlarges more for a longer document, or that the correction magnitude of the similarity is more influenced by a longer sentence. At the step 15 in FIG. 2, therefore, Eq. (4) can be replaced by: $\begin{array}{cc}{\mathrm{sim}}_c\left(X,l\right)=\mathrm{sim}\left(X,l\right)+a\underset{i=1}{\overset{L_G}{}}\underset{k=1}{\overset{K}{}}{\left(x_k^T{}_i\right)}^2/K-b\underset{i=1}{\overset{L_P}{}}\underset{k=1}{\overset{K}{}}{\left(x_k^T{}_i\right)}^2/K& \left(15\right)\end{array}$

[0049] As stated before, K denotes the number of sentences in the input document X. Thus, the influence of the length of the document can be reduced. The same holds true of Eq. (5). Alternatively, letting N_k denote the number of terms occurring in the k-th sentence of the input document, Eq. (4) may well be replaced by: $\begin{array}{cc}{\mathrm{sim}}_c\left(X,l\right)=\mathrm{sim}\left(X,l\right)+a\underset{i=1}{\overset{L_G}{}}\underset{k=1}{\overset{K}{}}{\left(x_k^T{}_i\right)}^2/N_k-b\underset{i=1}{\overset{L_P}{}}\underset{k=1}{\overset{K}{}}{\left(x_k^T{}_i\right)}^2/N_k& \left(16\right)\end{array}$

[0050] Thus, the influence of discrepancy in the length of the sentence can be reduced. The same holds true of Eq. (5).

[0051] Further, the sentence vector x_k of the input document in FIG. 3B may well be normalized as follows, so as to apply the normalized vector to Eqs. (4), (5), (15) and (16):

[0052] The positive and negative topic difference factor vectors can be obtained by similarly normalizing the sentence vectors d_mk and t_nk in FIGS. 6B and 6C.

[0053] As described above, according to the present invention, the distinctive information of each class can be used for classification, and hence, the precision of the classification can be remarkably heightened. In an experiment employing Reuters-21578 (where the number of training documents was 7770, the number of categories was 87, and the number of test documents was 3019), the data of the prior-art kNN method in which the corrections of the present invention were not made demonstrated a precision of 85.93%, a recall of 81.57% and an F measure of 83.69%. In contrast, the precision, recall and F measure were respectively enhanced to 90.03%, 84.40% and 87.14% by making the corrections of similarities based on Eq. (16) in accordance with the present invention. $\begin{array}{cccc}& \mathrm{Precision}& \mathrm{Recall}& F\mathrm{measure}\\ \mathrm{Prior}\text{-}\mathrm{art}\mathrm{kNN}\mathrm{method}& 85.93\%& 81.57\%& 83.69\%\\ \begin{array}{c}\mathrm{Method}\mathrm{according}\mathrm{to}\\ \mathrm{the}\mathrm{present}\mathrm{invention}\end{array}& 90.03\%& 84.40\%& 87.14\%\end{array}$

[0054] The definitions of the precision, recall and F measure are as stated before, and one document can belong to a plurality of classes in Reuters-21578.

[0055] This concludes the description of the example embodiments. Although the present invention has been described with reference to a number of illustrative embodiments thereof, it should be understood that numerous other modifications and embodiments can be devised by those skilled than the art that will fall within the spirit and scope of the principles of this invention. More particularly, reasonable variations and modifications are possible in the component parts and/or arrangements of the subject combination arrangement within the scope of the foregoing disclosure, the drawings, and the appended claims without departing from the spirit of the invention. In addition to variations and modifications in the component parts and/or arrangements, alternative uses will also be apparent to those skilled in the art.