Class description generation for clustering and categorization

[0001] The following U.S. patent applications relating generally at least to aspects of multiple view clustering and categorization are commonly assigned with the present application, and are incorporated herein by reference.

[0002] Goutte et al., “Method for Multi-Class, Multi-Label Categorization using Probabilistic Hierarchical Modeling,” U.S. patent application Ser. No. 10/774,966 filed 9 Feb. 2004 and published as U.S. Publ. Appl. 2005/0187892 A1 on 25 Aug. 2005 is incorporated by reference herein in its entirety.

[0003] Gaussier et al., “Categorization including Dependencies between Different Category Systems”, U.S. patent application Ser. No. 11/170,033 filed 29 Jun. 2005 is incorporated by reference herein in its entirety.

[0004] The following U.S. patent applications relating generally at least to aspects of training and use of categorization and clustering systems are commonly assigned with the present application, and are incorporated herein by reference.

[0005] Goutte et al., “Incremental Training for Probabilistic Categorizer”, U.S. patent application Ser. No. 11/170,019 filed 29 Jun. 2005 is incorporated by reference herein in its entirety.

[0006] Goutte et al., “Method and Apparatus for Explaining Categorization Decisions”, U.S. patent application Ser. No. 11/013,365 filed 15 Dec. 2004 is incorporated by reference herein in its entirety.

[0007] The following relates to the information arts. It is described with example reference to providing keywords, semantic descriptions, or other characterization of classes of probabilistic clustering or categorization systems. However, the following is amenable to other like applications.

[0008] Automated clustering and categorization systems classify documents into classes on the basis of automated analysis of content. Categorization and clustering systems can employ soft or hard classification. In soft classification, a single document can be assigned to more than one class. In hard classification, also called partitioning, each document is assigned to a single class.

[0009] In probabilistic categorization systems, a set of training documents are typically annotated with class assignment information respective to a pre-defined structure of classes. Probabilistic model parameters are computed that profile the usage of selected vocabulary words or word combinations in training documents, in training documents assigned to classes, or in other content groupings of interest. An algorithm is developed for assigning unclassified documents to classes based on application of the probabilistic model parameters. Henceforth, the developed algorithm is used to assign received unclassified documents to suitable classes.

[0010] Probabilistic clustering systems operate similarly, except that in clustering the pre-defined class structure and training document class annotations are not provided. Rather, training documents are clustered during the training to define classes based, for example, on similar usage of vocabulary words or word combinations, or other common document characteristics.

[0011] As some illustrative examples, Naïve Bayes type probabilistic classifiers employ Bayesian conditional vocabulary word probabilities, assuming statistical independence of the conditional probabilities. Another type of probabilistic classifiers are the probabilistic latent classifiers (PLC), which are described for example in Goutte et al., U.S. Publ. Appl. 2005/0187892 A1, and in Gaussier et al., “A hierarchical model for clustering and categorising documents”, in “Advances in Information Retrieval Proceedings of the 24th BCS-IRSG European Colloquium on IR Research”, vol. 2291 of Lecture Notes in Computer Science, pages 229 47 (Springer, 2002), Fabio Crestani, Mark Girolami, and Cornelis Joost van Rijsbergen, editors.

[0012] A problem arises in characterizing the classes of an automated clustering or categorization system. The classes are advantageously characterized by keywords, semantically descriptive phrases, sentences, or paragraphs, or so forth. Such characterization enables user browsing of the classification system by semantic description, or enables automated searching of the classification system using keywords, or so forth. Such class characterization substantially improves the efficiency and usefulness of the categorization or clustering system. However, deriving class-distinguishing keywords or semantic class descriptions is challenging.

[0013] One existing approach for characterizing classes of probabilistic categorizing or clustering systems is to identify as keywords for a class those vocabulary words that are most commonly used in the class. That is, vocabulary words that are most frequently used, or have the highest probability, in a class are assigned as keywords for that class. In a variant approach, the keywords for a class are limited to vocabulary words that are common to all documents in the class. Here, the user is sure that every document in the class contains the keyword.

[0014] These approaches have the disadvantage that common generic vocabulary words that occur frequently in more than one related class may be assigned as keywords for those related classes. For example, common computer-related words such as “keyboard”, “screen”, or “file” may be assigned as keywords for several different computer-related classes. Although these common generic keywords may help the user locate the group of computer-related classes, they typically provide little or no guidance to the user in refining the class selection to those computer-related classes that are of special interest to the user. The variant approach in which a keyword must be so common that it appears in every document of the class can further bias the class characterization toward selecting common generic terms as keywords.

[0015] In accordance with certain aspects illustrated herein, a method is disclosed for characterizing a class of a probabilistic classifier or clustering system that includes probabilistic model parameters. For each of a plurality of candidate words or word combinations, divergence of the class from other classes is computed based on one or more probabilistic model parameters profiling the candidate word or word combination. One or more words or word combinations are selected for characterizing the class as those candidate words or word combinations for which the class has substantial computed divergence from the other classes.

[0016] In accordance with certain aspects illustrated herein, a method is disclosed for identifying documents or document portions characteristic of a class of a probabilistic classifier or clustering system that includes probabilistic model parameters. For each of a plurality of candidate documents or document portions, a divergence of the candidate document or document portion from the class is computed based on one or more probabilistic model parameters profiling the candidate document or document portion in the class. One or more of the candidate documents or document portions are selected for characterizing the class as those candidate documents or document portions having a smallest divergence from the class.

[0017] In accordance with certain aspects illustrated herein, a storage medium is disclosed which stores instructions which when executed by a digital processor perform a method for characterizing classes of a probabilistic classifier or clustering system including (i) computing divergences of classes from other classes respective to candidate word combinations based on probabilistic model parameters of the probabilistic classifier or clustering system, and (ii) characterizing classes using candidate word combinations that provide substantial computed divergence from other classes.

[0018] The FIGURE diagrammatically shows a clustering or categorization system including a divergences-based class characterization system.

[0019] With reference to the FIGURE, a probabilistic categorization or clustering system 10 has been trained using a set of training documents 12 to identify a set of vocabulary words 14 and probabilistic model parameters 16 suitable for henceforth assigning unclassified documents to one or more classes of a group of classes 20 based on a classifying algorithm 22 that classifies documents based on application of the probabilistic model parameters 16. For convenience of description, the notation W={w_i} is used herein, where w_i (or simply w) denotes a vocabulary word and W denotes the set of vocabulary words 14. Similarly, the notation D={d_m} is used herein, where d_m (or simply d) denotes a training document and D denotes the set of training documents 12 used to train the probabilistic categorization or clustering system 10. Similarly again, the notation C={c_k} is used herein, where c_k (or simply c) denotes a class and C denotes the set of classes 20 of the probabilistic categorization or clustering system 10.

[0020] In some embodiments, the probabilistic categorization or clustering system 10 is a probabilistic latent categorization or clustering (PLC) system such as is described, for example, in Goutte et al., U.S. Publ. Appl. 2005/0187892 A1, which is incorporated by reference herein in its entirety. However, the class characterization approaches disclosed herein are suitably applied to other types of probabilistic categorization or clustering systems that generate probabilistic model parameters 16.

[0021] If the system 10 is a probabilistic categorization system, then each training document d_m of the set of training documents 12 was provided pre-annotated by one or more classes (for example by a human annotator or by another computer system or other automated annotation system). Thus, the set of classes 20 was constructed prior to initiation of training. The training process extracted the set of vocabulary words 14 and the probabilistic model parameters 16 from the training documents 12, and determined a suitable algorithm for classifying documents into an exclusive class (for hard categorization or partitioning) or into one or more classes (for soft categorization) based on application of the probabilistic model parameters 16. On the other hand, if the system 10 is a probabilistic clustering system, then the set of training documents 12 were not provided with pre-annotation of classes, and the set of classes 20 was constructed as part of the clustering aspect of the training process. The class characterization approaches set forth herein are readily applied to either categorization or classification systems, and to either hard or soft classification.

[0022] An example suitable probabilistic model parameter is the probabilistic profile of a vocabulary word w in a training document d given by P(w|d)=N_w/N_d where N_w denotes the number of occurrences of the word w in document d and N_d denotes the number of words in the document d. Another example suitable probabilistic model parameter is the probabilistic profile of a vocabulary word w in a class c given by P(w|c)=N_w/N_c where N_w denotes the number of occurrences of the word w in training documents assigned to class c and N_w denotes the number of words in the training documents assigned to class c. Another example suitable probabilistic model parameter is the probabilistic profile of a document d assigned to a class c given by P(d|c)=N_d/N_c where N_d denotes the number of words in the document d and N_c denotes the number of words in training documents assigned to class c. In some embodiments, the inverse ratio P(c|d)=N_c/N_d may be used in place of or in addition to P(c|d). Another example suitable probabilistic model parameter is the probabilistic profile of a class c given by P(c)=N_c/N_total where N_c denotes the number of words in training documents assigned to class c and N_total denotes the number of words in all of the training documents 12. It is to be appreciated that the word counts, such as N_w, N_c, N_d, N_total, and so forth optionally omit counting words of under a certain number of letters, optionally omit or optionally include counting numeric values, optionally count only vocabulary words, or so forth. Moreover, the probabilistic categorization or clustering system 10 may employ some or all of these example probabilistic model parameters, may employ different probabilistic model parameters, or so forth.

[0023] The vocabulary 14 can be constructed from the set of training documents 12 in various ways. In some embodiments, the vocabulary 14 includes any word used in more than a threshold number of documents. In some embodiments, the vocabulary 14 includes any word used more than a threshold number of times in the set of training documents 12. In some embodiments, vocabulary words are selected based on their contributions to a divergence between classes. For example, Dobrokhotov et al., “Combining NLP and Probabilistic Categorization for Document and Term Selection for Swiss-Prot Medical Annotation”, Bioinformatics 19 Suppl. 1:i91-94 (2003)) sets forth a vocabulary words selection method based on word contributions to the Kullback-Leibler divergence between classes.

[0024] With continuing reference to the FIGURE, a class characterization system 30 determines keywords, semantic descriptions, or other characterization of the classes c_k of the set of classes 20. A candidate words or word combinations selector 32 selects a plurality of candidate words or word combinations which are candidates for use in characterizing classes. For example, the selector 32 may select the vocabulary words 14, or may access a natural language processing (NLP) parser 34 to extract noun phrases, verb phrases, sentences, or other typically descriptive natural language components from the training documents 12 for uses as the candidate word combinations. In some embodiments, the training documents 12 are themselves treated as candidate word combinations for characterizing the classes 20. In some embodiments, selected portions of the training documents 12, such as document abstracts if present, are treated as candidate word combinations for characterizing the classes 20.

[0025] A divergence contributions calculator 40 computes for each class a divergence of the class from other classes based on the probabilistic model parameters 16 profiling each candidate word or word combination. The computed divergences for each word or word combination are indicative of how well the candidate word or word combination distinguishes between classes. A characteristic words or word combinations selector 42 selects one or more words or word combinations for characterizing each class as those candidate words or word combinations for which the class has substantial computed divergence from the other classes. By selecting candidate words or word combinations for which the class has substantial computed divergence from the other classes, the selected words or word combinations effectively distinguish the class from other classes of the set of classes 20. In some embodiments, the divergence contributions calculator 40 computes divergence contributions of candidate documents or other candidate word combinations from a class. In these embodiments, the characteristic words or word combinations selector 42 selects those candidate documents or word combinations that have the smallest divergence from the class, and which therefore are exemplary of the class.

[0026] A class characterization processor 44 constructs suitable class characterizations based on the selected words or word combinations. For example, if the candidate words or word combinations include vocabulary words, then the class characterization processor 44 may assign the vocabulary words selected for each class as keywords for that class. On the other hand, if the candidate words or word combinations include training documents, then the class characterization processor 44 may assign the training documents selected for each class as characteristic example documents of that class. If the candidate words or word combinations include natural language phrases or sentences, then the class characterization processor 44 may construct a semantic description of the class based on the natural language phrases or sentences.

[0027] With continuing reference to the FIGURE, some examples of suitable implementations of the divergence contributions calculator 40 and selector 42 are set forth. In one approach, the candidate words or word combinations include the set of vocabulary words W 14, and the divergence contributions calculator 40 calculates contributions ε_w(c_A, c_B) to the Kulback-Leibler divergence between a class c_A and another class c_B respective to the vocabulary W 14: $\begin{array}{cc}{\varepsilon }_w\left(c_A,c_B\right)=\left(P\left(w|c_A\right)-P\left(w|c_B\right)\right)·\ln \left(\frac{P\left(w|c_A\right)}{P\left(w|c_B\right)}\right),& \left(1\right)\end{array}$

[0028] where P(w|c_A) and P(w|c_B) denote the probabilistic model parameters profiling the usage of the candidate word w in classes c_A and c_B, respectively, and where ε_w(c_A, c_B) denotes the contribution to the Kulback-Leibler divergence of class c_A from class c_B for the vocabulary word w. Equation (1) applied across the set of classes C={c_k} produces a matrix of divergence elements for each candidate word w with one divergence matrix element ε_w(c_A, c_B) for each ordered pair of classes (c_A, c_B). For example, if the set of classes C={c_k} 20 includes five classes (index k ranging from 1 to 5) then the output of Equation (1) is suitably:

TABLE 1
εw(CA, CB) Divergence Matrix
	CA
CB	C1	C2	C3	C4	C5
C1	0	εw(C1, C2)	εw(C1, C3)	εw(C1, C4)	εw(C1, C5)
C2	εw(C2, C1)	0	εw(C2, C3)	εw(C2, C4)	εw(C2, C5)
C3	εw(C3, C1)	εw(C3, C2)	0	εw(C3, C4)	εw(C3, C5)
C4	εw(C4, C1)	εw(C4, C2)	εw(C4, C3)	0	εw(C4, C5)
C5	εw(C5, C1)	εw(C5, C2)	εw(C5, C3)	εw(C5, C4)	0

Each divergence matrix element ε_w(c_A, c_B) measures how discriminative word w is for differentiating classes c_A and c_B. Those vocabulary words with the largest contributions ε_w are suitably selected as keywords. Optionally, contributions are normalized in order to be comparable from class to class. This has no influence on the ranking of the keywords, but may have an influence on the choice of the number of keywords. In one example embodiment of the selector 42, enough keywords are chosen to cover a selected percentage of the total Kulback-Leibler divergence contribution. Optionally, this technique may also be used to find “anti-keywords”, that is, keywords that are more atypical for a class than they are for others. Those are negative divergence matrix elements ε_w(c_A, c_B).

[0029] An example variation of Equation (1) is: $\begin{array}{cc}{\varepsilon }_w\left(c_A\right)=\left(P\left(w|c_A\right)-P\left(w|C\right)\right)·\ln \left(\frac{P\left(w|c_A\right)}{P\left(w|C\right)}\right),& \left(2\right)\end{array}$

[0030] where P(w|C) denotes the probabilistic model parameter profiling the usage of the vocabulary word w in all of the classes C={c_k} 20, and ε_w(c_A) denotes the divergence vector element indicating the contribution of word w to the divergence of the class c_A from the average of the other classes. Equation (2) applied across the set of classes C={c_k} produces a vector of divergence elements for each candidate word w with one divergence vector element ε_w(c_A) for each class c_A. For example, if the set of classes C={c_k} 20 includes five classes (index k ranging from 1 to 5) then the output of Equation (2) is suitably:

TABLE 2
εw(CA) Divergence Vector
	CA	εw(CA)
	C1	εw(C1)
	C2	εw(C2)
	C3	εw(C3)
	C4	εw(C4)
	C5	εw(C5)

The approach of Equation (2) is less computationally intensive than the approach of Equation (1), but provides less selection information for choosing keywords since a keyword that is highly distinguishing between classes c₁ and c₃, for example, may nonetheless have a small contribution to the overall divergence vector ε_w(c₁) if that keyword is not very useful for distinguishing class c₁ from the other classes c₂, c₄, and c₅.

[0031] The example word-based implementations of the divergence contributions calculator 40 set forth with reference to Equations (1) and (2) provide discriminating words from which the class characterization processor 44 can readily construct a set of keywords for labeling classes. Such keywords provide a convenient basis for automated keyword-based searching of the probabilistic categorization or clustering system 10. However, the user may find that keywords provide less than fully satisfactory class descriptions for manually browsing the classes. The user presented with a list of keywords labeling a class must make his or her own inferences as to how those keywords collectively set forth the content of the class. For the purpose of browsing classes, the user may be better served by being provided with characteristic phrases, characteristic example documents, or other characteristic semantic word combination class labels that more fully describe or exemplify content of the class.

[0032] In some embodiments, the divergence contributions calculator 40 and selector 42 are configured to identify characteristic documents from the set of training documents 12 that exemplify each class. The Kulback-Leibler divergence between a class c_A and another class c_B respective to the set of training documents D 12 is suitably denoted KL_D(c_A, c_B) herein, and is suitably given by: $\begin{array}{cc}{\mathrm{KL}}_D\left(c_A,c_B\right)=\sum _{d\in D}{\eta }_d\left(c_A,c_B\right),& \left(3\right)\end{array}$
where in Equation (3) the term η_d(c_A, c_B) denotes the contribution of training document d to the Kulback-Leibler divergence between classes c_A and c_B, given by: $\begin{array}{cc}{\eta }_d\left(c_A,c_B\right)=\left(P\left(d|c_A\right)-P\left(d|c_B\right)\right)·\ln \left(\frac{P\left(d|c_A\right)}{P\left(d|c_B\right)}\right),& \left(4\right)\end{array}$
where P(d|c_A) and P(d|c_B) denote the probabilistic model parameters profiling the candidate document d in classes c_A and c_B, respectively. Equation (4) applied across the set of classes C={c_k} produces a matrix of divergence elements for each candidate document d with one divergence matrix element η_d(c_A, c_B) for each ordered pair of classes (c_A, c_B) indicating how effective the candidate document d is at discriminating between the classes c_A and c_B.

[0033] In some embodiments, Equation (4) is modified as: $\begin{array}{cc}{\eta }_d\left(c_A\right)=\left(P\left(d|c_A\right)-P\left(d|C\right)\right)·\ln \left(\frac{P\left(d|c_A\right)}{P\left(d|C\right)}\right).& \left(5\right)\end{array}$
where P(d|C) denotes the probabilistic model parameter profiling the document d respective to all the classes C={c_k} 20, and η_d(c_A) denotes the divergence vector element indicating contribution of document d to the divergence of the class c_A from the average of the other classes. Equation (5) applied across the set of classes C={c_k} produces a vector of divergence elements for each candidate document d with one divergence vector element η_d(c_A) for each class c_A.

[0034] Using the document probabilities P(d|c) can be problematic if documents vary substantially in length, since longer documents will typically have a higher probability P(d|c). This is suitably compensated by employing the posterior class probability P(c|d), so documents with the largest posterior for a class may be good candidates for class labeling.

[0035] In cases in which the training documents are relatively large, a document may be a disadvantageously large unit for providing a useful idea of the content of a class. In typical probabilistic categorization or clustering systems, however, the notion of document is arbitrary, so smaller portions of documents, such as paragraphs, or even sentences, may be considered as “documents”. For each document portion, a class-conditional probability P(d/c) (or posterior P(c/d)) is computed, and exemplary document parts are selected for each class using selected criteria of Equations (3)-(5). This approach is simple to implement because it is based on probabilistic model parameters available as a result of the clustering or training phase.

[0036] In the case of hard assignment, a problem arises in applying Equations (3)-(5). In hard assignment or partitioning, each document is assigned to a single class. Accordingly, P(d|c)=0 except for the assigned class. In some embodiments, this difficulty is addressed by keeping the class profiles P(w|c) fixed and re-estimating P(d|c) by maximizing the likelihood, for example using entropy maximization or another suitable technique. This is typically fast and does not call for expensive optimization techniques such as deterministic annealing. The re-estimated values of P(d|c) may be used in Equations (3)-(5) to look for typical documents or document portions, while retaining the hard assignment or partitioning for interactions with the user.

[0037] Another approach to applying Equations (3)-(5) in the case of hard assignment is to use the empirical profile of a document, denoted herein as {circumflex over (P)}(w|d)∝N_wd, that is, the empirical frequency of words in documents. The closer these empirical frequencies are to the conditional probabilities in the class profile, the more typical this document is of the corresponding class. This closeness may be expressed by a divergence, such as a Kulback-Leibler divergence: $\begin{array}{cc}\mathrm{KL}\left(d,c\right)=\sum _{w\in W}\left(\hat{P}\left(w|d\right)-P\left(w|c\right)\right)·\ln \left(\frac{\hat{P}\left(w|d\right)}{P\left(w|c\right)}\right),& \left(6\right)\end{array}$
or as a Euclidean distance, or so forth. The document or documents with the lowest divergence from the class c as computed by the divergence contributions calculator 40, that is, the document or documents having the lowest divergence value KL(d, c), are the most typical of class c and hence are suitably selected as prototypical examples of class c by the characteristic words or word combinations selector 42. Note that in this approach a document length bias exists. The class profiles are relatively more influenced by longer documents, which introduces a systematic decrease of the divergence. Also, the empirical profile is estimated with a quantization level that depends of the document length (for example, in the case of a very short document consisting of two words, the empirical frequencies are limited to 0, 0.5 or 1). Longer documents may therefore be able to yield finer empirical frequencies and therefore a smaller divergence. The document length bias (other than quantization) is suitably compensated by normalizing by the document length or other similar techniques.

[0038] The example document-based implementations of the divergence contributions calculator 40 set forth with reference to Equations (3)-(6) provide example documents that are class-discriminating or class-representative. These documents can help the user to browse the classes probabilistic categorization or clustering system 10.

[0039] In other example embodiments, the candidate word combinations selected by the candidate words or word combinations selector 32 are phrases, such as noun phrases identified by the natural language parser 34. Noun phrases can be identified using dedicated linguistic tools, for example based on syntactic patterns, or by a statistical approach based, for example, on collocations. The divergence contributions calculator 40 in these embodiments calculates a divergence contribution for each noun phrase which combines the estimated divergence contributions ε_w of the words in the noun phrase and the document divergence contributions η_d of the documents in which the noun phrase is found. One suitable phrasal divergence contribution expression, denoted DIV(NP), that is characteristic of a noun phrase (NP) is given by: $\begin{array}{cc}\mathrm{DIV}\left(\mathrm{NP}\right)=\frac{1}{2}·\left(\frac{1}{\mathrm{NP}}·\sum _{w\in \mathrm{NP}}{\varepsilon }_w+\frac{1}{d|\mathrm{NP}\in d}·\sum _{d,\mathrm{NP}\in d}{\eta }_d\right),& \left(7\right)\end{array}$
which employs an arithmetic combination of (i) a word-based divergence based on one or more probabilistic model parameters of the probabilistic classifier or clustering system 10 that profile one or more words of the candidate noun phrase and (ii) a phrase-based divergence based on one or more probabilistic model parameters of the probabilistic classifier or clustering system 10 that profile those training documents containing the candidate phrase. Other suitable phrasal divergence contributions may include geometric averaging for the combination, for example using the maximum contribution rather than average, or so forth. The characteristic words or word combinations selector 42 labels the class by picking the noun phrases having the top phrasal divergence contributions.

[0040] Because probabilistic classifier or clustering systems typically are able to identify typical parts inside a document, application of the divergence contribution formula of Equation (7) optionally focuses on those typical parts, rather than searching for relevant phrases throughout the entire documents. This is especially useful with soft assignment, where all documents may be assigned to all classes with a non-zero probability. Moreover, the example phrasal divergence contribution formula of Equation (7) is optionally applied to phrases other than noun phrases (such as verb phrases or so forth), optionally including phrasal text units larger than noun phrases, such as clauses, sentences, paragraphs, or so forth.

[0041] In some embodiments in which the candidate words or word combinations are phrases or sentences (where phrases broadly includes portions of documents such as abstracts or so forth), the characteristic words or word combinations selector 42 employs a multi-document summarization approach. Such approaches are known for summarizing, for example, clusters of documents returned by search engines. Typical multi-document summarization employing sentence units involves: (i) calculating the weight of words in the collection using some heuristics; (ii) combining word weights in each sentence to calculate a sentence score; and (iii) ranking the sentences according to score and outputting a certain number of top-ranked sentences to achieve a desired summary text length. In applying multi-document summarization techniques, the characteristic words or word combinations selector 42 imposes constraints, such that the multi-document summarization is adapted to provide a useful description. In particular, the classification model provides information on which words are more or less important in the context of each class, and which documents are more or less prototypical of the class. In order to obtain an informative summary describing the content of a class, the multi-document summarization is constrained to favor sentences or other word combinations from prototypical documents as well as to favor sentences or other word combinations that contain important words (for example in the sense of the Kulback-Leibler divergence contributions set forth herein). The importance (or weight) of a word for a class is known from the model using, for example, the word-based divergence contributions ε_w. The sentence or other word combination score further takes into account the importance of the document (or document part) the sentence was extracted from, for example using the document-based divergence contributions η_d. (For hard partitioning, sentences are extracted only from documents belonging to the owning class). One suitable embodiment of the characteristic words or word combinations selector 42 which uses a multi-document summarization technique is as follows: (i) combine word contributions in each sentence to calculate a sentence score; (ii) calculate the weighted sentence score by multiplying it by the contribution of the document (or document portion) the sentence was extracted from (for identical sentences, accumulate or filter the weighted scores); and (iii) rank the sentences according to the weighted score and output the top-ranked sentences (depending on the desired summary length). The thusly extracted sentences meet two goals: they contain words that are important to the class, and they come from documents that are typical of the class.

[0042] In general, the class labeling approaches disclosed herein are applicable to a continuum of word combinations ranging from individual words to whole documents.

[0043] In some probabilistic categorization or clustering systems the probabilistic model parameters may be updated after the initial training as unclassified documents are assigned to classes. Some suitable approaches for updating the probabilistic model parameters as later unclassified documents are processed by the system are disclosed in Goutte et al., “Incremental Training for Probabilistic Categorizer”, U.S. patent application Ser. No. 11/170,019 filed 29 Jun. 2005, which is incorporated by reference herein in its entirety. The class characterization approaches disclosed herein are readily applied to such systems that update the probabilistic model parameters. Optionally, the class characterization system 30 is re-applied occasionally to ensure that the class labels remain up-to-date as the probabilistic model parameters are updated by processing of new unclassified documents.

[0044] The illustrated probabilistic categorization or clustering system 10 is a one-dimensional system having a single class dimension including the set of classes C={c_k}. Some systems may employ a multi-dimensional classification system, such as a two-dimensional classification system having a first class dimension C⁽¹⁾={c_k⁽¹⁾} and a second class dimension C⁽²⁾={c_k⁽²⁾}. Multi-dimensional classification systems are disclosed for example in U.S. Publ. Appl. 2005/0187892 A1, which is incorporated by reference herein in its entirety, and in Gaussier et al., “Categorization including Dependencies between Different Category Systems”, U.S. patent application Ser. No. 11/170,033 filed 29 Jun. 2005 which is incorporated by reference herein in its entirety. The class characterization approaches disclosed herein are readily applied to class labeling in probabilistic categorization or clustering systems employing either the illustrated single-dimensional, or multi-dimensional, classification systems.

[0045] The class characterization system 30 can be physically embodied in various ways. Typically, a storage medium stores instructions which when executed by a digital processor perform class characterization in accordance with the described operation of the class characterization system 30. The storage medium can be, for example, a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (RAM), a storage medium of a network server, or so forth. The digital processor can be, for example, a digital computer, a digital calculator, a microprocessor, a microcontroller, network server, or so forth. In some embodiments, the storage medium and the digital processor may be integrated as a unit. For example, a desktop, laptop, or other computer may include a hard drive serving as the storage medium and a microprocessor and/or other digital circuitry serving as the digital processor. In other embodiments, the storage medium and digital processor may be distinctly separate. For example, the instructions may be stored on a remote storage medium connected with a network, and executed by a processor of a computer connected with the network. The instructions stored on the storage medium may access library functions or other instructions stored elsewhere to perform certain operations of the class characterization system 30. For example, the instructions stored on the storage medium which when executed by a digital processor perform the class characterization may include instructions accessing remotely stored mathematical library functions that perform certain mathematical operations of the class characterization system 30.

[0046] It will be appreciated that various of the above-disclosed and other features and functions, or alternatives thereof, may be desirably combined into many other different systems or applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.