Method and apparatus for multi-class, multi-label information categorization

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 60/076,809 entitled “BoosTexter: A System for Multiclass Multi-Label Text Categorization” and filed Mar. 3, 1998.

[0003] The present invention relates to information categorization. More particularly, the present invention relates to multi-class, multi-label information categorization.

[0004] Information categorization is the process of classifying information samples into categories or classes. By way of example, text categorization is the process of classifying a text document, such as into a “politics,” a “business” or a “sports” category, based on the document's content. When used in connection with a speech recognition device, information categorization can be used, for example, by a telephone network provider to automatically determine the purpose of a telephone call received from a customer. If the customer says, “I would like to charge this call to my credit card,” the system could automatically recognize that this is a calling-card request and process the call accordingly. Note that the information is categorized “automatically” in that human input is not required to make the decision. Although this example involves a speech-categorization problem, a text-based system can be used if the customer's spoken message is passed through a speech recognizer.

[0005] It is known that an information categorization algorithm can “learn,” using information samples, to perform text-categorization tasks, such as the ones described above. For example, a document might be classified as either “relevant” or “not relevant” with respect to a pre-determined topic. Many sources of textual data, such as Internet news feed, electronic mail and digital libraries, include different topics, or classes, and therefore pose a “multi-class” categorization problem.

[0006] Moreover, in multi-class problems, a document may be relevant to several different classes. For example, a news article may be relevant to “politics” and “business.” Telephone call-types are also not mutually exclusive (i.e., a call can be both “collect” and “person-to-person”).

[0007] One approach to multi-class, multi-label information categorization is to break the task into disjoint binary categorization problems, one for each class. To classify a new information sample, such as a document, all the binary classifiers are applied and the predications are combined into a single decision. The end result can be, for example, a list of which classes the document probably belongs to, or a ranking of possible classes. Such an approach, however, can ignore any correlation that might exist between different classes. As a result, the information categorization is less effective and/or efficient than may be desired.

[0008] In view of the foregoing, it can be appreciated that a substantial need exists for an information categorization method and apparatus that is directed to the multi-class, multi-label problem and addresses the problems discussed above.

[0009] The disadvantages of the art are alleviated to a great extent by a method and apparatus for multi-class, multi-label information categorization. A weight is assigned to each information sample in a training set, the training set containing a plurality of information samples, such as text documents, and associated labels. A base hypothesis is determined to predict which labels are associated with a given information sample. The base hypothesis may predict whether or not each label is associated with the information sample, or may predict the likelihood that each label is associated with the information sample. In the case of a document, the base hypothesis may evaluate words in each document to determine one or more words that predict the associated labels.

[0010] When a base hypothesis is determined, the weight assigned to each information sample in the training set is modified based on the base hypothesis predictions. For example, the relative weight assigned to an information sample may be decreased if the labels associated with that information sample are correctly predicted by the base hypothesis. These actions are repeated to generate a number of base hypotheses which are combined to create a combined hypothesis. An un-categorized information sample can then be categorized with one or more labels in accordance with the combined hypothesis. Such categorization may include predicting which labels are associated with each information sample or ranking possible labels associated with each information sample.

[0011] With these and other advantages and features of the invention that will become hereinafter apparent, the nature of the invention may be more clearly understood by reference to the following detailed description of the invention, the appended claims and to the several drawings attached herein.

[0012] FIG. 1 is a flow diagram of a method for information categorization according to an embodiment of the present invention.

[0013] FIG. 2 is a block diagram of an apparatus that categorizes information according to an embodiment of the present invention.

[0014] The present invention is directed to multi-class, multi-label information categorization. According to an embodiment of the present invention, an algorithm learns from examples to perform multi-class text and speech categorization tasks. The standard notion of classification is extended by allowing each instance to be associated with multiple labels.

[0015] Rather than breaking a multi-class problem into separate binary problems, an embodiment of the present invention uses the “AdaBoost” algorithm disclosed in Yoav Freund and Robert E. Schapire, “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting,” Journal of Computer and System Sciences, 55(1):119-139 (August 1997) and U.S. Pat. No. 5,819,247, the entire contents of which are hereby incorporated by reference, such that information categorization algorithms can represent and handle sets of labels. The AdaBoost algorithm has been studied and shown to perform standard machine-learning tasks.

[0016] Several embodiments of the present invention use extensions of AdaBoost that handle multi-label problems. According to one embodiment, the goal of the learning algorithm is to predict all—and only all—of the correct labels. Thus, the classifier is evaluated in terms of its ability to predict a good approximation of the set of labels associated with a given document. According to another embodiment, the goal is to design a classifier that “ranks” the labels such that the correct labels will receive the highest ranks.

[0017] By way of example only, consider the problem of categorizing a text document representing a news story into one or more of “politics,” “business” and “sports.” A sample of several thousand news stories that have been manually classified can be used as the training set. Each sample in the training set can be assigned a set of three weights, one for each possible class. Initially, the weight for each class, and the weights for all training set documents, may be set to equal values. For example, every training set document may be assigned a set of weights as follows: “politics—0.33;” “business—0.33;” and “sports—0.33.”

[0018] Using these sets of weights, a base hypothesis is determined that predicts which labels are associated with a given training set document. For example, every word that appears in any training set document can be evaluated to see how well that word predicts the labels associated with the training set documents. The word “baseball” might predict that a document (i) is not associated with “politics” or “business” and (ii) is associated with “sports.” The word “Washington,” on the other hand, may be less useful when making such a prediction. In this case, the base hypothesis may be “if the word ‘baseball’ appears in the document, the document has the label ‘sports’ and does not have the labels ‘politics’ and business.’”

[0019] The set of weights assigned to each document in the training set is then modified. The set of weights for a document that was correctly predicted by the base hypothesis may be reduced, while the set of weights for a document that was not correctly predicted by the base hypothesis may be increased. For example, a document that had the word “baseball” and was correctly predicted to be in “sports,” and not in “politics” or “business,” may now be assigned a set of weights as follows: “politics—0.20;” “business—0.20;” and “sports—0.20.” Another document that had the word “baseball” but should have been in both “politics” and “business,” and not in “sports,” may now be assigned a set of weights as follows: “politics—0.40;” “business—0.40;” and “sports—0.40.” Of course, a document for which the base hypothesis correctly predicted some labels and incorrectly predicted other labels can have some weights increased and other weights decreased. According to another embodiment of the present invention, a training set document may be assigned only a single weight as opposed to a set of weights.

[0020] The modified sets of weights are used to determine a new base hypothesis to predict which labels are associated with a given training set document. Because different weights are used, the new base hypothesis can be different from the initial hypothesis. Using the example above, the weights assigned to documents that were correctly predicted with the word “baseball” have been reduced, and these documents are therefore given less importance when evaluating the accuracy of a hypothesis. In this way, the system concentrates on those documents that have been less successfully categorized with the previous hypotheses. These steps are repeated a number of times to generate a number of “base” hypotheses that are combined into a single, more accurate, “combined” hypothesis. The combined hypothesis is then used to determine labels for unclassified new stories.

[0021] This example illustrates only one embodiment of the present invention, and many different variations are possible. For example, boosting algorithms related to four embodiments of the present invention will be described in detail to illustrate some of the implementation issues that arise in multi-label text categorization. It will be clear to those skilled in the art, however, that other boosting algorithms may be used instead in accordance with these and other embodiments of the present invention.

[0022] Some embodiments of the present invention may use θ(mk) space and time per boosting iteration, where m is the number of training documents and k is the number of different classes. Although a number of evaluation measures have shown that information categorization according to embodiments of the present invention is generally better than known algorithms, sometimes by a wide margin, there may be a price for this performance. Namely, a boosting algorithm according to some embodiments of the present invention may be an order of magnitude slower to train as compared with other information categorization algorithms.

[0023] Multi-Label Text Categorization—General Notations

[0024] Let X denote the domain of possible text documents and let Υ be a finite set of labels or classes. The size of Υ is denoted by k=|Υ|. In the traditional text categorization setting, each document x∈X is assigned a single class y∈Υ. The typical goal is to find a classifier H:X→Υ which minimizes the probability that y≠H(x) on a newly observed example (x, y). Note that the single-label case is therefore a special case in which |Y|=1 for all observations.

[0025] In the multi-label case, each document x∈X may be assigned multiple labels in Υ. For example, in a multi-class news filtering problems in which the possible classes are “politics,” “business,” and “sports,” a document may belong to both “politics” and “business.” Thus, a labeled example is a pair (x, Y) where YΥ is the set of labels assigned to x.

[0026] For YΥ, let us define Y{l} for l∈Υ to be: $Y\left\{l\right\}=\{\begin{array}{cc}+1& \text{if}l\in Y\\ -1& \text{if}l\in /Y.\end{array}$

[0027] Some classifiers produce a “ranking” of the possible labels for a given document, with the hope that the appropriate labels will appear at the top of the ranking. To be more formal, the goal of such learning is to produce a function of the form 𝑓: X×Υ→ with the interpretation that, for a given instance x, the labels in Υ should be ordered according to 𝑓(x, •). That is, a label l₁is considered to be ranked higher than l₂if 𝑓(x, l₁)>𝑓(x, l₂). If Y is the associated label set for x, then a successful learning algorithm will tend to rank labels in Y higher than those not in Y.

[0028] To simplify the notation, for any predicate π, let [[π]] be 1 if π holds, and 0 otherwise.

[0029] Boosting Algorithms for Multi-Label Multi-Class Problems

[0030] Two new boosting algorithms for multi-class, multi-label information classification will now be described, and four versions of these algorithms will be discussed in connection with the problem of text categorization. Note that similar algorithms may also be used in connection with other multi-class, multi-label information classification problems.

[0031] The purpose of “boosting” is to find a highly accurate classification rule by combining many weak or “base hypotheses,” each of which may be only moderately accurate. Access to a separate procedure called the “weak learner,” or “weak learning algorithm,” for computing the weak hypotheses is assumed. A weak learner may simply, for example, examine every word in all text samples to determine which word is the most accurate label predictor.

[0032] Referring now in detail to the drawings wherein like parts are designated by like reference numerals throughout, there is illustrated in FIG. 1 a flow diagram of a method for information categorization according to an embodiment of the present invention. As shown in FIG. 1, an initial weight, or set of weights, is assigned to each information sample in a training set (step 102).

[0033] The boosting algorithm finds a set of base or weak hypotheses by calling the weak learner repeatedly in a series of rounds. In particular, the weak learner determines a base hypotheses, using the weight or weights assigned to each information sample (step 104). The weight or weights assigned to each information samples is then modified based on the base hypothesis prediction for that information sample (step 106). This is repeated a number of times (steps 108) to create a plurality of base hypotheses. After the base hypotheses are complete, the base hypotheses are combined into a single rule called the “final” or “combined” hypothesis (step 110).

[0034] FIG. 2 is a block diagram of an apparatus 200 that categorizes information according to an embodiment of the present invention. That apparatus 200 includes a processing system 250 that uses information stored in a training set database 260 to generate a combined hypothesis as described with respect to FIG. 1. A speech input may be converted to text, such as by a speech recognizer 300, and input to the processing system 250 through a communication port 210. The processing system can then use the combined hypothesis to output the categorized information, such as the labels associated with a text document. Such information may be used, for example, to automatically process a telephone call.

[0035] Note the apparatus 200 is shown in FIG. 2 performs both the learning and information categorization functions. Such an arrangement is provided merely as an illustration of an embodiment of the present invention. It will be clear to those skilled in the art, however, that other systems may be used instead. For example, one device may perform the learning function and generate a combined hypothesis, while one or more other devices perform the information categorization function, using that combined hypothesis.

[0036] According to one embodiment of the present invention, which may be used, for example, with a simple version of AdaBoost for single-label classification, the boosting algorithm maintains a set of importance weights over training examples. These weights are used by the weak learning algorithm, whose goal is to find a weak hypothesis with moderately low error with respect to these weights. Thus, the boosting algorithm can use these weights to force the weak learner to concentrate on the examples which are hardest to classify.

[0037] For multi-class, multi-label problems, a set of weights may be maintained over training examples and labels according to another embodiment of the present invention. As boosting progresses, training examples, and corresponding labels, that are hard to predict correctly get incrementally higher weights while examples and labels that are easy to classify get lower weights. For instance, with respect to a text document classification problem, it might be easy to classify a document as a “politics” item but hard to determine whether or not it belongs to the “business” section. In this case, as boosting progresses the weight of the “politics” label may decrease while the weight of the “business” label increases. The intended effect is to force the weak learning algorithm to concentrate on examples and labels that will be most beneficial to the overall goal of finding a highly accurate classification rule.

[0038] AdaBoost.MH

[0039] Let S be a sequence of training examples <(x₁, Y₁), . . . , (x_m, Y_m)> where each instance x_i∈X and each Y_iΥ. The first boosting algorithm for multi-class multi-label classification problems according to an embodiment of the present invention called “AdaBoost.MH,” is as follows:

[0040] Given: (x₁, Y₁), . . . , (x_m, Y_m) where x_iΥX and Y_iΥ

[0041] Initialize D₁(i, l)=1/(mk)

[0042] For t=1, . . . , T:

[0043] Pass distribution D_tto weak learner

[0044] Get weak hypothesis h_t: X×Υ→

[0045] Choose a_t∈

[0046] Update: $D_{t+1}\left(i,l\right)=\frac{D_t\left(i,l\right)\exp \left(-{\alpha }_t{Y}_i\left\{l\right\}{h}_t\left(x_i,l\right)\right)}{Z_t}$

[0047] where Z_tis a normalization factor, chosen so that D_t+1will be a distribution. The final hypothesis is output as follows: $f\left(x,l\right)=\sum _{t=1}^T{\alpha }_t{h}_t\left(x,l\right)$

[0048] This AdaBoost.MH algorithm maintains a set of weights as a distribution D_tover examples and labels. Initially, this distribution is uniform. On each round t, the distribution D_t, together with the training sequence S, is passed to the weak learner which computes a weak hypothesis h_t. The output of the weak learner is a hypothesis h: X×Υ→. The sign of h(x, l) represents a prediction as to whether the label l is, or is not, assigned to x (i.e., a prediction of the value of Y{l}). The magnitude of the prediction |h(x, l)| is interpreted as a measure of “confidence” in the prediction. The precise goal of the weak learner is described below, as are examples of weak learners.

[0049] A parameter α_tis then chosen and the distribution D_tis updated. In the typical case that α_tis positive, the distribution D_tis updated in a manner that increases the weight of example-label pairs which are misclassified by h_t(i.e., for which Y_i{l} and h_t(x_i, l) differ in sign). The final hypothesis ranks documents using a weighted vote of the weak hypotheses.

[0050] This algorithm is derived using a natural decomposition of the multi-class, multi-label problem into k orthogonal binary classification problems. That is, each observed label set Y may be thought of as specifying k binary labels (depending on whether a label l is or is not included in Y), and binary-prediction boosting algorithms can then be applied.

[0051] This view of the algorithm leads to a simple analysis. Specifically, there is a bound on the empirical “Hamming loss” of this algorithm, i.e., the fraction of examples i and labels l for which the sign of 𝑓(x_i, l) differs from Y_i{l}. The Hamming loss of this algorithm is at most: $\prod _{t=1}^T{Z}_t$

[0052] where Z_tis the normalization factor computed on round t. This upper bound can be used in guiding both the choice of α_tand the design of the weak learning algorithm. Together, these choices should be geared on each round t toward the minimization of: $Z_t=\sum _{i=1}^m\sum _{l\in {\rm Y}}{D}_t\left(i,l\right)\exp \left(-{\alpha }_t{Y}_i\left\{l\right\}{h}_t\left(x_i,l\right)\right)$

[0053] Methods used for choosing α_tand the implementation of the weak learning algorithm for text categorization are described after a description of the “AdaBoost.MR” algorithm.

[0054] AdaBoost.MR

[0055] Whereas AdaBoost.MH is designed to minimize Hamming loss, according to another embodiment of the present invention “AdaBoost.MR” is designed specifically to find a hypothesis that ranks labels in a manner with the correct labels placed at the top of the ranking.

[0056] With respect to a labeled observation (x, Y), focus now on the relative ordering of a “crucial pair” l₀, l₁for which l₀∉Y and l₁∈Y. A classification rule 𝑓“misorders” the crucial pair l₀, l₁if 𝑓(x, l₀)≦(x, l₀) so that 𝑓 fails to rank l₁above _l0. The goal here is to find a function 𝑓 with a small number of misorderings so that the labels in Y are ranked above the labels not in Y. Put another way, the goal is to minimize the average fraction of crucial pairs which are misordered, referred to herein as the empirical “ranking loss:” $\frac{1}{m}\sum _{i=1}^m\frac{1}{\left|Y_i\right|\left|{\rm Y}-Y_i\right|}|\left\{\left(l_0,l_1\right)\in \left({\rm Y}-Y_i\right)\times Y_i:f\left(x,l_1\right)\le f\left(x,l_0\right)\right\}|$

[0057] It can be assumed that Y_iis never empty nor equal to all of Υ for any instance. If there are such instances in the training set, they can be discarded since there is no ranking problem to be solved and they do not carry any information.

[0058] AdaBoost.MR is as follows:

[0059] Given: (x₁, Y₁), . . . , (x_m, Y_m) where x_i∈X and Y_iΥ $D_1\left(i,l_0,l_1\right)=\{\begin{array}{cc}1/(m·\left|Y_i\right|·{\rm Y}-Y_i|)& \text{if}{l}_0\in Y_i\text{and}{l}_1\in Y_i\\ 0& \text{else}\end{array}$

[0060] For t=1, . . . , T:

[0061] Train weak learner using distribution D_t

[0062] Get weak hypothesis h_t: X×Υ→

[0063] Choose α_t∈

[0064] Update: $D_{t+1}\left(i,l_0,l_1\right)=\frac{D_t\left(i,l_0,l_1\right)\exp (\frac{1}{2}{\alpha }_t\left(h_t\left(x_i,l_0\right)-h_t\left(x_i,l_1\right)\right)}{Z_t}$

[0065] where Z_tis a normalization factor, chosen so that D_t+1will be a distribution. The final hypothesis is then output as follows: $f\left(x,l\right)=\sum _{t=1}^T{\alpha }_t{h}_t\left(x,l\right)$

[0066] This embodiment of the present invention maintains a distribution D_tover {1, . . . , m}×Υ×Υ and denotes the weight for instance x_iand the pair l₀, l₁by D_t(i, l₀, l₁). This distribution is zero, however, except on the relevant triples (i, l₀, l₁) for which l₀, l₁is a crucial pair relative to (x_i, Y_i).

[0067] As before, weak hypotheses have the form h_t: X×Υ→; these are thought of as providing a ranking of labels as described above. The update rule, however, is a bit different. Let _l0, l₁be a crucial pair relative to (x_i, Y_i) and recall that D_tis zero in all other cases. Assuming momentarily that α_t>0, this rule decreases the weight D_t(i, _l0, l₁) if h_tgives a correct ranking (h_t(x_i,_l1)>h_t(x_i,_l0)), and increases the weight otherwise.

[0068] As for the Hamming loss, it can be shown that the empirical ranking loss of this algorithm is at most: $\prod _{t=1}^T{Z}_t$

[0069] Thus, as before, the goal in choosing α_tand h_tshould be minimization of: $Z_t=\sum _{i,l_0,l_1}{D}_t\left(i,l_0,l_1\right)\exp \left(\frac{1}{2}\alpha \left(h_t\left(x_i,l_0\right)-h_t\left(x_i,l_1\right)\right)\right)$

[0070] A description of the technique used for this purpose is provided in the next section.

[0071] The algorithm may be somewhat inefficient when there are many labels since, naively, |Y_i|·|Υ−Y_i| weights are maintained for each training example (x_i, Y_i), and each weight is updated on each round. Thus, the space complexity and time-per-second complexity can be as bad as θ(mk²). In fact, the same algorithm can be implemented using only θ(mk) space and time per round. By the nature of the updates, it can be shown that weights v_tonly need to be maintained over {1, . . . , m}×Υ. If _l0, l₁is a crucial pair relative to (x_i, Y_i), then:

[0072] D_t(i, l₀, l₁)=v_t(i, l₀)·v_t(i, l₁)

[0073] at all times. Recall that D_tis zero for all other triples (i,_l0, l₁). An example of pseudocode for this embodiment of the present invention is as follows:

[0074] Given: (x₁, Y₁), . . . , (x_m, Y_m) where x_i∈X and Y_iΥ

[0075] Initialize v₁(i, l)=(m·|Y_i|·|Υ−Y_i|)^−½

[0076] For t=1, . . . , T:

[0077] Train weak learner using distribution D_t

[0078] Get weak hypothesis h_t: X×Υ→

[0079] Choose α_t∈

[0080] Update: $v_{t+1}\left(i,l\right)=\frac{v_t\left(i,l\right)\exp \left(-\frac{1}{2}{\alpha }_t{Y}_i\left\{l\right\}{h}_t\left(x_i,l\right)\right)}{\sqrt{Z_t}}$

[0081] where: $Z_t=\sum _i\left[\left(\sum _{l\in Y_t}{v}_t\left(i,l\right)\exp \left(\frac{1}{2}{\alpha }_t{h}_t\left(x_i,l\right)\right)\right)\left(\sum _{l\in Y_t}{v}_t\left(i,l\right)\exp \left(-\frac{1}{2}{\alpha }_t{h}_t\left(x_i,l\right)\right)\right)\right]$

[0082] The final hypothesis is output as follows: $f\left(x,l\right)=\sum _{t=1}^T{\alpha }_t{h}_t\left(x,l\right)$

[0083] Note that all space requirements and all per-round computations are θ(mk), with the possible exception of the call to the weak learner. The next section demonstrates an implementation of a weak learner for text categorization that uses only mk weights. Thus the total time and space requirements of the classification algorithm are θ(mk).

[0084] Weak Hypotheses for Text Categorization

[0085] So far, the actual form and implementation of the weak learner, as well as the choice of the parameter α_t, have been left unspecified. In this section, four implementations of weak learners, three for AdaBoost.MH and one for AdaBoost.MR, are described. A system for multi-label text categorization may be used with any of the four methods described below.

[0086] In every case, the weak hypotheses have the same basic form as a one-level decision tree. The test at the root of this tree is a simple check for the presence or absence of a term in the given document. All words and pairs of adjacent words are potential terms. In fact, the implementation is capable of using arbitrarily long (sparse) n-grams, but the examples here are restricted to words and word bigrams for comparison purposes.

[0087] Based only on the outcome of this test, the weak hypothesis outputs predictions and confidences that each label is associated with the document. For example, going back to the news categorization example, a possible term can be “White House,” and the corresponding predictor may be:

[0088] If the term “White House” appears in the document then predict that the document belongs to “politics” with high confidence, to “business” with low confidence, and that it does not belong to “sports” with high confidence. If, on the other hand, the term does not appear in the document, then predict that it does not belong to any of the classes with low confidence.

[0089] Formally, denote a possible term by w, and let us define w∈x to mean that w occurs in document x. Based on the term, a weak hypothesis h may make predictions of the form: $h\left(x,l\right)=\{\begin{array}{cc}{c}_{0}l& \text{if}w\in x\\ c_{1}l& \text{if}w\in x\end{array}$

[0090] where the c_jl's are real numbers. The three weak learners described herein for AdaBoost.MH differ only with respect to possible restrictions which are placed on the values of these numbers.

[0091] The weak learners may search all possible terms. For each term, values c_jlare chosen as described below, and a score is defined for the resulting weak hypothesis. Once all terms have been searched, the weak hypothesis with the lowest score is selected and returned by the weak learner.

[0092] For AdaBoost.MH, this score will always be an exact calculation of Z_tas defined above with respect to AdaBoost.MH since minimization of Z_tis a reasonable guiding principle in the design of the weak learning algorithm. For AdaBoost.MR, there is no analytical solution for the problem of minimizing Z_t. Instead, an approximation of Z_tis used as described below.

[0093] AdaBoost.MH with Real-Valued Predictions

[0094] For the first weak learner according to an embodiment of the present invention, referred to herein as “real AdaBoost.MH,” permit unrestricted real-valued predictions c_jl. With minimization of Z_tin mind, the values c_jlshould be calculated as follows for a given term w: Let X₀={x: w∈x} and X₁={x: w∈x}. Given the current distribution D_t, calculate the following for each possible label l, for j∈{0, 1}, and for b∈{−1, +1}: $W_b^{\mathrm{jl}}=\sum _{i=1}^m\sum _{l\in {\rm Y}}{D}_t\left(i,l\right)\left[x_i\in X_j\bigwedge Y_i\left\{l\right\}=b\right]$

[0095] For readability of notation, the subscripts +1 and −1 in W₊₁^jland W₋₁^jlare abbreviated as W₊^jland W₋^jl, respectively. In words, W₊^jlis the weight, with respect to the distribution D_t, of the documents in partition X_jwhich are labeled by l, and W₋^jlis the weight of the documents in partition X_jwhich are not labeled by l.

[0096] Z_tis minimized for a particular term by choosing: $c_{\mathrm{jl}}=\frac{1}{2}\ln \left(\frac{W_{+}^{\mathrm{jl}}}{W_{-}^{\mathrm{jl}}}\right)$

[0097] and by setting α_t=1. These settings imply that: $Z_t=2\sum _{j\in \{0,1)}\sum _{l\in {\rm Y}}\sqrt{W_{+}^{\mathrm{jl}}{W}_{-}^{\mathrm{jl}}}$

[0098] Thus, the term w may be selected for which this value of Z_tis smallest.

[0099] In fact, it may be that W₊^jlor W₋^jlis very small, or even zero. In this case c_jlwill be very large or infinite in magnitude. In practice, such large predictions may cause numerical problems, and there may be theoretical reasons to suspect that large, overly confident predictions will increase the tendency to “over-fit.” To limit the magnitudes of the predictions, “smoothed” values may be used instead according to an embodiment of the present invention: $c_{\mathrm{jl}}=\frac{1}{2}\ln \left(\frac{W_{+}^{\mathrm{jl}}+ϵ}{W_{-}^{\mathrm{jl}}+ϵ}\right)$

[0100] According to another embodiment of the present invention, ∈ is set to 1/mk. Since both W₊^jland W₋^jlare bounded between 0 and 1, this has the effect of bounding c_jlby approximately ½ ln(1/∈).

[0101] AdaBoost.MH with Real-Valued Predictions and Abstaining

[0102] The real AdaBoost.MH method assigns confidence values both when a term appears in a document and when it does not. Thus, it employs a tacit assumption that the absence of a term carries information about the possible classes a document may belong to. However, a definitive prediction may be difficult in the latter case and it would be better to “abstain” by assigning confidence values of zero to all the classes. In other words, the weak hypothesis could effectively say “I don't know” whenever the given term does not appear in a document. This embodiment of the present invention is referred to herein as “real abstaining AdaBoost.MH.”

[0103] For a given term w, the weak learner chooses predictions c_1lfor documents which contain w, exactly as before. According to an embodiment of the present invention, these values are also smoothed as before. For the rest of the documents, the prediction values c_0lare all set to zero. Hence, the term w has no influence on the classification if it does not appear in the document. As before, α_tis set to 1.

[0104] Let: $W_0=\sum _{i:x_i\in X_0}{D}_t\left(i,l\right)$

[0105] be the weight of all the document that do not contain w. Then it can be shown that: $Z_t=W_0+2\sum _{l\in {\rm Y}}\sqrt{W_{+}^{1l}{W}_{-}^{1l}}$

[0106] and, as before, on each round a term w is selected for which the value Z_tis smallest.

[0107] One advantage of this weak learner over the first one is an improvement in the running time because only the documents that include a given term w are considered when computing Z_t. Since, typically, the number of documents that include a non-trivial term is only a small fraction of the training data, this may be faster than the previous one while maintaining comparable performance.

[0108] AdaBoost.MH with Discrete Predictions

[0109] The next weak learner forces the predictions cj_lof the weak hypotheses to be either +1 or −1. This is the more standard setting in which predictions do not carry confidences. This embodiment of the present invention is referred to herein as “discrete AdaBoost.MH.”

[0110] With this restriction on the range of the weak hypotheses, Z_tcan still be minimized for a given term w by setting:

[0111] c_jl=sign(W₊^jl−W₋^jl)

[0112] which can be viewed as a (weighted) majority vote over examples in block X_jfor each label l.

[0113] Let: $r_t=\sum _{j\in \left\{0,1\right\}}\sum _{l\in {\rm Y}}\left|W_{+}^{\mathrm{jl}}-W_{-}^{\mathrm{jl}}\right|$

[0114] Then it can be shown that, for the purposes of minimizing Z_t: ${\alpha }_t=\frac{1}{2}\ln \left(\frac{1+r_t}{1-r_t}\right)$

[0115] giving: $Z_t=\sqrt{1-r_t^2}$

[0116] AdaBoost.MR with Discrete Predictions

[0117] The following is a weak learner for AdaBoost.MR. As previously noted, the goal is to minimize Z_t. Unfortunately, the exact minimization of this quantity is not as straightforward as it was for AdaBoost.MH. Therefore only discrete predictions in {+1, −1} are considered, and an approximation, rather than an exact computation, for Z_tis used as a score. This method is refered to herein as “discrete AdaBoost.MR.”

[0118] For a given hypothesis h, let: $r_t=\frac{1}{2}\sum _{i,l_0,l_1}{D}_t\left(i,l_0,l_1\right)\left(h\left(x_i,l_1\right)-h\left(x_i,l_0\right)\right)$

[0119] Then, similar to the analysis for discrete AdaBoost.MH, it can be shown that $Z_t\le \sqrt{1-r_t^2}$ $\text{if:}$ ${\alpha }_t=\frac{1}{2}\ln \left(\frac{1+r_t}{1-r_t}\right)$

[0120] Since it is not known how to efficiently minimize Z_texactly, instead a weak hypothesis is found minimizes the upper bound $\sqrt{1-r_t^2}.$

[0121] This upper bound is then used as a score in choosing the best weak hypothesis.

[0122] The quantity r_tcan be computed efficiently in terms of the weights v_t. Let: $d_t\left(i,l\right)=\frac{1}{2}{v}_t\left(i,l\right)\sum _{l^l\text{:}{Y}_l\left\{l^l\right\}\ne Y_i\left\{l\right\}}{v}_t\left(i,l^l\right)$

[0123] Then it can be shown that: $r_t=\sum _{i,l}{d}_t\left(i,l\right)Y_i\left\{l\right\}h\left(x_i,l\right)$

[0124] Thus, for a particular term w: $c_{\mathrm{jl}}=\mathrm{sign}\left(\sum _{i\text{:}{x}_i\in X_j}{d}_t\left(i,l\right)Y_i\left\{l\right\}\right)$

[0125] which gives: $r_t=\sum _{j\in \left\{0,1\right\}}\sum _{l\in {\rm Y}}\left|\sum _{i\text{:}{x}_i\in X_j}{d}_t\left(i,l\right)Y_i\left\{l\right\}\right|$

[0126] The term w is selected which maximizes this quantity, and corresponding predictions are assigned. The parameter α_tis define as before.

[0127] According to embodiments of the present invention, a multi-label text categorization system may be used with, for example, any of the four weak learner implementations described above.

[0128] Although various embodiments are specifically illustrated and described herein, it will be appreciated that modifications and variations of the present invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention. For example, although particular system architectures were used to illustrate the present invention, it can be appreciated that other architectures may be used instead. Similarly, although particular equations and algorithms have been illustrated, other equations and algorithms will also fall within the scope of the invention. Finally, although software or hardware are described to control certain functions, such functions can be performed using either software, hardware or a combination of software and hardware, as is well known in the art. As is also known, software may be stored on a medium, such as, for example, a hard or floppy disk or a Compact Disk Read Only Memory (CD-ROM), in the form of instructions adapted to be executed by a processor. The instructions may be stored on the medium in a compressed and/or encrypted format. As used herein, the phrase “adapted to be executed by a processor” is meant to encompass instructions stored in a compressed and/or encrypted format, as well as instructions that have to be compiled or installed by an installer before being executed by the processor.