Active learning methods for evolving a classifier

[0001] The subject development relates to active learning methods and classifying systems for data items such as structured document systems and especially to such systems for adjusting a classifier for document systems wherein the documents or portions thereof can be characterized and classified for improved automated information retrieval. The development relates to a system and method for annotating document elements or adjusting classes of the classifier for the document data elements so the document and its contents can be more accurately categorized and stored, and thereafter better accessed upon selective demand.

[0002] In legacy document systems comprising substantial databases, such as where an entity endeavors to maintain an organized library of semi-structured documents for operational, research or historical purposes, the document files often have been created over a substantial period of time and storage is primarily for the purposes of representation in a visual manner to facilitate its rendering to a human reader. There are often no corresponding annotations to the document to facilitate its automated retrieval by some characterization or classification system sensitive to a recognition of the different logical and semantic constituent elements.

[0003] Accordingly, these foregoing deficiencies evidence a substantial need for somehow acquiring an improved system for logical recognition of content and semantic elements in semi-structured documents for better reactive presentations of the documents and response to retrieval, search and filtering tasks.

[0004] Concept models for annotating such systems usually start with a training set of annotations that can identify element instances in the document or data item being classified, for example, element instances such as author, title or abstract. Such annotations correspond to identification of distinctive features that can be determined to collectively define a class of the element instance which in turn can be interpreted to suggest the appropriate annotation. The training set originates from an annotator/expert involved in the classifying of the data items.

[0005] As the complexity and voluminous extent of documents or data collections increase, the difficulties in accurately and quickly classifying the data items in the collections as well as elements in the documents also increase. Better models for the annotating process need to be developed, which if were obtained through only manual efforts of the annotator/expert, would result in highly undesirable inefficiencies in evolving the annotating model. Accordingly, there is a need for a better machine implemented active learning method for evolving a classifier.

[0006] The subject development thus also relates to machine training of a classifying system. A wide number of machine learning techniques have also been applied to document classification. An example of these classifiers are neural networks, support vector machines [Joachims, Thorsten, “Text categorization with support vector machines: Learning with many relevant features”, Machine Learning: ECML-98. 10^th European Conference on Machine Learning, p. 137-42 Proceedings, 1998], genetic programming, Kohonen type self-organizing maps [Merkl, D., “Text classification with self-organizing maps: Some lessons learned”, Neurocomputing Vol. 21 (1-3), p. 61-77, 1998], hierarchical Bayesian clustering, Bayesian network [Lam, Wai and Low, Kon-Fan, “Automatic document classification based on probabilistic reasoning: Model and performance analysis”, Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Vol. 3, p. 2719-2723, 1997], and Naïve Bayes classifier [Li, Y. H. and Jain, A. K., “Classification of text documents”, Computer Journal, 41(8), p. 537-46, 1998]. The Naïve Bayes method has proven its efficiency, in particular, when using a small set of labeled documents and in the semi-supervised learning, when the class information is learned from the labeled and unlabeled data [Nigam, Kamal; Maccallum, Andrew Kachites; Thrun, Sebastian and Mitchell, Tom, “Text Classification from labeled and unlabeled documents using EM”, Machine Learning Journal, 2000].

[0007] Active learning refers to a framework where the learning algorithm selects the instances to be labeled and then included in the training set. It often allows a significant reduction in the amount of training data needed to train a supervised learning method. Instead of annotating random instances to produce the training set, the active learning suggests to annotate those instances that are expected to maximally benefit the supervised learning method.

[0008] The conventional principle of active learning assumes a predefined and fixed concept definition, where the concept refers to the set of classes and their interpretation. The most traditional situation is one of evaluation testbeds, where the concept is predefined and given by a set of classes and a fully annotated collection of examples. Such testbeds are used in different domains in order to test, compare, and eventually improve existing machine learning techniques.

[0009] The concept evolution for the annotating model is a change in the class set or a shift in their interpretation. Current systems disallow any concept evolution since any change makes inconsistent the previous concept, along with the associated learning model and training sets. It often requires to restart the training process or, in the best case, to revise a part of the training set concerned with the change.

[0010] On the other hand, a possibility to evolve a concept appears as very important in real applications. It often originates from the complexity of input collections and a certain flexibility or even fuzziness in the task definition. For example, in the domain of meta-data extraction from digital and scanned documents and the semantic annotation of Web pages, the design of a learning model starts with some initial “idea” and often goes through a sequence of different corrections and adjustments. Such evolution of the concept may be critical in pursuing the following goals:

[0011] 1. Refining the problem in a way that better corresponds to given collections, including a discovery of a hidden knowledge (new elements, finer interpretation of existing ones, relations between elements, etc.) that can beneficial for the final application, for faster learning, etc.

[0012] 2. Better matching quality constraints, imposed by the pricing and contracting causes. It is often preferable to recognize instances of a sub-class AA with 98% accuracy, than instances of a super-class A with accuracy 70%.

[0013] 3. If the modeling task is shared between different partners that follow (slightly) different concept definitions, the unification of their efforts and models might impose something similar to a concept change.

[0014] 4. If the deployment of extracted data is changed due to some external reasons, like the domain ontology update.

[0015] Unfortunately, any concept change makes a part of the annotations inconsistent. If the investment in annotated samples has been important, retraining a model from scratch represents a substantial cost. To avoid the re-start of the process or the re-annotation of “become-inconsistent” training examples, an active learning principle can assist a designer in pivoting the system toward the new concept definition and tuning up the associated learning model.

[0016] Accordingly, there is a need for improved methods and systems for retraining a maximum entrophy classifier when incremental changes are made to the definitions of the classes it must detect. The retraining should occur in an active learning framework and the system may choose new instances to be annotated in order to increase classification accuracy with less effort from human annotators.

[0017] According to aspects illustrated herein, there is provided a method for adjusting a system for classifying a data item in a data collection or an element instance within a document. Any element instance in the document is associated with a set of features indicative of accurate identification of the element instance. A training set comprising a subset of elements within the document labeled with class labels is provided to train a learning method for predicting class labels for unlabeled elements. If an accurate prediction of the class for the element instance is achieved, then the element instance can be accurately annotated so that the item can be identified with the set of predicted classes for selected element instances. Upon the identification of the data item with the training set, a confidence factor is computed that a selected element instance is accurately identified by the predicted class. When a selected element instance has a low confidence factor, a human annotator/expert is queried for a true class label of the instance. The training set is extended with the true labeled instance, iterating the identifying and computing for other element instances with low confidence factors is executed and a human annotator/expert is queried for a true class label of the instance. The training set is extended with the true labeled instance.

[0018] The changing of the features for the predicted class comprises an annotator/expert generating a concept evolution command for the redefining of the predicted class. The generated concept evolution command adjusts the training set class in an incremental manner so that a part of a feature definition for the changed predicted class is common with the training set class feature definition.

[0019] In accordance with another illustrated aspect, two different methods are disclosed for the changing and retraining of the subject classifying system, including a local approach and a global approach. The local approach associates a local model for each evolution command. The global approach includes an associated global model for the most recent changing of the associate features for the predicted class comprising issuing of a concept evolution command by the annotating expert.

[0020] According to further aspects illustrated herein, there is provided a method for evolving an annotating model for classifying a data item. A first annotating model is a composed as a training set comprised of a first set of selectively determinable annotations detectable within the document and the first concept evolution model. A learning algorithm is trained with the training set and the concept evolution model to predict classes for unlabeled elements within the document. The data item is then annotated with the training set by selectively applying annotations to selected element instances. A confidence factor is computed that the applied annotation is accurate. When an element instance is identified such that a corresponding applied annotation has a low confidence factor, the applied annotation is adjusted by changing the classifying defining features of the selected item. A second annotating model is then composed for more accurate classifying of the data item.

[0021] The composing and adjusting of the model are executed by a designer of the annotating model while the computing is machine implemented. The computing comprises determining a probability that a detected element instance corresponds to a predetermined class, and when these probabilities for all classes correspond to the confidence factor satisfying the predetermined condition of the uncertainty, the class is suggested for further annotating to the annotator/expert.

[0026] The subject developments relate to evolving/retraining methods that can be integrated into an active learning framework so that the classifying system can choose element instances to be alternatively annotated in order to increase classification accuracy with less effort from human annotators. Such classifying systems have an overall purpose of better organizing/storing/retrieving data items such as documents or computer stored records. The basic component for applying different annotation strategies in such systems is a supervised probabilistic algorithm and associated classifier C. Such cases typically consider the extraction of document fragments, like words/lines/segments in scanned and OCR-ed documents and words/leaves/sub-trees in HTML documents. For each unlabeled element (unannotated) element x in a source document, classifier C estimates conditional probabilities P(y_i|x) for all classes y_i ε Y. The basic probabilistic algorithm can be one of the following: the k-Nearest Neighbor, the Maximum Entropy, the multi-class Support Vector Machines, an assembly method, etc.

[0027] To produce probabilistic estimates for the element x, a Maximum Entropy framework is adopted, according to which the best model for estimating probability distributions from data is the one that is consistent with certain constraints derived from the training set, but otherwise makes the fewest possible assumptions. The distribution with the fewest possible assumptions is one with the highest entropy, and closest to the uniform distribution. Each constraint expresses some characteristic of the training set that should also be present in the learned distribution. The constraint is based on a binary feature, it constrains the expected value of the feature in the model to be equal to its expected value in the training set.

[0028] One important advantage of maximum entropy models is their flexibility, as they allow the extension of the rule system with additional syntactic, semantic and pragmatic features. Each feature f is binary and can depend on y ε Y and on any properties of the input elements x. In the case of textual annotation, the core features that express properties on the fragments, like f₁(x,y)=“1 if y is title and x has less then 10 words, 0 otherwise” are considered.

[0029] With the constraints based on the selected features f(x,y), the maximum entropy method attempts to maximize the conditional likelihood of P(y|x) represented as an exponential model:

$\begin{array}{cc}P\left(y\text{}x\right)=\frac{1}{Z_{\alpha }\left(x\right)}\exp \left(\sum _{\alpha }{\lambda }_{\alpha }·f_{\alpha }\left(x,y\right)\right)& \left(1\right)\end{array}$

where Z_α(x) is a normalizing factor to ensure that all the probabilities sum to 1,

$\begin{array}{cc}{Z}_{\alpha }\left(x\right)=\sum _y\exp \left(\sum _{\alpha }{\lambda }_{\alpha }f_{\alpha }\left(x,y\right)\right)& \left(2\right)\end{array}$

[0030] For the iterative parameter estimation of the Maximum Entropy exponential models, one often selects the quasi Newton methods, namely the Limited Memory BFGS method, which is observed to be more effective than the Generalized Iterative Scaling (“GIS”) and Improved Iterative Scaling (“IIS”) for NLP and information extraction tasks.

[0031] The active learning process by which the classifying concept model evolves involves an adoption of a major active learning principle, namely uncertainty based sampling. It is based on measuring the learner confidence on unlabeled instances. According to the principle, the classifier would benefit more on labeling and including instances on which it is more uncertain (i.e., less confident) when attempting to classify them. Uncertainty sampling reasons using probabilities P(y_i|x) are assigned by the classifier C to every possible class y_i ε Y on each unlabeled observation x.

[0032] The learner confidence of its predictions are measured using the maximum entropy (ME) principle. Given a probability distribution P(y_i|x) for an observation x, the ME-confidence metric is defined as follows:

$\begin{array}{cc}{\mathrm{conf}}_{\mathrm{me}}\left(x\right)=\sum _iP\left(y_i\text{}x\right)\log P\left(y_i\text{}x\right)& \left(3\right)\end{array}$

[0033] The ME-confidence metric yields negative values; it approximates 0 as the classifier becomes certain of its decision and returns low negative values for observations where the classifier is uncertain. There is no minimal value for conf_me(x). Indeed, it depends on a number of classes in Y,

inf_xεXconf_me(x)=log|Y|,

where |Y| is the number of classes in Y.

[0034] To cope with varying number of classes, below we propose a normalized version of the ME-confidence metric is proposed. The normalized version returns values between −1 (the maximum uncertainty) and 0 (the maximum certainty) for x:

$\begin{array}{cc}{\mathrm{conf}}_{\mathrm{me}}^n\left(x\right)=\frac{1}{\log Y}\sum _iP\left(y_i\text{}x\right)\log P\left(y_i\text{}x\right)& \left(4\right)\end{array}$

[0035] Once the confidence values are obtained according to the above metric for all unlabeled observations x, the m most uncertain observations denoted x_l^u, i=1, . . . , m are proposed to the designer (or a qualified annotator/expert), possibly with their projection on the document rendering. Once the designer labels all or some of x_l^u with classes y ε Y; the labeled instances (x_l^u, y) update the training set thus allowing retraining of the model, and can iteratively identify a new set of the most uncertain observations, and so on.

[0036] With particular reference to FIG. 1, it can be seen that data item 10 intended to be classified is initially annotated in accordance with a training set of predicted classes to be associated with element instances within the data item 10. Data item 10 is interfaced (what you see, is what you get interface “WYSIWYG”) with the annotation and concept evolution database 12 and an annotator/expert 14. The database 12 is further associated with an active learning system 16 which detects element instance features within the data item 10 for computing a confidence factor that the initial training set of predicted classes defined by element instance features within the data item, is acceptably accurate. For those instances having an undesirable confidence factor, as herein to be more fully explained, the active learner suggests alternative annotation/classification for the element instance 18 which can be communicated to the annotator/expert 14 who will make annotation actions by executing concept evolution commands.

[0037] It is assumed that the classifying model concept C evolves through a sequence of elementary events e_n, n=1, . . . , N where event e_n transforms the previous concept into a new one, C_n=e_n(C_n−i). Accordingly, each event en transforms the previous concept's class set Y^n−l into the new one, Yⁿ. Each event is explicitly declared by the designer and is assumed to be one of the following operations:

• • [0038] split (y,y₁, . . . , y_k), k>=2, replaces an existing class y ε Yⁿ⁻¹ with two or more classes y₁, . . . , y_k,Yⁿ={Yⁿ⁻¹−y} U {y₁, . . . , y_k}.
  • [0039] add(y,{y₁, . . . , y_k}),k>=2, adds a new class y. If the optional list of classes {y₁, . . . , y_k} is given, instances of y may be among those previously annotated with those classes. In no classes are given, class y can interfere with any class in Yⁿ⁻¹. The operation updates the class set as Yⁿ=Y^n−l U Y_n.
  • [0040] merge (y, y₁, . . . , y_k), k>=2, merges two or more existing classes y₁, . . . , y_k ε Y^n−l into a new class y, Yⁿ={Yⁿ⁻¹ U y}−{y₁, . . . , Y_k}.
  • [0041] remove (y, {y₁, . . . , y_k}), removes class y from Yⁿ⁻¹, Yⁿ=Yⁿ⁻¹−y_n. If the optional class list {y₁, . . . , y_k} is given, instances of y may be inherited by any of those classes. If no classes are given, instances of class y may be inherited by any class remaining in Yⁿ.

[0042] The above four operations are considered as a core set which can satisfy all major needs to express the concept evolution process. However, other operations may be included and are intended to be within the scope of this disclosure. For example, this operation set can be extended with a minimal effort by similar or derivate operations, like class renaming, shifting, etc.

[0043] Directed acyclic graphs are useful representations of the evolution of learning concepts. The concept evolution DAG has a unique root referring to step n=0. Each node in the DAG corresponds to a class occurring at least once during the evolution. There exist two types of arcs in the DAG. Solid arcs are associated with operations split and merge; doted arcs are used for operations add and remove (FIG. 2). A solid arc is interpreted as a replacement of the arc's source class with the target class (and possibly sharing this function with other classes). For a dotted arc, both source and target classes coexist, with the target class being a spin-off of the source one.

[0044] For any given DAG, the set of the most recent concept C^N is determined in a unique way. FIG. 2, explicitly indicated classes of Y^N in the DAG.

[0045] The operations update the DAG as follows. Any operation add {y,{y₁, . . . , y_k}) adds a node for y and connects all nodes y₁, . . . , y_k to the new node by dotted arcs. Split (y, y₁, . . . , y_k) creates k new nodes for y₁, . . . , y_k and connects node y to the new nodes with sold links. Remove (y,{y₁, . . . , y_k}) creates no nodes but connects the node for y to all nodes y₁, . . . , y_k by solid links. Finally, merge (y, y₁, . . . , y_k) creates a new node for class y and links all nodes y₁, . . . , y_k to node y by solid links.

[0046] Split, merge and delete operations erase one or more classes and make associated annotations obsolete. Add operation erases no classes, though it makes certain annotations obsolete.

[0047] For example, assume the designer builds a learning model for meta-data extraction from collections of scientific publications. FIG. 2 shows the DAG of the concept evolution. As the initial concept C¹, the designer selects classes, author, co-author, title, abstract and None. Then, after a deeper analysis of the documents, the designer decides to add the reference class (with add(reference, None,) operation). Next, the designer realizes that references have an inner structure and the designer needs only ref_title and ref_author within reference (add(ref_title,title) and (add(ref_author,title,), while the designer is not interested in other fragments of references (remove(reference, None)). Finally, when the model is trained, the client comes to require the merge of authors and co-authors (merge(author,author,co_author)) in order to easily map the extracted instances into the application ontology. The final concept C⁴ includes six classes, author, title, abstract, ref_author, ref^—author and None.

[0048] For example, assume the designer is training a model that separates instances of two classes y₁ and y₂, represented by stars and circles in FIG. 3. The model function is defined as a complex curb (or non-linear SVM kernels), and an active learning principle is used to estimate the parameters for the optimal separation curb. In FIG. 3, question marks point to the most uncertain elements according to the last model trained with the available training set. Table 1 reports details on some instances x and uncertainty estimation. It shows four instances x₁ to x₄ annotated with classes y₁ and y₂ (marked with ✓) and at least three unlabeled instances x₅ to x₇. Core features f₁ to f₃ are reported for all instances and are used to train the basic learning algorithm and evaluating the normalized ME-confidence conf_meⁿ(x). Observation x₇ is the most uncertain and therefore is the top candidate for new labeling.

TABLE 1
The model training board before the split command.
			Class/
	Annot	Features	Prediction	con fmenorm
Instances	status	f1	f2	f3	y1	y2	(x)
X1	valid	5	arial	23	✓		0.00
X2	valid	7	times	7		✓	0.00
X3	valid	1	arial	6		✓	0.00
X4	valid	8	arial	15		✓	0.00
X5	no	8	times	13	0.6	0.4	−0.97
X6	no	9	times	9	0.25	0.75	−0.81
X7	no	5	arial	20	0.45	0.55	−0.993
. . .	. . .	. . .	. . .	. . .	. . .	. . .	. . .

[0049] Assume now that the designer decides to split the class y₂ into two subclasses, y₂₁, and y₂₂. Assume that x₂ and x₃ previously marked as y₂ have been re-annotated as y₂,₂ and y_2,1, respectively (see FIG. 3). All other examples of y₂ including x₄ become inconsistent.

[0050] Provided that the designer keeps the same class of curves to separate instances of three classes, the concept migration endeavors to quickly and with minimal effort turn around the models and training sets to capture the new concept and to efficiently manage the sets of valid and obsolete annotations and unlabeled elements.

[0051] Two methods discussed below address these and other issues in order to support the concept evolution. One, a so-called local approach creates a local model for one or few events e_n in the concept evolution. Once a local model is created, its definition (the set of classes) remains unchanged till the end. Moreover, the associated training set S_n can be extended with new examples, but no annotations can become inconsistent within the model. Instead, the local models are assembled in order to make the uncertainty predictions

${\mathrm{pred}}_{\mathrm{me}}^n\left(y_i^N\text{}x\right)$

for the most recent concept C^N.

[0052] The alternative, global approach maintains a unique model addressing the most recent concept C^N. At any evolution event e_n, the global approach re-shapes the list of classes, according to the event. In order to cope with three sets of valid, inconsistent and unlabeled instances, the global approach extends the core feature set f(x,y) with evolution-relevant features f^ev(n,x,y).

[0053] With reference to the local approach, one local model is associated with each of the evolution events e_n, which in turn can be expressed by a group of operations introduced in the previous section. There exists a minimal set of local models covering the entire evolution process, and this minimal set can be uniquely determined from the sequence of evolution events. In such a minimal set, each event model Mⁿ corresponds to an internal node of the concept evolution DAG that has at least one dotted or two solid outgoing arcs.

[0054] There are three main issues relevant to the local approach:

• • [0055] 1. How to combine the local predictions of local models Mⁿ,n=1, . . . , N in order to get confidence estimations for the most recent model C^N?
  • [0056] 2. How to propagate newly labeled instances of C^N back to training sets Sⁿ of event models?
  • [0057] 3. How to manage training sets Sⁿ which are not aligned (what is a valid annotation for one event model may be an unlabeled instance for another one)? Can new annotation examples get in conflict with older examples?

[0058] For each class y_i^N ε Y^N in the most recent concept C^N, we consider the set II of simple (directed) paths in the DAG leading from the root to node y_i^N is considered. For each path π ε Π, all inner nodes in the DAG composing the path π, and corresponding event model M^j are considered. For each event model M^J in the path, one denotes as y_i^j a class from Y^j that “follows” the path π. This class y_i^j is actually an ascendant of y_i^N on C^j. Using estimations P^j(y_i*|x) by event models, one can estimate P(y_i^N|x) as follows:

$\begin{array}{cc}P\left(y_i^N\text{}x\right)=\sum _{\pi \in \Pi }^{}\prod _{j\in \pi }^{}P^j\left(y_i^j\text{}x\right)& \left(5\right)\end{array}$

[0059] To work with unaligned training sets Sⁿ, one treats all obsolete and unlabeled examples in a uniform manner. For each training example (x, y) ε Sⁿ, we consider that P(y|x)=1 and P(y′|x)=0 for all classes y′ different from y, y′ε Yⁿ.

[0060] Finally, once a low confidence instance x^u gets annotated with class y_i^N, the annotation (x^u, y_i^N) is propogated to all local models as follows. For all paths π ε Π we consider the corresponding local models M^j and ascendant classes y_i^j for y_i^N are considered. Then we update training sets Sⁿ with projected samples (x^u, y_i^j).

[0061] An active learning algorithm for the local approach comprises, while (labeling and unlabeled instances are available):

• • [0062] Train all event models Mⁿ with associated training sets Sⁿ
  • [0063] Classify all unlabeled instances x and obsolete annotations
  • [0064] Use models' estimations to get the most recent estimates using (5)
  • [0065] Calculate the uncertainty using the normalized ME-uncertainty metric using (4)
  • [0066] Select the m most uncertain instances x_l^u,l==l, . . . , m
  • [0067] Ask an expert to label the selected instances x_l^u
  • [0068] Project the labeled examples (x_l^u,y_i^N) on the local models M^j and update the local training sets S^j with (x_l^u, y_i^j)

[0069] Considering the example DAG in FIG. 2 and the most recent model C⁴ with the class set Y⁴={author, title, abstract, ref_author, ref_title, None}. One can dispose three event models associated with inner no-split nodes in the DAG. The first local model M¹ is associated with the DAG root. It classifies instances x into classes Y¹={author, co-author, title, abstract, None}. The local model M² is associated with the inner node None and is intended to recognize reference instances. Finally, the local model M³ is dedicated to recognizing ref_author and ref_title within references. Assume each of these models produce corresponding predictions. Finally, the inner nodes for author and co-author form no local models since they have only one solid outgoing arc.

[0070] For the most recent model with Y⁴:

P(author|x) =	P1 (author|x) + P(co_author|x)
P(title|x) =	P1(title|x)
P(abstract|x) =	P1(abstract|x)
P(ref_author|x) =	P1(None|x) × P2(reference|x) × P3(ref_author|x)
P(ref_title|x) =	P1(None|x;) × P2(reference|x) × P3(ref_title|x)
P(None|x) =	P1 (None\x) × (1 − P2(reference|x) ×
	x(P3(ref_author|x) + P3(ref_author|x))

[0071] Any element x annotated with class y_i^N ε Y^N can contribute in annotation for one or more local models. In the above example, an element x labeled as ref_title results in including three annotation pairs for three local models: (x,None) for M¹, (x,reference) for M², and (x,ref_title) for M³.

[0072] With reference to FIG. 4, it can be seen that local approach is implemented by the expert first emitting 60 a concept evolution command. A new event e^N node is added 62 to the concept evolution DAG and all DAG paths for each recent class y^N are detected. If the expert has annotated 64 some of the elements X_i^an, then all event models Mⁿ need be trained 66 with the associated training sets. For obsolete and unlabeled element instances, local model estimations are calculated 68; using formula (5) the most recent estimations are estimated and the ME-uncertainty is determined using formula (4). The m most uncertain instances x_i^u are then selected and communicated to the expert for annotator/expert annotation 70 of them.

[0073] With reference to the global approach, it maintains a unique model that addresses the most recent concept definition. At each evolution event, it changes the set of classes according to the operation issued and takes care of annotations that become obsolete. This is achieved through a dedicated feature extraction mechanism. It extends the conventional set of features f(x,y) describing the observations with evolution-relevant features f^ev(n,x,y).

[0074] The partial evidence of obsolete annotations are captured by introducing special features and adding them to the feature set used by the exponential model. The evolution-relevant features depend on the operation issued at step n:

• • [0075] For add (y,y₁, . . . , y_k) and merge (y,y₁, . . . , y_k), k features f_n^ev (x, y_k) are added. f_n^ev(x, y_k) is 1 if x; was annotated as y_k before step n, and 0 otherwise.
  • [0076] For split (y,y₁, . . . , y_k) and remove(y,y₁, . . . , y_k), one feature f^ev(n,x,y) is added; it is 1 if x; was annotated as y before the command was issued, 0 otherwise.

[0077] For example, according to the global approach, a set of core features F={f₁, f₂, f₃} is extended with the evolution feature f^ev(n,x,y₂) that returns 1 of x was annotated as y₂ before the split, and 0 otherwise. Then one trains the exponential model with the feature set F′=F U f^ev(n,x, f₂) and evaluates the uncertainty for all obsolete and unlabeled elements. Details of the new evaluation are reported in Table 2. According to it, the split reshapes the landscape of element uncertainty, with element x₆ becoming the most uncertain one, and not x₇ any longer.

[0078] The global approach has a benefit of maintenance of evolution features which are associated over relationships over

TABLE 2
The uncertainty estimation after the split.
In-	Annot	Features	Class/Prediction
stances	status	f1	f2	f3	fe(y2)	y1	y2,1	y2,2	con fmen
X1	valid	5	arial	23	0	✓			0.0
X2	valid	7	times	7	1			✓	0.0
X3	valid	1	arial	6	1		✓		0.0
X4	obsolete	8	arial	15	1	0.0	0.62	0.38	−0.60
X5	no	8	times	13	0	0.6	0.25	0.15	−0.85
X6	no	9	times	9	0	0.25	0.40	0.30	−0.97
X7	no	5	arial	20	0	0.45	0.45	0.1	−0.86
. . .	. . .	. . .	. . .	. . .		. . .	. . .		. . .

the evolution process. They are volatile and some or all of these features may become useless at some point and can be dropped off, for the sake of reduction of the space and calculation time, performance of ME algorithm.

[0079] An evolution feature f^ev(n,x,y_k) is called legacy if there is no instance x which status is partial and f^ev(n,x,y_k)=1. One can show that any legacy evolution feature can dropped from the feature set without any harm to the model performance. To keep track of evolution features, one maintains an index of obsolete annotations and check for legacy features before any model re-training.

[0080] With reference to FIG. 4, it can be seen that upon an expert emitting a concept evolution command 60 for an event e^N, evolutionary relevant features f^ev(N,x,y) are added and the legacy evolution-relevant feature is detected and removed 72. If the expert has annotated some elements x_i^an 64, then the global model M^N must be trained from the training set using both basic and evolution-relevant features 74. For obsolete and unlabeled instances the global model is used for estimating a most recent estimation 76 and determination of the ME-uncertainty using formula (4). Thereafter, the m most uncertain instances x_i^un are selected and the annotator/expert annotates them.

[0081] It will be appreciated that various of the above-disclosed and other features and functions, or alternatives thereof, may be desirably combined into many other different systems or applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.