Method and apparatus for classification of high dimensional data

[0002] The present invention relates to computer software for classifying data. In particular, the invention uses flattening and addition of attributes to perform classification of sparse high dimensional data to more accurately predict a data class based on data attributes.

[0003] Classification is the process of assigning a data object, based on the data object's attributes, to a specific class from a predetermined set. Classification is a common problem studied in the field of statistics and machine learning. Some well-known classification methods are decision trees, statistical methods, rule induction, genetic algorithms, and neural networks.

[0004] A classification problem has an input dataset called the training set that includes a number of entries each having a number of attributes (or dimensions). A training set with n possible attributes is said to be n-dimensional. The objective is to use the training set to build a model of the class label based on the attributes, such that the model can be used to classify other data not from the training set. The model often takes the form of a decision tree, which is known in the art.

[0005] An example of a typical classification problem is that of determining a driver's risk for purposes of calculating the cost of automobile insurance. A single driver (or entry) has many associated attributes (or dimensions), such as age, gender, marital status, home address, make of car, model of car, type of car, etc. Using these attributes, an insurance company determines what degree of risk the driver imposes to the insurance company. The degree of risk is the resultant class to which the driver belongs.

[0006] Another example of a classification problem is that of classifying patients' diagnostic related groups (DRGs) in a hospital. That is, determining a hospital patient's final DRG based on the services performed on the patient. If each service that could be performed on the patient in the hospital is considered an attribute, the number of attributes (dimensions) is large but most attributes have a “not present” value for any particular patient because not all possible services are performed on every patient. Such an example results in a high-dimensional, sparse dataset.

[0007] A problem exists in that artificial ordering induced on the attributes lowers classification accuracy. That is, if two patients each have the same six services performed, but they are recorded in different orders in their respective files, a classification model would treat the two patients as two different cases, and the two patients may be assigned different DRGs.

[0008] Another problem that exists in classification pertaining to high-dimensional sparse datasets is that the complexity required to build a decision tree is high. There are often hundreds, even thousands or more, possible attributes for each entry. Thus, there are hundreds, or thousands, of possible attributes on which to base each node's splitting criterion in the decision tree. The large number of attributes directly contributes to a high degree of complexity required to build a decision tree based on each training set.

[0009] A goal of the invention is to provide a classification system that overcomes the identified problems.

[0010] In one embodiment, the present invention provides a method and apparatus for classifying high-dimensional data. The invention performs classification by storing the data in a computer memory, flattening the data into a boolean representation, and building a classification model based on the flattened data. The classification model can be a decision tree or other decision structure. In one aspect of the invention, large itemsets are used as additional attributes on which to base the decision structure. In another aspect of the invention, clustering is performed to provide additional attributes on which to base the decision structure.

[0011] In another embodiment, the invention provides a method and apparatus for classifying high-dimensional data using nearest neighbor techniques. The data is stored in a computer memory, flattened into a boolean representation, and classified based on the m nearest neighbors of an entry.

[0012] An advantage of the invention is that flattening the data removes any artificial ordering introduced into the data as a result of non-uniform recording procedures, thus yielding more accurate results.

[0013] Another advantage of the present invention is that the use of additional attributes based on large itemsets and clustering improves the accuracy of the resulting decision tree on which classification is based. This is achieved by determining which itemsets are large itemsets, and then using large itemsets as additional attributes on which a tree node's splitting criterion might be based. Clustering may also be used to increase accuracy in building a decision structure.

[0014] Additional advantages of the invention will become apparent upon reading the following detailed description and upon reference to the drawings, in which:

[0015] FIG. 1 depicts a computer system according to an embodiment of the invention;

[0016] FIG. 2 depicts a flow chart for classifying data;

[0017] FIG. 3 depicts a first sample set of raw data;

[0018] FIG. 4 depicts the raw data of FIG. 3 after it has been flattened according to the invention;

[0019] FIG. 5 depicts a second sample set of raw data, with the resultant class of each entry; and

[0020] FIG. 6 depicts a decision tree based on the raw data of FIG. 5.

[0021] Exemplary embodiments are described with reference to specific configurations. Those skilled in the art will appreciate that various changes and modifications can be made while remaining within the scope of the claims.

[0022] An exemplary embodiment of the invention is described with reference to FIGS. 1-6. FIG. 1 depicts a computer system 10 according to an embodiment of the invention. The computer system includes a processor (CPU) 12 coupled to a bus 13. A random access memory 14 and a hard disk memory 16 are also coupled to the bus 13 and are accessible by the processor. The hard disk 16 is configured to store programs and data necessary for the invention, as described below. An optional user interface 18 is provided for input of raw data. In one embodiment, the input/output devices include a keyboard, mouse, and monitor. An optional network interface 20 is also provided. It should be appreciated that raw data may be entered in various different ways, for example via the user interface 18 or the network interface 20.

[0023] The hard disk 16 is configured to store the program and data in the computer system 10. The memory, including the RAM and the hard disk, is divided into three primary components: communications procedures 22, control procedures 28, and data 42. The communications procedures include routines 24 and 26 for receiving raw data into the memory. The control procedures include routines 30-40 that perform the invention's classification functions. The data portion of the memory stores the raw data 44, the flattened data 46, the decision structure 48, and the pruned decision structure 50. These routines are described in greater detail below.

[0024] Operation of the invention is described with reference to the FIG. 2 flowchart. In step 102, the communications procedures 22 accept raw data as input. The raw data includes entries and a number of attributes for each entry. The attributes are typically categorical attributes, although they can be continuous attributes as well. An example of raw data is shown in FIG. 3. The raw data may or may not be ordered. That is, the attributes of each entry are not necessarily in any sort of ascending or descending order. Often, categorical attribute data is of a nature that is not suited to being sorted in an ascending or descending order. Also, for purposes of classification, the order of the attributes is often irrelevant. Data is said to be of the no dimension where n is the total number of different possible attributes of a data entry. In step 104, the data is organized by entry and attributes.

[0025] After the raw data is received, in step 106, the flattening procedures 30 then flatten the data into a boolean representation. Flattening converts the categorical attributes into boolean attributes by using a boolean representation of length n, where n is the total number of different possible attributes of a data entry. The n^thposition in the boolean representation is ‘1’ if the corresponding attribute is present in the raw data entry, and the no position representation is ‘0’ if the corresponding attribute is absent from the raw data entry. The resultant data is said the be n-dimensional. The raw data in FIG. 3 is depicted in flattened form in FIG. 4.

[0026] One aspect of the invention uses large itemsets. This aspect employs steps 108 and 110. Step 108 identifies the use of large itemsets. Step 110 uses the flattened data employing the large itemset procedures 34 to determine which attributes are large itemsets. Let l=I₁, I₂, . . . , I_nbe a set of binary attributes. Let T be the training set consisting of patterns. Each pattern t is a subset of l. A pattern t supports an itemset X if X⊂t. Itemsets of cardinality k are referred to as k-itemsets. The support of an itemset is a measure of statistical significance and is defined to be the fraction of patterns that contain the itemset. Large itemsets are itemsets that have support greater than a given threshold. Such a threshold is typically 3-10%, and may be determined based on the percentage that yields the most accurate decision tree. However, it should be appreciated that the threshold may be lower than 3% or higher than 10%.

[0027] Several known algorithms may be used to derive large itemsets. One such algorithm which may be used is the Apriori algorithm, as presented in Agrawal and Srikant, Fast Algorithms for Mining Association Rules, Proc. of the 20th International Conference on Very Large Databases, Santiago, Chile, 1994. The decision tree, as detailed below, is then generated also using large itemsets as attributes on which to base the splitting criterion during classification. If the support is chosen to be high enough, the number of large itemsets may be substantially smaller than the large number of initial attributes. In addition to increasing the accuracy of the decision tree, if the large itemsets are appropriate attributes on which to base the splittin criterion, the total time required to build the decision tree mey be substantially less than the time required to build a decision tree based on all of the initial attributes. Also, the resulting decision tree may be substantially smaller than a decision tree based on the initial attributes.

[0028] Another aspect of the invention uses clustering. This aspect employs steps 111 and 112. Step 111 identifies the use of clustering. Step 112 uses the flattened data employing the clustering procedures 35 to add clustering attributes to each data entry. Clustering is performed by first grouping the set of possible attributes into clusters. Next, each entry is assigned one or more weights pertaining to the entry's degree of inclusion in each cluster. These weights are added to the entry as additional attributes on which classification may be based.

[0029] For example, in a grocery store there might be 10,000 items which a consumer could buy. However, a typical consumer only purchases about thirty items at one time. Each of the 10,000 items which the consumer could have bought belongs to a cluster, such as meats, dairy, vegetable, bread, etc. Each of the about thirty items which the consumer did buy belongs to one of the predetermined clusters. The consumer is then assigned a weight for each cluster, based on the number of items bought which belong to a cluster versus the total number of items bought. The weights represent the degree to which the consumer belongs to each cluster.

[0030] Clustering is well known in the art. One clustering method which may be used is association rule hypergraph clustering, which is described in detail in Han, Karypis, Kumar, and Mobasher, Clustering Based on Association Rule Hypergraplhs, SIGMOD '97 Workshop on Research Issues on Data Mining and Knowledge Discovery, 1997.

[0031] Using the flattened data and any combination of large itemsets and clusters, including the combination of neither large itemsets nor clusters, in step 113 the classification procedures 32 build a model on which future classification is based. Decision trees, which are known in the art, are one form of such a model that may be created.

[0032] The creation of a decision tree includes two phases: the construction phase and the pruning phase. The construction phase requires that the training set be recursively partitioned into two or more subpartitions until a stopping criterion is met, e.g. each subpartition dominantly (or entirely) includes examples of one class. Thus, creation involves applying a splitting criterion to every internal node of the tree. An internal node is defined as any node that has at least one child node. These splitting criterion are determined by applying a predetermined splitting function. The splitting criterion at each internal node of the tree is based on one of the attributes in the set of possible attributes of the entries, including large itemset and cluster attributes, if used. The tree is created with a decreasing entropy as the nodes get farther away from the root node of the tree. The decision tree is the resulting hierarchical tree structure that is generated with the root representing the entire dataset. FIG. 6 depicts a decision tree created based on the sample data depicted in FIG. 5. In FIG. 6, the dataset evaluated at each node is in parentheses, with the splitting criterion below. It should be noted that a decision tree is not always symmetrical.

[0033] The creation and pruning of decision trees is known in the art, and examples of decision tree creation and pruning algorithms may be found in Mehta, Agrawal, and Rissanen, SLIQ: A Fast Scalable Classifier for Data Mining, Proc. of the Fifth International Conference on Extending Database Technology, Avignon, France; 1996.

[0034] In another embodiment of the invention, after the data has been flattened, a class determination for an entry e is made using a nearest neighbor technique. This is done by finding the m nearest neighbors of e, and then assigning the class of e based on the class assignments of the m nearest neighbors. The Euclidean distance between two data entries is used to find the nearest m neighbors, as represented by equation 1. $\begin{array}{cc}{\left[\sum _{i=1}^n{\left(x_i-y_i\right)}^2\right]}^{1/2}& \left(1\right)\end{array}$

[0035] Determination of nearest neighbors is known in the art. There are two main parameters: the value of m, generally between 3 and 5 inclusive, and the choice of which class to be assigned based on the class memberships of the m neighbors. There are three possible scenarios that may occur: all m neighbors belong to the same class, one class has a majority, or there is a tie. In the first case, the assigned class is the class of the m neighbors. In the second case, the assigned class is the class of the majority of the neighbors. In the third case, the tie can be broken arbitrarily. In another embodiment, in the second and third cases, all classes of the nearest neighbors are reported, and an appropriate choice can be made by an end user based on additional factors. In another aspect of the invention, more than one class may be assigned to the entry e.

[0036] In one aspect of the invention, the nearest neighbor attributes are weighted as will now be described. In the second and third cases above, the simple majority and arbitrary rules are modified to provide for weighted votes based on the inverse frequency of different classes present in the training set. That is, each vote is weighted inversely to the class' percentage representation in the training set. In order to avoid weighing an attribute too heavily, each dimension is generally normalized using the standard of deviation along that dimension.

[0037] The inventive apparatus and method were constructed and performed, with respect to flattening, on data derived from the in-patient stay records at a Hospital. The dataset consisted of two delivery related DRGs: delivery with complications and delivery without complications. As shown below, the raw data training set classification error rate was 11.6%. The raw data test set classification error rate was 14.9%. The flattened data training set classification error rate was 0.8%, and the flattened data test set classification error rate was 1.4%. Thus, flattening the data has a significant impact on the overall accuracy rate.

[0038] Having disclosed exemplary embodiments and the best mode, modifications and variations may be made to the disclosed embodiments while remaining within the scope of the present invention as defined by the following claims.