Data Analysis Computer System and Method For Fast Discovery Of Multiple Markov Boundaries

[0001] Benefit of U.S. Provisional Application No. 61/791,654 filed on Mar. 15, 2013 is hereby claimed.

[0002] The field of application of the invention is data analysis especially as it applies to (so-called) “Big Data” (see sub-section 1 “Big Data and Big Data Analytics” below). The methods, systems and overall technology and knowhow needed to execute data analyses is referred to in the industry by the term data analytics. Data analytics is considered a key competency for modern firms [1]. Modern data analytics technology is ubiquitous (see sub-section 3 below “Specific examples of data analytics application areas”). Data analytics encompasses a multitude of processes, methods and functionality (see sub-section 2 below “Types of data analytics”).

[0003] Data analytics cannot be performed effectively by humans alone due to the complexity of the tasks, the susceptibility of the human mind to various cognitive biases, and the volume and complexity of the data itself. Data analytics is especially useful and challenging when dealing with hard data/data analysis problems (which are often described by the term “Big Data”/“Big Data Analytics” (see sub-section 1 “Big Data and Big Data Analytics”).

[0004] Big Data Analytics problems are often defined as the ones that involve Big Data Volume, Big Data Velocity, and/or Big Data Variation [2].

• • [0005] Big Data Volume may be due to large numbers of variables, or big numbers of observed instances (objects or units of analysis), or both.
  • [0006] Big Data Velocity may be due to the speed via which data is produced (e.g., real time imaging or sensor data, or online digital content), or the high speed of analysis (e.g., real-time threat detection in defense applications, online fraud detection, digital advertising routing, high frequency trading, etc.).
  • [0007] Big Data Variation refers to datasets and corresponding fields where the data elements, or units of observations can have large variability that makes analysis hard. For example, in medicine one variable (diagnosis) may take thousands of values that can further be organized in interrelated hierarchically organized disease types.

[0008] According to another definition, the aspect of data analysis that characterizes Big Data Analytics problems is its overall difficulty relative to current state of the art analytic capabilities. A broader definition of Big Data Analytics problems is thus adopted by some (e.g., the National Institutes of Health (NIH)), to denote all analysis situations that press the boundaries or exceed the capabilities of the current state of the art in analytics systems and technology. According to this definition, “hard” analytics problems are de facto part of Big Data Analytics [3].

[0009] The main types of data analytics [4] are:

• • [0010] a. Classification for Diagnostic or Attribution Analysis: where a typically computer-implemented system produces a table of assignments of objects into predefined categories on the basis of object characteristics.
    • [0011] Examples: medical diagnosis; email spam detection; separation of documents as responsive and unresponsive in litigation.
  • [0012] b. Regression for Diagnostic Analysis: where a typically computer-implemented system produces a table of assignments of numerical values to objects on the basis of object characteristics.
    • [0013] Examples: automated grading of essays; assignment of relevance scores to documents for information retrieval; assignment of probability of fraud to a pending credit card transaction.
  • [0014] c. Classification for Predictive Modeling: where a typically computer-implemented system produces a table of assignments of objects into predefined categories on the basis of object characteristics and where values address future states (i.e., system predicts the future).
    • [0015] Examples: expected medical outcome after hospitalization; classification of loan applications as risky or not with respect to possible future default; prediction of electoral results.
  • [0016] d. Regression for Predictive Modeling: where a typically computer-implemented system produces a table of assignments of numerical values to objects on the basis of object characteristics and where values address future states (i.e., system predicts the future). Examples: predict stock prices at a future time; predict likelihood for rain tomorrow; predict likelihood for future default on a loan.
  • [0017] e. Explanatory Analysis: where a typically computer-implemented system produces a table of effects of one or more factors on one or more attributes of interest; also producing a catalogue of patterns or rules of influences.
    • [0018] Examples: analysis of the effects of sociodemographic features on medical service utilization, political party preferences or consumer behavior.
  • [0019] f. Causal Analysis: where a typically computer-implemented system produces a table or graph of causes-effect relationships and corresponding strengths of causal influences describing thus how specific phenomena causally affect a system of interest.
    • [0020] Example: causal graph models of how gene expression of thousands of genes interact and regulate development of disease or response to treatment; causal graph models of how socioeconomic factors and media exposure affect consumer propensity to buy certain products; systems that optimize the number of experiments needed to understand the causal structure of a system and manipulate it to desired states.
  • [0021] g. Network Science Analysis: where a typically computer-implemented system produces a table or graph description of how entities in a mg system inter-relate and define higher level properties of the system.
    • [0022] Example: network analysis of social networks that describes how persons interrelate and can detect who is married to whom; network analysis of airports that reveal how the airport system has points of vulnerability (i.e., hubs) that are responsible for the adaptive properties of the airport transportation system (e.g., ability to keep the system running by rerouting flights in case of an airport closure).
  • [0023] h. Feature selection, dimensionality reduction and data compression: where a typically computer-implemented system selects and then eliminates all variables that are irrelevant or redundant to a classification/regression, or explanatory or causal modeling (feature selection) task; or where such as system. reduces a large number of variables to a small number of transformed variables that are necessary and sufficient for classification/regression, or explanatory or causal modeling (dimensionality reduction or data compression).
    • [0024] Example: in order to perform web classification into family-friendly ones or not, web site contents are first cleared of all words or content that is not necessary for the desired classification.
  • [0025] i. Subtype and data structure discovery: where analysis seeks to organize objects into groups with similar characteristics or discover other structure in the data.
    • [0026] Example: clustering of merchandize such that items grouped together are typically being bought together; grouping of customers into marketing segments with uniform buying behaviors.
  • [0027] j. Feature construction: where a typically computer-implemented system pre-processes and transforms variables in ways that enable the other goals of analysis. Such pre-processing may be grouping, abstracting, existing features or constructing new features that represent higher order relationships, interactions etc.
    • [0028] Example: when analyzing hospital data for predicting and explaining high-cost patients, co-morbidity variables are grouped in order to reduce the number of categories from thousands to a few dozen which then facilitates the main (predictive) analysis; in algorithmic trading, extracting trends out of individual time-stamped variables and replacing the original variables with trend information facilitates prediction of future stock prices.
  • [0029] k. Data and analysis parallelization, chunking, and distribution: where a typically computer-implemented system performs a variety of analyses (e.g., predictive modeling, diagnosis, causal analysis) using federated databases, parallel computer systems, and modularizes analysis in small manageable pieces, and assembles results into a coherent analysis. Example: in a global analysis of human capital retention a world-wide conglomerate with 2,000 personnel databases in 50 countries across 1,000 subsidiaries, can obtain predictive models for retention applicable across the enterprise without having to create one big database for analysis.

[0030] The following Listing provides examples of some of the major fields of application for the invented system specifically, and Data Analytics more broadly [5]:

• • [0031] 1. Credit risk/Creditworthiness prediction.
  • [0032] 2. Credit card and general fraud detection.
  • [0033] 3. Intention and threat detection.
  • [0034] 4. Sentiment analysis.
  • [0035] 5. Information retrieval filtering, ranking, and search.
  • [0036] 6. Email spam detection.
  • [0037] 7. Network intrusion detection.
  • [0038] 8. Web site classification and filtering.
  • [0039] 9. Matchmaking.
  • [0040] 10. Predict success of movies.
  • [0041] 11. Police and national security applications
  • [0042] 12. Predict outcomes of elections.
  • [0043] 13. Predict prices or trends of stock markets.
  • [0044] 14. Recommend purchases.
  • [0045] 15. Online advertising. 16, Human Capital/Resources: recruitment, retention, task selection, compensation.
  • [0046] 17. Research and Development.
  • [0047] 18. Financial Performance.
  • [0048] 19. Product and Service Quality.
  • [0049] 20. Client management (selection, loyalty, service)
  • [0050] 21. Product and service pricing.
  • [0051] 22. Evaluate and predict academic performance and impact.
  • [0052] 23, Litigation: predictive coding, outcome/cost/duration prediction, bias of courts, voire dire.
  • [0053] 24. Games (e.g., chess, backgammon, jeopardy).
  • [0054] 25. Econometrics analysis.
  • [0055] 26. University admissions modeling.
  • [0056] 77. Mapping fields of activity.
  • [0057] 28. Movie recommendations.
  • [0058] 29. Analysis of promotion and tenure strategies,
  • [0059] 30, intension detection and lie detection based on fMRI readings.
  • [0060] 31. Dynamic Control (e.g., autonomous systems such as vehicles, missiles; industrial robots; prosthetic limbs).
  • [0061] 32. Supply chain management.
  • [0062] 33. Optimizing medical outcomes, safety, patient experience, cost, profit margin in healthcare systems.
  • [0063] 34, Molecular profiling and sequencing based diagnostics, prognostics companion drugs and personalized medicine.
  • [0064] 35. Medical diagnosis, prognosis and risk assessment
  • [0065] 36. Automated grading of essays.
  • [0066] 37. Detection of plagiarism.
  • [0067] 38, Weather and other physical phenomena forecasting.

[0068] With regards to methods for Markov boundary discovery, they constitute one of the most important recent developments in pattern recognition and applied modeling, primarily because they offer a principled solution to the variable/feature selection problem and also give insight about local causal structure. The present invention is a novel fast method to discover multiple Markov boundaries of the response variable. The usefulness of the invention is first demonstrated in simulated data where the Markov boundaries of the response variable are known exactly. Then the usefulness of the invention is demonstrated with 13 real datasets from a diversity of application domains, where the invention can efficiently identify multiple Markov boundaries of the response variable. The resulting Markov boundaries can be used (a) to predict response variables of interest, (b) to develop highly compressed predictive models of the response variables, and (c) to understand what factors influence the response variables of interests and how to manipulate the system toward desired behaviors.

[0069] The invention can be applied to practically any field where discovery of causal or predictive models and/or feature selection is desired because it relies on extremely broad distributional assumptions that are valid in numerous fields. Because the discovery of Markov Boundaries also facilitates model conversion and explanation, inference and practically all aspects of data analytics, the invention is applicable and useful all the above mentioned types of data analysis and application areas.

[0070] The problem of variable/feature selection is of fundamental importance in applied machine learning, especially when it comes to analysis, modeling, and discovery from high-dimensional datasets [6, 7]. In addition to the promise of cost-effectiveness (as a result of reducing the number of observed variables), two major goals of variable selection are to improve the predictive performance of classification/regression models and to provide a better understanding of the data-generative process [6]. A state of the art class of filter methods approaches the solution of the variable selection problem by identification of a Markov boundary of the response variable of interest [8-14]. The Markov boundary M is a minimal set of variables conditioned on which all the remaining variables in the dataset, excluding the response variable T, are rendered statistically independent of the response variable T. Under certain assumptions about the learner and the loss function, Markov boundary is the solution of the variable selection problem [12], i.e. it is the minimal set of variables with optimal predictive performance for the current distribution and response variable. Furthermore, in faithful distributions, Markov boundary corresponds to a local causal neighborhood of the response variable and consists of all its direct causes, effects, and causes of the direct effects [12, 15].

[0071] An important theoretical result states that if the distribution satisfies the intersection property, then it is guaranteed to have a unique Markov boundary of the response variable [16]. Faithful distributions, which constitute a subclass of distributions that satisfy the intersection property, also have a unique Markov boundary [12, 15]. However, some real-life distributions contain multiple Markov boundaries and thus violate the intersection property and faithfulness condition. For example, a phenomenon ubiquitous in analysis of high-throughput molecular data, known as the “multiplicity” of molecular signatures (i.e., different gene/biomarker sets perform equally well in terms of predictive accuracy of phenotypes) suggests existence of multiple Markov boundaries in these distributions [17-19]. Likewise, many engineering systems such as digital circuits and engines typically contain deterministic components and thus can lead to multiple Markov boundaries [20, 21].

[0072] Even though there are several well-developed methods for learning a single Markov boundary [8-14], little research has been done in development of methods for identification of multiple Markov boundaries. The most notable advances in the field are stochastic Markov boundary methods that involve running multiple times either a standard or approximate Markov boundary induction method initialized with a random seed, e.g. KIAMB [11], EGS-NCMIGS and EGS-CMIM [22]. Another approach exemplified in the EGSG method [A] involves first grouping variables into multiple clusters such that each cluster (i) has variables that are similar to each other and (ii) contributes “unique” information about the response variable, and then randomly sampling a representative from each cluster for the output Markov boundaries. In genomics data analysis, researchers try to induce multiple variable sets (that sometimes approximate Markov boundaries) via application of a standard variable selection method to resampled data, e.g., bootstrap samples [24-26]. Finally, other bioinformatics researchers proposed a multiple variable set selection method that iteratively applies a standard variable selection method after removing from the data all variables that participate in the previously discovered variable sets with optimal classification performance [27]. The above early approaches are either highly heuristic and/or cannot be practically used to induce multiple Markov boundaries in high-dimensional datasets with relatively small sample size.

[0084] The inventive methods iTIE* and Generalized-iTIE* are shown in FIGS. 1 and 2, respectively. iTIE* is a configuration of the method Generalized-iTIE* which has the following inclusion sub-process in step 2:

• • [0085] a) sort in descending order the variables in the priority queue according to their pairwise association with the response variable T;
  • [0086] b) remove from the priority queue variables with zero association with the response variable T;
  • [0087] c) insert in the seed Markov boundary M the highest-priority variable A in the priority queue and remove it from the priority queue;
    the following elimination/analysis sub-process in step 3:
  • [0088] a) if the priority queue is empty, then remove every member A of the seed Markov boundary M that is probabilistically independent of the response variable T given a subset B of the remaining variables in M;
  • [0089] b) else if the priority queue is not empty, then if the last variable A that entered the seed Markov boundary M is probabilistically independent from the response variable T given a subset B of the rest of the variables in M, then remove A from M;
  • [0090] c) If a variable A that has been removed from M in the above step b) because A is probabilistically independent of the response variable T given an subset B of M, record in the equivalency catalogue Θ that A and B contain equivalent information (with respect to T) if B is probabilistically independent of the response variable T given A;
    its interleaving sub-process consists of repeating inclusion and elimination/analysis strategies until the priority queue is empty.

[0091] Consider running the iTIE* method on data D generated from the example causal Bayesian network shown in FIG. 3. The response variable T is directly caused by C, D, F. The underlying distribution is such that variables A and C contain equivalent information about T; likewise variables B and D contain equivalent information about T. iTIE*, when applied to data D would output correctly 4 Markov boundaries of T: (A, B, F), (A, D, F), (C, B, F), and (C, D, F).

[0092] In what follows, we present an empirical evaluation of methods for extraction of multiple

[0093] Markov boundaries and variable sets. The evaluated methods and their parameterizations are shown in FIG. 4. These methods were chosen for our evaluation as they are the current state-of-the-art techniques for discovery of multiple Markov boundaries and variable sets.

[0094] All experiments involving assessment of classification performance were executed by holdout validation or cross-validation (see below), whereby Markov boundaries and variable sets are discovered in a training subset of data samples (training set), classification models based on the above variables are also developed in the training set, and the reported performance of classification models is estimated in an independent testing set. Assessment of classification performance of the extracted Markov boundaries and variable sets was done in the presented validation using Support Vector Machines (SVMs) [28]. We chose to use SVMs due to their excellent empirical performance across a wide range of application domains (especially with high-dimensional data and relatively small sample sizes), regularization capabilities, ability to learn both simple and complex classification functions, and tractable computational time [28-31]. When the response variable was multiclass, we applied SVMs in one-versus-rest fashion [30]. We used libSVM v.2.9.1 (http://www.csie.ntu.edu.tw/˜cjlin/libsvm/) implementation of SVMs in all experiments [32]. Polynomial kernels were used in SVMs as they have shown good classification performance across the data domains considered in this study. The degree d of the polynomial kernel and the penalty parameter C of SVM were optimized by cross-validation on the training data. Each variable in a dataset was scaled to [0, 1] range to facilitate SVM training. The scaling constants were computed on the training set of samples and then applied to the entire dataset.

[0095] Below we present an evaluation of methods for extraction of multiple Markov boundaries and variable sets in simulated data. Simulated data allows us to evaluate methods in a controlled setting where the underlying causal process and all Markov boundaries of the response variable T are known exactly. Two datasets were used in this evaluation. One of these datasets, referred to as TIED, was previously used in an international causality challenge [33]. TIED contains 30 variables, including the response variable T. The underlying causal graph and its parameterization are given in [33] There are 72 distinct Markov boundaries of T. Each Markov boundary contains 5 variables: variable X₁₀ and one variable from each of the four subsets (X₁, X₂, X₃, X₁₁), (X₅, X₉), (X₁₂, X₁₃, X₁₄) and (X₁₉, X₂₀, X₂₁). Another simulated dataset, referred to as TIED1000, contains 1,000 variables in total and was generated by the causal process of TIED augmented with an additional 970 variables that have no association with T. TIED1000 has the same set of Markov boundaries of T as TIED. TIED1000 allows us to study the behavior of different methods for learning multiple Markov boundaries and variable sets in an environment where the fraction of variables carrying relevant information about T is small.

[0096] For each of the two datasets, 750 observations were used for discovery of Markov boundaries/variable sets and training of the SVM classification models of the response variable T (with the goal to predict its values from the inferred Markov boundary variables), and an independent testing set of 3,000 observations was used for evaluation of the models' classification performance.

[0097] All methods for extracting multiple Markov boundaries and variable sets were assessed based on the following six performance criteria:

• • [0098] I. The number of distinct Markov boundaries/variable sets output by the method.
  • [0099] II. The average size of an output Markov boundary/variable set (number of variables).
  • [0100] III. The number of true Markov boundaries identified exactly, i.e., without false positives and false negatives.
  • [0101] IV. The average Proportion of False Positives (PFP) in the output Markov boundaries/variable sets.
  • [0102] V. The average False Negative Rate (FNR) in the output Markov boundaries/variable sets.
  • [0103] VI. The average SVM classification performance (weighted accuracy) over all output Markov boundaries/variable sets. We also compared the average classification performance of the SVM models with the maximum a posteriori classifier in the true Bayesian network (denoted as MAP-BN) using the same data sample.

[0104] As can be seen in FIGS. 5-7, iTIE* identified exactly all and only true Markov boundaries of T in both simulated datasets, and their classification performance with the SVM classifier was statistically comparable to performance of the MAP-BN classifier. None of the comparator methods, regardless of the number of Markov boundaries/variable sets output, were able to identify exactly any of the 72 true Markov boundaries, except for Resampling+RFE (without statistical comparison) and IR-HITON-PC that identified exactly 1-2 out of 72 true Markov boundaries, depending on the dataset. Overall prior methods had either large proportion of false positives or large false negative rate, and often their classification performance was significantly worse that the performance of the MAP-BN classifier.

[0105] For evaluation of methods for learning multiple Markov boundaries and variable sets in real data, we used 13 datasets that cover a broad range of application domains (clinical outcome prediction, gene expression, proteomics, drug discovery, text categorization, digit recognition, ecology and finance), dimensionalities (from 86 to over 100,000), and sample sizes (from hundreds to thousands) that are representative of those appearing in practical applications. These datasets have recently been used in a broad benchmark [8] of the current state-of-the-art single Markov boundary induction and feature selection methods, which is another reason why we chose to use the same data in this study. The datasets are described in detail in FIG. 8. The datasets were preprocessed (imputed, discretized, etc.) as described in [8].

[0106] In datasets with relatively large sample sizes (>600), classification performance of the output Markov boundaries and variable sets was estimated by holdout validation with 75% of samples used for Markov boundary/variable set induction and SVM classifier training, and the remaining 25% of samples used for estimation of classification performance. In small-sample datasets, 10-fold cross-validation was used instead. Markov boundary/variable set induction and classifier training were both performed on the training sets from the 10-fold cross-validation design, with classification performance being subsequently estimated on the respective testing sets.

[0107] Evaluation of Markov boundary/variable selection methods in real data is challenging due to the lack of knowledge of the true Markov boundaries. In practical applications, however, the interest typically lies in the most compact subsets of variables that give the highest classification performance for reasonable and widely used classifiers [6]. This consideration motivated the following two primary evaluation criteria (with the averages taken over all Markov boundaries/variable sets output by each method):

• • [0108] I. The average Proportion of Variables (PV) in the output Markov boundaries/variable sets.
  • [0109] II. The average classification performance (AUC) of the output Markov boundaries/variable sets.

[0110] In addition to the above two primary criteria, in some problems we are also interested in extracting as many of the maximally compact and predictive variable sets (i.e., optimal solutions to the variable selection problem) as possible.

[0111] Detailed results of iTIE* are shown in FIG. 9 and comparison with other methods is given in FIG. 10. As can be seen, iTIE* extracted multiple compact Markov boundaries with high classification performance and surpassed all other methods on the combined (PV, AUC) criterion.

• • [0112] AUC—Area under ROC curve (classification performance metric)
  • [0113] CV—Cross-validation (method for estimating classification performance)
  • [0114] EGS-CMIM—Ensemble gene selection with conditional mutual information maximization criterion (method for selecting multiple variable sets)
  • [0115] EGS-NCMIGS—Ensemble gene selection with normalized conditional mutual information gene selection (method for selecting multiple variable sets)
  • [0116] EGSG—Ensemble gene selection by grouping (method for selecting multiple variable sets)
  • [0117] FNR—False negative rate (Markov boundary discovery performance metric)
  • [0118] Generalized-iTIE*—Generalized individual target information equivalency (method for discovery of multiple Markov boundaries)
  • [0119] IR-HITON-PC—Iterative removal with HITON-PC (method for discovery of multiple Markov boundaries)
  • [0120] IR-SPLR—Iterative removal with sparse logistic regression (method for selecting multiple variable sets)
  • [0121] iTIE*—individual target information equivalency (method for discovery of multiple Markov boundaries)
  • [0122] KIAMB—K-incremental association Markov boundary (method for discovery of multiple Markov boundaries)
  • [0123] MAP-BN—Maximum a posteriori classification method in the true/data generating Bayesian network
  • [0124] MB—Markov boundary (variable set)
  • [0125] PFP—Proportion of false positives (Markov boundary discovery performance metric)
  • [0126] PV—Proportion of variables relative to the number of variables in the original dataset before variable selection
  • [0127] Resampling+RFE—Resampling followed by application of support vector machines-based recursive feature elimination (method for selecting multiple variable sets)
  • [0128] Resampling+UAF—Resampling followed by application of univariate attribute filtering (method for selecting multiple variable sets)
  • [0129] SVM—Support vector machines classification method
  • [0130] TIED—Target information equivalency dataset, original version
  • [0131] TIED1000—Target information equivalency dataset, version with 1,000 variables
  • [0132] VS—Variable set

[0133] The relationships, correlations, and significance (thereof) discovered by application of the method of this invention may be output as graphic displays (multidimensional as required), probability plots, linkage/pathway maps, data tables, and other methods as are well known to those skilled in the art. For instance, the structured data stream of the method's output can be routed to a number of presentation, data/format conversion, data storage, and analysis devices including but not limited to the following: (a) electronic graphical displays such as CRT, LED, Plasma, and LCD screens capable of displaying text and images; (b) printed graphs, maps, plots, and reports produced by printer devices and printer control software; (c) electronic data files stored and manipulated in a general purpose digital computer or other device with data storage and/or processing capabilities; (d) digital or analog network connections capable of transmitting data; (e) electronic databases and file systems. The data output is transmitted or stored after data conversion and formatting steps appropriate for the receiving device have been executed.

[0134] Due to large numbers of data elements in the datasets, which the present invention is designed to analyze, the invention is best practiced by means of a general purpose digital computer with suitable software programming (i.e., hardware instruction set) (FIG. 11 describes the architecture of modern digital computer systems). Such computer systems are needed to handle the large datasets and to practice the method in realistic time frames. Based on the complete disclosure of the method in this patent document, software code to implement the invention may be written by those reasonably skilled in the software programming arts in any one of several standard programming languages including, but not limited to, C, Java, and Python. In addition, where applicable, appropriate commercially available software programs or routines may be incorporated. The software program may be stored on a computer readable medium and implemented on a single computer system or across a network of parallel or distributed computers linked to work as one. To implement parts of the software code, the inventors have used MathWorks Matlab® and a personal computer with an Intel Xeon CPU 2.4 GHz with 24 GB of RAM and 2 TB hard disk.

• [0135] 1. Davenport T H, Harris J G: Competing on analytics: the new science of winning: Harvard Business Press; 2013.
• [0136] 2. Douglas L: The Importance of ‘Big Data’: A Definition. Gartner (June 2012) 2012.
• [0137] 3. NIH Big Data to Knowledge (BD2K) [http://bd2k.nih.gov/about_bd2k.html#bigdata]
• [0138] 4. Provost F, Fawcett T: Data Science for Business: What you need to know about data mining and data-analytic thinking: “O'Reilly Media, Inc.”; 2013.
• [0139] 5. Siegel E: Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die: John Wiley & Sons; 2013.
• [0140] 6. Guyon I, Elisseeff A: An introduction to variable and feature selection. Journal of Machine Learning Research 2003, 3(1):1157-1182.
• [0141] 7. Kohavi R, John G H: Wrappers for feature subset selection. Artificial Intelligence 1997, 97(1-2):273-324.
• [0142] 8. Aliferis C F, Statnikov A, Tsamardinos I, Mani S, Koutsoukos X D: Local Causal and Markov Blanket Induction for Causal Discovery and Feature Selection for Classification. Part I: Algorithms and Empirical Evaluation. Journal of Machine Learning Research 2010, 11:171-234.
• [0143] 9. Aliferis C F, Tsamardinos I, Statnikov A: HITON: a novel Markov blanket algorithm for optimal variable selection. AMIA 2003 Annual Symposium Proceedings 2003: 21-25.
• [0144] 10. Mani S, Cooper G F: Causal discovery using a Bayesian local causal discovery algorithm. Medinfo 2004 2004, 11(Pt 1):731-735.
• [0145] 11. Pena J, Nilsson R, Bj″rkegren J, Tegn, r J: Towards scalable and data efficient learning of Markov boundaries. International Journal of Approximate Reasoning 2007, 45(2): 211-232.
• [0146] 12. Tsamardinos I, Aliferis C F: Towards principled feature selection: relevancy, filters and wrappers. Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics (AI & Stats) 2003.
• [0147] 13. Tsamardinos I, Aliferis C F, Statnikov A: Time and sample efficient discovery of Markov blankets and direct causal relations. Proceedings of the Ninth International Conference on Knowledge Discovery and Data Mining (KDD) 2003:673-678.
• [0148] 14. Tsamardinos I, Aliferis C F, Statnikov A: Algorithms for large scale Markov blanket discovery. Proceedings of the Sixteenth International Florida Artificial Intelligence Research Society Conference (FLAIRS) 2003:376-381.
• [0149] 15. Neapolitan R E: Learning Bayesian networks. Upper Saddle River, N.J.: Pearson Prentice Hall; 2004.
• [0150] 16. Pearl J: Probabilistic reasoning in intelligent systems: networks of plausible inference. San Mateo, Calif.: Morgan Kaufmann Publishers; 1988.
• [0151] 17. Dougherty E, Brun M: On the number of close-to-optimal feature sets. Cancer Informatics 2006, 2:189-196.
• [0152] 18. Somorjai R L, Dolenko B, Baumgartner R: Class prediction and discovery using gene microarray and proteomics mass spectroscopy data: curses, caveats, cautions. Bioinformatics 2003, 19(12):1484-1491.
• [0153] 19. Statnikov A, Aliferis C F: Analysis and Computational Dissection of Molecular Signature Multiplicity. PLoS Computational Biology 2010, 6(5):e1000790.
• [0154] 20. Gopnik A, Schulz L: Causal learning: psychology, philosophy, and computation. Oxford: Oxford University Press; 2007.
• [0155] 21. Lemeire J: Learning Causal Models of Multivariate Systems and the Value of it for the Performance Modeling of Computer Programs. Ph.D. Thesis, Vrije Universiteit Brussel, Faculty of Engineering, Department of Electronics and Informatics; 2007.
• [0156] 22. Liu H, Liu L, Zhang H: Ensemble gene selection for cancer classification. Pattern Recognition 2010, 43(8):2763-2772.
• [0157] 23. Liu H, Liu L, Zhang H: Ensemble gene selection by grouping for microarray data classification. J Biomed Inform 2010, 43(1):81-87.
• [0158] 24. Ein-Dor L, Kela I, Getz G, Givol D, Domany E: Outcome signature genes in breast cancer: is there a unique set? Bioinformatics 2005, 21(2):171-178.
• [0159] 25. Michiels S, Koscielny S, Hill C: Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet 2005, 365(9458):488-492.
• [0160] 26. Roepman P, Kemmeren P, Wessels L F, Slootweg P J, Holstege F C: Multiple robust signatures for detecting lymph node metastasis in head and neck cancer. Cancer Res 2006, 66(4):2361-2366.
• [0161] 27. Natsoulis G, El G L, Lanckriet G R, Tolley A M, Leroy F, Dunlea S, Eynon B P, Pearson C I, Tugendreich S, Jarnagin K: Classification of a large microarray data set: algorithm comparison and analysis of drug signatures. Genome Res 2005, 15(5):724-736.
• [0162] 28. Vapnik V N: Statistical learning theory. New York: Wiley; 1998.
• [0163] 29. Cristianini N, Shawe-Taylor J: An introduction to support vector machines and other kernel-based learning methods. Cambridge: Cambridge University Press; 2000.
• [0164] 30. Scholkopf B, Burges C J C, Smola A J: Advances in kernel methods: support vector learning. Cambridge, Mass.: MIT Press; 1999.
• [0165] 31. Shawe-Taylor J, Cristianini N: Kernel methods for pattern analysis. Cambridge, UK: Cambridge University Press; 2004.
• [0166] 32. Fan R E, Chen P H, Lin C J: Working set selection using second order information for training support vector machines. Journal of Machine Learning Research 2005, 6(1889):1918.
• [0167] 33. Statnikov A, Aliferis C F: TIED: An Artificially Simulated Dataset with Multiple Markov Boundaries. Journal of Machine Learning Research Workshop and Conference Proceedings, Volume 6: Causality: Objectives and Assessment (NIPS 2008) 2010, 6:249-256.
• [0168] 34. Good P I: Permutation tests: a practical guide to resampling methods for testing hypotheses, vol. 2nd. New York: Springer; 2000.
• [0169] 35. DeLong E R, DeLong D M, Clarke-Pearson D L: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988, 44(3):837-845.
• [0170] 36. Mani S, Cooper G F: A Study in Causal Discovery from Population-Based Infant Birth and Death Records. Proceedings of the AMIA Annual Fall Symposium 1999, 319.
• [0171] 37. Joachims T: Learning to classify text using support vector machines. Boston: Kluwer Academic Publishers; 2002.
• [0172] 38. Aphinyanaphongs Y, Tsamardinos I, Statnikov A, Hardin D, Aliferis C F: Text categorization models for high-quality article retrieval in internal medicine. J Am Med Inform Assoc 2005, 12(2):207-216.
• [0173] 39. Rosenwald A, Wright G, Chan W C, Connors J M, Campo E, Fisher R I, Gascoyne R D, Muller-Hermelink H K, Smeland E B, Giltnane J M et al: The use of molecular profiling to predict survival after chemotherapy for diffuse large-B-cell lymphoma. N Engl J Med 2002, 346(25):1937-1947.
• [0174] 40. Guyon I, Gunn S, Nikravesh M, Zadeh L A: Feature extraction: foundations and applications. Berlin: Springer-Verlag; 2006.
• [0175] 41. Conrads T P, Fusaro V A, Ross S, Johann D, Rajapakse V, Hitt B A, Steinberg S M, Kohn E C, Fishman D A, Whitely G et al: High-resolution serum proteomic features for ovarian cancer detection. Endocr Relat Cancer 2004, 11(2):163-178.
• [0176] 42. Wang Y, Klijn J G, Zhang Y, Sieuwerts A M, Look M P, Yang F, Talantov D, Timmermans M, Meijer-van Gelder M E, Yu J et al: Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer. Lancet 2005, 365(9460):671-679.
• [0177] 43. Foster D P, Stine R A: Variable Selection in Data Mining: Building a Predictive Model for Bankruptcy. Journal of the American Statistical Association 2004, 99(466):303-314.