Partial stepwise regression for data mining

[0001] The present invention relates to the computer technology of the area of data mining. More particularly the presented invention relates to the data mining problem of predicting a given dependent variable using a multitude of independent variables.

[0002] Over the past two decades there has been a huge increase in the amount of data being stored in databases as well as the number of database applications in business and the scientific domain. This explosion in the amount of electronically stored data was accelerated by the success of the relational model for storing data and the development and maturing of data retrieval and manipulation technologies. While technology for storing the data developed fast enough to keep up with the demand, little attention was paid to developing software for analyzing the data until recently, when companies realized that hidden within these masses of data was a resource that was being ignored. The huge amounts of stored data contain knowledge about a number of aspects of their business waiting to be harnessed and used for more effective business decision support. Database Management Systems used to manage these data sets at present only allow the user to access information explicitly present in the databases i.e. the data. The data stored in the database is only a small part of the ‘iceberg of information’ available from it. Contained implicitly within this data is knowledge about a number of aspects of their business waiting to be harnessed and used for more effective business decision support. This extraction of knowledge from large data sets is called Data Mining or Knowledge Discovery in databases and is defined as the non-trivial extraction of implicit, previously unknown and potentially useful information from data. The obvious benefits of Data Mining have resulted in a lot of resources being directed towards its development.

[0003] Data mining involves the development of tools that analyze large databases to extract useful information from them. As an application of data mining, customer purchasing patterns may be derived from a large customer transaction database by analyzing its transaction records. Such purchasing habits can provide invaluable marketing information. For example, retailers can create more effective store displays and more effective inventory control than otherwise would be possible if they know consumer purchase patterns. As a further example, catalog companies can conduct more effective mass mailings if they know that, given that a consumer has purchased a first item, the same consumer can be expected, with some degree of probability, to purchase a particular second item within a particular time period after the first purchase.

[0004] An important problem within this area of technology is the problem of predicting a given dependent data mining variable using a multitude of independent data mining variables. Typically a multitude of records is available representing expressions of a particular, yet unknown functional relationship between the independent variables and a dependent data variable. The difficulty is to use said multitude of records as training set for deriving the unknown functional relationship between the independent and the dependent variables, which can serve as a prediction model. Then the prediction model can be used to determine for all possible values of the independent variable the value of the dependent variable. On one hand the prediction model should be able to reproduce the values of the training set as well as possible, on the other hand the prediction model allows one to determine and thus to predict the value of the independent variable for value combinations of the independent variables not comprised by the training set.

[0005] The most common prediction methodology in data mining is the multiple linear regression approach and the so-called multiple polynomial regression approach, with the former representing a special case of the latter. The first category uses one and the same continuous function for each independent variable based on the assumption of a linear relationship between the dependent variable and one or more independent variables. Multiple polynomial regression assumes a certain polynomial relationship between the dependent and the independent variables and thus can be viewed as an extension of the multiple linear regression methodology. Multiple polynomial regression is a regression methodology that fits and approximates a given dependent variable y with n independent variables X_i based on a fitting model that uses a polynomial of a certain predefined degree m common for all independent variables. Thus the assumed polynomial degree for each independent variable is identical.

[0006] A simplified multiple polynomial regression model according to the state of the art is of the following form: $\begin{array}{cc}y=A+\sum _{i=1}^{n}f\left(\mathrm{Xi}\right)+\mathrm{Error}& \left(\mathrm{eq}.1\right)\end{array}$
where f(X_i)=B_i1*X_i¹+B_i2*X_i²+ . . . +B_im*X_i^m, y is the dependent variable, X_i are the independent variables, A and B_i denote the unknown coefficients. So-called “linkage terms” of the independent variables (for instance the linkage terms for 2 independent variables and a polynomial degree of 3 are X₁* X₂, X₁²*X₂, X₁*X₂²) are not introduced for convenience.

[0007] Both of the above mentioned methodologies suffer from deficiencies relating to the same cause. As the actual polynomial degree of the functional relationship of the dependent and the independent variables is not known and both methods are based on a fixed assumed functional relationship—the multiple linear regression approach assuming a linear functional relationship and the multiple polynomial regression approach assuming a function relationship of a common predefined polynomial of degree m for all independent variables—their results are not satisfactory. Prediction models based on a multiple linear regression approach fail to predict non-linear functional relationships in an acceptable manner. Prediction models based on a multiple polynomial regression approach generally will generate unsatisfactory results if the assumed common polynomial degree m deviates from the actual functional relationship. In these cases prediction quality (of an actual functional relationship of polynomial degree K approximated by a polynomial of degree m) fits only well in a close neighborhood of the trained data areas. This deficiency even increases by increasing the degree m for the fitting polynomials.

[0008] Moreover, the multiple polynomial regression approach is characterized by poor computational efficiency. It always has to compute polynomials for the independent variables with an assumed maximum polynomial degree m even if the functional relationship is of simpler form (for instance linear); the computational process of determining said polynomials as well as the process of evaluating the polynomial suffers from this drawback.

[0009] The present invention teaches a computerized data mining method for determining a prediction model for a dependent data mining variable based on a multitude of n independent data mining variables.

[0010] The method comprises a variable replacement step replacing the independent data mining variable with potential values from a global range by a multitude of independent local data mining variables. Each independent local data mining variable relates to potential values from a sub-range of the global range.

[0011] The method further comprises an initialization step initializing a current prediction model and a looping sequence. The looping sequence comprises a first step of determining for every independent local data mining variable not yet reflected in said current prediction model a multitude of partial regression functions (depending only on one of the independent local data mining variables); and of determining for each of the partial regression functions a significance value; and of selecting the most significant partial regression function and the corresponding not yet reflected local data mining variable. The looping sequence comprises a second step of adding said most significant partial regression function to the current prediction model.

[0012] The proposed method achieves improved prediction quality within the training data areas without losing the stability for the untrained data areas. Moreover, the proposed teaching of the current invention is able to detect different functional dependencies of the dependent data mining variable within different sub-ranges of an independent data mining variable by introducing a multitude of local data mining variables. This capability is most important to cope with dependent variables occurring in practical applications which are very often characterized by varying functional dependencies within the individual sub-ranges of the independent variables.

[0016] In the drawings and specification there has been set forth a preferred embodiment of the invention and, although specific terms are used, the description thus given uses terminology in a generic and descriptive sense only and not for purposes of limitation. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0017] The present invention can be realized in hardware, software, or a combination of hardware and software. Any kind of computer system—or other apparatus adapted for carrying out the methods described herein—is suited. A typical combination of hardware and software could be a general purpose computer system with a computer program that, when being loaded and executed, controls the computer system such that it carries out the methods described herein. The present invention can also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which—when being loaded in a computer system—is able to carry out these methods.

[0018] Computer program means or a computer program in the present context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following a) conversion to another language, code or notation; b) reproduction in a different material form.

[0019] The current invention is using term “data mining” in its most general meaning generically referring to all types of mining technologies which sometimes are distinguished further in “data mining” and “text mining”.

[0020] A significant improvement has been made available by a commonly-assigned patent application, Stepwise Polynomial Regression, U.S. Pat. Ser. No. 09/608,468, which is incorporated herein by reference.

[0021] Stepwise Polynomial Regression uses a set of polynomials, but for each independent variable the ‘best’ fitting polynomial is selected. The ‘best’ fitting polynomial shows the least squared error of the predicted and the observed values from the dependent variable. Actually the best fitting polynomial is determined according to a significance measure. The regression formula of Stepwise Polynomial Regression is: $\begin{array}{cc}y=A+\sum _{i=1}^{n}f\left(\mathrm{Xi}\right)+\mathrm{Error}\mathrm{where}f\left(X_i\right)=\left\{\mathrm{select}\mathrm{one}\mathrm{of}\begin{array}{c}\begin{array}{c}\begin{array}{c}{B}_{\mathrm{i1}}*X_i^1\\ B_{\mathrm{i1}}*X_i^1+B_{\mathrm{i2}}*X_i^2\end{array}\\ \cdots \end{array}\\ B_{i1}*X_i^1+B_{i2}*X_i^2+\dots +B_{\mathrm{iM}}*X_i^M\end{array}\right\}& \left(\mathrm{eq}.2\right)\end{array}$
and where y is the dependent variable, X_i are the n independent variables, A and B_ij denote the unknown coefficients, M is maximum degree for the potential regression polynomials; from within this multitude of potential regression polynomial the most significant one is selected to be used as the actual prediction model.

[0022] In spite of all improvements provided by this technology, predictors determined according to stepwise polynomial regression suffer from the deficiency that a functional relationship can be described only by one of the provided continuous functions. In reality there are situations where an dependent variable has to be described by different functional dependencies in different areas of the overall range of the independent variables. The above mentioned teaching would be able to generate only a compromise solution as a prediction model. Moreover, noncontinuous relationships or functional relationships which are not part of the provided set of regression polynomials can only be approximated in a limited manner. In addition, if the maximum polynomial degree is increased in the Stepwise Polynomial Regression approach to increase the spectrum of potential functional dependencies, it can be observed that this decreases the stability of the prediction model for data which are not in the neighborhood of the trained data areas.

[0023] Thus it can be observed that even the current Stepwise Polynomial Regression methodology requires further improvements.

[0024] For an explanation of a further embodiment of the current invention relating to the specific technology of calculating a regression function, the focus of the discussion is put temporarily on FIG. 1. FIG. 1 is a flow chart depicting in more details how according to a preferred embodiment of the invention a regression function is calculated by an iteration process. FIG. 1 concentrates as a first approach on the calculation of specific regression functions, namely regression polynomials. In the next chapter an extension to general regression functions will be described.

[0025] The current invention proposes a new methodology for determining a prediction model for data mining variables which is called Partial Stepwise Polynomial Regression.

[0026] The invention proposes two major approaches. The partial stepwise polynomial regression using a fixed number of ranges or regions for each independent variable and a dynamic approach using variable number of ranges or regions for each independent variable. The fixed number approach does not result in region intervals with the same length. In contrast to that it is explicitly noted that the variable region number approach will result in variable region intervals and variable number of regions.

1.2.1 The Partial Stepwise Polynomial Regression Methodology with a Fixed Number of Regions

[0027] Partial Stepwise Polynomial Regression with a fixed number of regions is an enhanced regression method further improving the stepwise polynomial regression approach. Stepwise polynomial regression predicts the value of the dependent variable on the basis of n independent variables, where each independent variable is expressed by a single continuous polynomial with its individual polynomial degree over the complete range of each independent variable.

[0028] According to a fundamental observation leading to the current invention it is pointed out that in practical situations the functional dependency of a dependent variable on a set of independent variables very often is of a nature not allowing to approximate for prediction purposes that dependent variable with a prediction model which is identical throughout the whole range of the values of the independent variables. This situation is caused by a multitude of reasons. For instance, within a first sub-range of an independent variable the functional dependency to the dependent variable may be of a first type while in another sub-range the functional dependency may be of a second type. Also it might be possible that the functional dependency shows discontinuities. Due to these reasons a dependent data mining variable within typical applications cannot be approximated by a prediction model which is identical throughout the whole range of values of the independent variables. Therefore a methodology is suggested which is able to provide multiple partial regression functions for each independent variable. Instead of using an identical analytical function for the complete data area, that is the complete range, of an independent variable, the data area (range) is split up into regions or sub-ranges. Each region/sub-range is contributing its own continuous prediction function to the overall prediction model. This results in a prediction function for each independent variable which is determined by multiple different analytical functions. Each region/sub-range has it own prediction function to describe the relationship for the specific data area. Expressed in other words, an independent data-mining variable with potential values from a global-range is replaced by a multitude of independent local-data-mining-variables, each independent local-data-mining-variable with potential values from a sub-range of said global-range.

[0029] The above teaching may not be merely reduced to modeling a functional dependency on a global range of the independent variables by simply breaking it up into a multitude of prediction models dedicated to the individual sub-ranges of the global-range. As the individual sub-ranges of the independent variables are represented by independent local-data-mining-variables participating within the iteration process described below, each individual sub-range of each individual independent data mining variable with its associated prediction model is competing independently from one another for recognition within the selection process based on its significance to the overall prediction model. This results in an optimization approach going far beyond the normal stepwise polynomial regression methodology.

[0030] Thus Partial Stepwise Polynomial Regression suggests the use of multiple polynomials of potentially different degrees for each independent variable. Each polynomial describes a partial data area—called region or sub-range—of an independent variable. The invention proposes a technique allowing to determine individually a partial relationship for each region of an independent variable. The partial regression polynomials determined along these lines are combined to form the overall prediction model for the independent variable.

[0031] Assuming a maximum degree M for the potential regression polynomials and H regions for each independent variable, the Partial Stepwise Polynomial Regression method may be expressed with the following formula: $\begin{array}{cc}y=A+\sum _{i=1}^{n}f\left(\mathrm{Xi}\right)+\mathrm{Error}\mathrm{where}f\left(X_i\right)=\sum _{k=1}^{H}f\left(Z_{\mathrm{ik}}\right)\{\begin{array}{c}{Z}_{\mathrm{ik}}=\mathrm{Xi};{\mathrm{reg}}_{\mathrm{i0}}<Z_{i1}<{\mathrm{reg}}_{i1}\mathrm{else}{\mathrm{reg}}_{\mathrm{ik}-1}\le Z_{\mathrm{ik}}<{\mathrm{reg}}_{\mathrm{ik}}\\ Z_{\mathrm{ik}}=0\end{array}\}& \left(\mathrm{eq}.3\right)\end{array}$
and where y is the dependent variable, X_i are the independent variables, k is the index number of regions, Z_ik are the independent variables of specific regions where reg_ik determines the upper border of region k.

[0032] For each $f\left(Z_{\mathrm{ik}}\right)=\left\{\mathrm{select}\mathrm{one}\mathrm{of}\begin{array}{c}\begin{array}{c}\begin{array}{c}{B}_{ik1}*Z_{\mathrm{ik}}^1\\ B_{\mathrm{ik}1}*Z_{\mathrm{ik}}^1+B_{\mathrm{ik}2}*Z_{\mathrm{ik}}^2\end{array}\\ \cdots \end{array}\\ B_{\mathrm{ik}1}*Z_{\mathrm{ik}}^1+B_{\mathrm{ik}2}*Z_{\mathrm{ik}}^2+\dots +B_{\mathrm{ikM}}*Z_{\mathrm{ik}}^M\end{array}\right\}$
where A and B_ikj denote the unknown coefficients. Partial Stepwise Polynomial Regression determines region specific regression polynomial, i.e. the fitting curve (eq. 3), in such a manner, that for each region from all potential regression polynomials up to a maximum degree M a specific regression polynomial is selected. The combination of all region specific polynomials determines the regression function of an independent variable. The sum of all regression functions from the independent variables is the regression function for the dependent variable.

[0033] FIG. 1 visualizes a summarizing overview on the proposed Partial Stepwise Polynomial Regression methodology by a flow diagram.

[0034] Partial Stepwise Polynomial Regression starts by setting the maximum polynomial degree 101 M. This step limits the set of regression polynomials from which the method selects the most significant one.

[0035] Within step 102 the global-ranges for the independent data-mining-variables X_i are set.

[0036] In step 103 the number of regions/sub-ranges H are set and the H regions themselves are determined for each independent variable. This step can be viewed as a variable-replacement-step replacing the independent data-mining variable with potential values from a global-range by a multitude of new independent local-data-mining-variables, each independent local-data-mining-variable with potential values from a sub-range of said global-range. In “ex post” observation with the current invention it turned out that in addition to the number of regions H, the determination of the regions themselves causes the most impact to the prediction quality of the finally determined prediction model.

[0037] Within an initialization step 104 a current prediction model is initialized. In most cases the method starts with an empty prediction model, which means that none of the regions of the independent variables is represented by a regression polynomial in its functional relationship with the independent variable.

[0038] The steps 105 to 108 represent a looping sequence which can be repeated until a certain termination criterion is fulfilled. Within step 105 the method determines, if all independent candidate regions, i.e. the sub-ranges, have been reflected already in the current version of the prediction model. This can be achieved by just counting the number of already reflected regions/sub-ranges. Within step 106 for every region/sub-range not yet reflected in the prediction model a multitude of regression polynomials with different polynomial degrees is determined based on the set of training data. In the most far-reaching embodiment of the invention, for every not reflected region all regression polynomials according to eq. 3 of all degrees up to the maximum degree M are determined. Next, for each of said candidate polynomials their significance value is determined. The significance measures, based on the set of training data, the degree of improvement of the current prediction model, if a regression polynomial would be added to the prediction model. The significance is thus a measure of the appropriateness of a regression polynomial to reflect the functional relationship with the dependent variable, i.e. showing the “closest” relationship with the dependent variable. Within step 107 the most significant potential regression polynomial according to this significance measure and its corresponding region is then selected and added to the current prediction model, thus reflecting said region of an independent variable within the prediction model.

[0039] Within step 108 the method checks the fulfillment of a termination criterion. According to a basic implementation of the invention the termination criterion is a test whether all regions of all independent variables have been included in the prediction model by contributing a regression polynomial. The final prediction model at the point in time when the method terminated represents the prediction model as determined by the invention.

[0040] It is important to recognize that the suggested methodology is able to remove the constraint that a regression polynomial of an independent variable must have a higher polynomial degree to describe more complex relationship. Moreover the invention even allows one to describe non-polynomial relationships, for instance the Gaussian relationship.

1.2.2 The Region Determination

[0041] The region borders of the i'th independent variable are defined as $\begin{array}{cc}{\mathrm{reg}}_{\mathrm{ik}}=\left\{\begin{array}{cc}-\infty & k<1\\ C_i-{\mathrm{dI}}_i/2+\frac{{\mathrm{dI}}_i}{H}*k& k\ge 1,k<H\\ \infty & k=H\end{array}\right\}& \left(\mathrm{eq}.4\right)\end{array}$
where H defines the maximum number of regions, dI_i determines the definition interval for the region borders, i.e. the global range, and C_i is a specific center of the i'th independent variable. Various methods to define such a center are possible; one most successful, concrete definition for the center is given below.

[0042] A specific implementation of a region determination according to the current invention is based on the following features.

[0043] The most significant measures describing distributions of data are the mean value and the standard deviation.

[0044] The mean value is defined as $\begin{array}{cc}{Z}_{\mathrm{Mean}}=\frac{\sum _{i=1}^n{Z}_i}{n}& \left(\mathrm{eq}.5\right)\end{array}$
the number of observations is n and the standard deviation is the square root of the empirical variance s=√{square root over (S)} with the empirical variance $\begin{array}{cc}S=\frac{\sum _{i=1}^n{\left(Z_i-Z_{\mathrm{Mean}}\right)}^2}{\left(n-1\right)}& \left(\mathrm{eq}.6\right)\end{array}$

[0045] Assuming the mean value of the i'th independent variable as center C_i the definition interval dI_i for the region borders as four times of its standard deviation s_i this results in a specific region determination of $\begin{array}{cc}{\mathrm{reg}}_{\mathrm{ik}}=\left\{\begin{array}{cc}-\infty & k<1\\ Z_{i\mathrm{Mean}}-2s_i+\frac{4s_i}{H}*k& k\ge 1,k<H\\ \infty & k=H\end{array}\right\}& \left(\mathrm{eq}.4.1\right)\end{array}$
1.2.3 The Significance Measure

[0046] As a first observation the significance measure of a regression polynomial for a region of an independent variable is reduced to the simpler problem of a significance measure of the individual powers of a regression polynomial.

[0047] According to a preferred implementation the significance of a polynomial is >=the minimum significance of all its linear predictors:
B_ik1* Z_ik¹,B_ik2*Z_ik², . . . ,B_ikM*Z  (eq. 7)

[0048] Thus the significance of a regression polynomial is determined by the smallest significance of any of its powers.

[0049] Starting from this definition a significance measure for the linear predictors is required. For the linear predictors, the invention suggests to exploit the F-test to test whether a predictor influences the dependent variables or not. The F-test is a statistical test, well-known in the statistics, that checks whether two estimates of the variance of two independent samples are the same. In addition, the F-test checks whether the so-called NULL hypothesis is true or false. In application to the current situation assuming the inverse hypothesis, that “a predictor has no influence on the dependent variable”, this leads to the following NULL hypothesis for the F-test: “a coefficient B_ikj in a linear regression model (with respect to the various B_ikj) is zero”.

[0050] For a single linear predictor the test statistic is $F=\left(B_{\mathrm{ikj}}^2\right)/\left(S_{Z_{\mathrm{ik}}Y}^2*S_{Z_{\mathrm{ik}}}^2\right)=t^2$
with a degree of freedom of (n−2) where S_ZikY is the empirical covariance between Z_ik (a region of an independent variable) and Y (the dependent variable) and S_Zik is the empirical variance of the region variable Z_ik. In this special case the T-statistic t is equal to the root of the F-statistic. This remark indicates that also other statistical measures (like the T-Test) could be used as foundation for the significance measure approach of the current invention.

[0051] From the perspective of a practical computation, the F-Test value of a certain regression polynomial the F-Test value is determined on the basis of tentatively adding the regression polynomial to the current prediction model and on the basis on the training data set.

[0052] Based on this calculated F-test value the probability of obtaining a larger F-test value (Probability>F) can be determined according to the statistic theory of the F-test. If this probability tends to zero there is a statistical evidence for rejecting the NULL hypothesis. Or in other words: the more the F-test value approaches the value of 1, the larger is the support that the NULL hypothesis is true, indicating a small significance of the corresponding linear predictor (power of the regression polynomial); vice versa: the more the F-test value approaches the value of 0, the larger is the support that the NULL hypothesis is false, indicating a large significance of the corresponding linear predictor.

[0053] Thus the invention proposes as significance measure of a linear predictor to use the (Probability>F) value based on the F-test theory.

1.2.4 Partial Stepwise Polynomial Regression Conditionally Adding Regions (Sub-Ranges) of Independent Variables

[0054] Partial Stepwise Polynomial Regression allows for several optimization strategies of the methodology depending on the particular objective. The proposed improvements are targeted at reducing the number regions (sub-ranges) from the independent variables which contribute to the prediction model. Stated in other terms, the improvements of the method will reflect not all of the possible regions (sub-ranges) within the prediction model and will limit the number of regions to those which contribute to a “larger degree” to the functional relationship with the dependent variable. The coefficients of the eliminated regions are zero.

[0055] A first improvement of Partial Stepwise Polynomial Regression will add regions to the set of regions reflected in the prediction model conditionally only. This first improvement is exploiting the so-called adjusted R square measure also called adjusted correlation coefficient. The adjusted R square measure is well known within the state of the art.

[0056] This first improvement results in an enhanced step 107 of FIG. 1. Instead of unconditionally adding the most significant regression polynomial to the prediction model, it is first determined, if its inclusion would improve the adjusted correlation coefficient of the resulting prediction-model with respect to the set of training data. Only in the affirmative case that regression polynomial and the corresponding region is added to the prediction model. Otherwise the corresponding region is excluded from said method without further participation in the iteration process.

[0057] More particularly, if step 106 indicates the most significant regression polynomial and its corresponding region and if this region is the j-th region to be added to the prediction model, the selection criteria for actual adding this region to the prediction model is:
R′_j²>R′_j−1²  (eq. 8)
where the adjusted R square coefficient R′ for linear predictors is:
R′_p²=1−(q−1)/(q−p−1)*(1−R²)  (eq. 9)
where R² is the squared correlation coefficient with respect to the fitted and observed values, q is the number of observations (i.e. the number of training records), p is the number of independent predictors comprised by the regression polynomials within the current prediction model. In other words, the number of independent predictors p is equal to the number of the unknown coefficients B_ikm.

[0058] The correlation coefficient R is calculated by dividing the covariance from the observed (i.e. according to the training data) and the predicted values with the variance from the observed values and the variance from the predicted values.

[0059] Therefore
R=S_yy′/(S_yS_y′)  (eq. 10)
where S_yy′ is the empirical covariance of Y and Y′ which is determined by $\begin{array}{cc}{S}_{{\mathrm{yy}}^{\prime }}=\left(\sum _{i=0}^q\left(Y_i-Y_{\mathrm{Mean}}\right)\left(Y_i^{\prime }-Y_{\mathrm{Mean}}^{\prime }\right)\right)/\left(q-1\right)& \left(\mathrm{eq}.11\right)\end{array}$
and where Y_i are the observed values of the dependent variables and Y′ are the predicted values.

[0060] As a result the Partial Stepwise Polynomial Regression with adjusted R square optimization eliminates all regions which do not improve the adjusted correlation coefficient, i.e. which do not improve the prediction model. As an important computational advantage this results in a prediction model requiring a smaller number of regions. Only those regions which improve the prediction quality will become part of the prediction model.

1.2.5 Partial Stepwise Polynomial Regression Conditionally Adding and Removing Regions

[0061] A second improvement of Partial Stepwise Polynomial Regression will likewise add regions conditionally only to the set of regions reflected in the prediction model. Moreover it also will remove regions from the prediction model again in case of certain conditions. Thus the second improvement is targeted to determine a prediction model with as few regions as possible.

[0062] This second improvement results in an enhanced step (106) of FIG. 1. Instead of unconditionally adding the most significant regression polynomial to the prediction model, it is first determined if the significance of the currently most significant regression polynomial is above a predefined threshold significance value. In the affirmative case, only said currently most significant polynomial is added to the prediction model. Additionally, this second improvement of the invention enhances the looping sequence reflected in FIG. 1 by a third step succeeding step 107. Within this new step it is determined if the significance of a certain regression polynomial (or a multitude of regression polynomials) comprised within the current prediction model is reduced after the last regression polynomial has been added to the prediction model. If this is the case, said certain regression polynomial together with its corresponding region is removed from the current prediction model. Though this region is no longer reflected in the prediction model, it may of course participate within the further iteration process; i.e. a removed region can be added again in one of the next steps of the iteration. Another alternative handling is to exclude a region, which once has been removed from the prediction model, from said method without further participation in the iteration process.

[0063] These steps (adding, removing of regions) are repeated until all regions whose significance is higher than the specified threshold significance value are added to the model. This algorithm is called stepwise regression with full forward (adding regions) and backward (removing regions) capabilities. Expressing the termination criterion in other words, the looping sequence is terminated if the significance of the currently most significant regression polynomial is below said threshold significance.

[0064] With respect to the comparison of significance values it has to be stressed that the significance of a variable is higher if its (Probability>F) value tends to zero. That means, a variable is added when its (Probability>F) value is lower than the given significance threshold.

[0065] As the result of this second improvement the suggested methodology provides the possibility to find a prediction model in terms of minimizing the number of required regions. This results in a minimization of independent variables.

[0066] Besides operating on data mining variables being numeric in nature, the proposed methodology can be enhanced to also handle so-called categorical variables. Categorical variables are variables which can have a value out of a set of discrete values. A categorical variable will be converted into a vector wherein the vector elements are treated like independent variables representing the potential values of the categorical variable. Thus a categorical variable is treated as a [0,1] vector of its potential values.

[0067] For instance the categorical variable X having n members, i.e. n vector elements [“member_—1”, “member_—2” . . . , “member_n”] is mapped to the independent variables (i.e.) predictors):

• • [0068] X(“member_—1”)=[0,1]
  • [0069] X(“member_—2”)=[0,1]
    • [0070] . . .
  • [0071] X(“member_n”)=[0,1]

[0072] In the following, a comparison of the results of linear regression, stepwise polynomial regression and partial stepwise polynomial regression (according to the current invention) applied to the same training data set is presented. Based on the training data set all regressions predict the independent (mining) variable “income/capital” from the three independent variables (the concrete nature of these variables is not important for the current explanation):

• • [0073] [AGRI, SERVICE, INDUS]

[0074] All independent variables are numeric. The results of each prediction model is shown in FIG. 2. FIG. 2 visualizes the fitting curve showing the residuals (that is, the deviation between the exact values of the independent variable and its value according to the prediction model) of the observed values versus the predicted values.

[0075] The prediction model generated according to the linear regression method resulted in a R² of 0.634596, where R² is the squared correlation coefficient between observed and fitted values, and the following prediction model: $\begin{array}{cc}\mathrm{income}/\mathrm{capital}=-4660.13211905839+40.9309547592514*\mathrm{AGRI}+59.239686679445*\mathrm{SERVICE}+59.8879962521713*\mathrm{INDUS}& \left(\mathrm{sample}1\right)\end{array}$

[0076] The prediction model generated according to the Stepwise Polynomial Regression methodology resulted in an R² of 0.796181 and the following prediction model: $\begin{array}{cc}\mathrm{income}/\mathrm{capital}=-11771.9127789895-3.28336940177977*{\mathrm{AGRI}}^2+0.0469395359133741*{\mathrm{AGRI}}^3+287.254225684245*\mathrm{INDUS}-7.66112617982519*{\mathrm{INDUS}}^2+0.057519013287087*{\mathrm{INDUS}}^3+1123.30302703306*\mathrm{SERVICE}-35.2179332880916*{\mathrm{SERVICE}}^2+0.346957993399851*{\mathrm{SERVICE}}^3& \left(\mathrm{sample}2\right)\end{array}$

[0077] The prediction model generated according to the Partial Stepwise Polynomial Regression methodology resulted in an R² of 0.997138 and the following prediction model: $\begin{array}{cc}\text{income/captial}=406.178558348123+26.337989486862*\left(\mathrm{AGRI}>=7.145\mathrm{and}\text{<}25.6\right)-45.450827692626591*{\left(\mathrm{AGRI}>=7.145\mathrm{and}\text{<}25.6\right)}^2+833.49621951468*\left(\mathrm{AGRI}>=25.6\mathrm{and}\text{<}44.0544\right)-45.450827692626591*{\left(\mathrm{AGRI}>=25.6\mathrm{and}\text{<}44.0544\right)}^2+0.600474152375959*{\left(\mathrm{AGRI}>=25.6\mathrm{and}\text{<}44.0544\right)}^3-20.5338136610743*\left(\mathrm{AGRI}>=44.054\right)+14.5607462031416*\left(\mathrm{INDUS}<31.580\right)+444.293086313939*\left(\mathrm{INDUS}>=43.35\mathrm{and}\text{<}55.319\right)-22.2991534042399*{\left(\mathrm{INDUS}>=43.35\mathrm{and}\text{<}55.319\right)}^2+0.263576382781448*{\left(\mathrm{INDUS}>=43.35\mathrm{and}\text{<}55.319\right)}^3+216.7300647299952*\left(\mathrm{SERVICE}<22.607\right)-9.0408896205051*{\left(\mathrm{SERVICE}<22.607\right)}^2-176.731141985718*\left(\mathrm{SERVICE}>=31.15\mathrm{and}\text{<}39.69\right)+8.6463067407576*{\left(\mathrm{SERVICE}>=31.15\mathrm{and}\text{<}39.69\right)}^2-0.0869526250292018*{\left(\mathrm{SERVICE}>=31.15\mathrm{and}\text{<}39.69\right)}^3-419.782430750505*\left(\mathrm{SERVICE}>=39.69\right)+21.5409362796845*{\left(\mathrm{SERVICE}>=39.69\right)}^2-0.26503551689648*{\left(\mathrm{SERVICE}>=39.69\right)}^3& \left(\mathrm{sample}3\right)\end{array}$

[0078] As can be seen from the comparison of the resulting prediction models visualized within FIG. 2, the Partial Stepwise Polynomial Regression approach has been able to detect for certain regions, i.e. sub-ranges, very precisely the different functional relationships between the independent data mining variables and the dependent data mining variable differing from sub-region to sub-region. One example in this respect is the variable “AGRI”; in this case the Partial Stepwise Polynomial Regression approach moreover denied a polynomial functional contribution for the region (AGRI>=44.054) in contradiction to the prediction model generated by the Stepwise Polynomial Regression approach.

[0079] Based on the training data set of the previous chapter with the problem of determining a prediction model for the dependent variable “income/captial” dependent on the 3 independent variables, the prediction models determined according to various variants of the Partial Stepwise Polynomial Regression methodology are compared in FIG. 3. FIG. 3 reflects the squared correlation coefficient “R square”, the “number regions”, i.e. the number of sub-ranges and their corresponding independent local data mining variables, selected as most appropriate for the model versus the following Partial Stepwise Polynomial Regression variants:

• • [0080] the standard Partial Stepwise Polynomial Regression method (“Standard”)
  • [0081] the Partial Stepwise Polynomial Regression conditionally adding independent variables based on the adjusted correlation coefficient (“Adjusted R square”)
  • [0082] the Partial Stepwise Polynomial Regression conditionally adding and removing independent variables based on the significance threshold of 0.3 (“Stepwise 0.3”)
  • [0083] the Stepwise Polynomial Regression conditionally adding and removing independent variables based on the significance threshold of 0.4 (“Stepwise 0.4”)

[0084] By referring to FIG. 3 the standard Partial Stepwise Polynomial Regression algorithm will predict the dependent data mining variable “income/captial” with a R²=0.997138. Using the adjusted R squared optimization results in a decrease of the prediction quality to R²=0.912805, but requiring only 9 regions (instead of 18). Finally Partial Stepwise Polynomial Regression conditionally adding and removing regions with a significance threshold of 0.3 results in a prediction quality of R²=0.998117 with 15 regions. By increasing the significance threshold from 0.3 to 0.4 (remember: a reduction of the numerical value corresponds in a lowering of the significance level of the threshold) the prediction quality slightly increases to R²=0.998752 but the number of regions also increases from 15 to 16.

[0085] Thus, the standard Partial Stepwise Polynomial Regression methodology may be further improved in terms of the prediction model quality by the two variants. Most remarkably is that both variants lead to a reduction of the number used regions and this leads to a reduction of the number of independent variables, but there is a major usage differentiation. While adjusted R squared optimization does not need any user knowledge, the conditionally adding and removing regions function requires a deeper knowledge of the prediction result quality.

[0086] Partial Stepwise Polynomial Regression with variable number of regions is an enhanced regression method improving the partial stepwise polynomial regression with fixed number of regions approach. Partial Stepwise Polynomial Regression with fixed number of regions predicts the value of the dependent variable on the basis on n independent variables. Each independent variable is expressed by a fixed number of polynomials, one polynomial per region. The number of regions is fixed and each of them has potentially different polynomial degrees. This further embodiment of the invention proposes a new technique in addition, allowing to determine the number and sizes of regions/sub-ranges for each independent variable individually.

[0087] Assuming a maximum degree M for the potential regression polynomials and H_Max as maximum number of regions (not a fixed number H as in the previous embodiment) for each independent variable; this also means, that the final number of sub-regions for the individual independent variables may be different from one another. Partial Stepwise Polynomial Regression with variable number of regions method may be expressed with the following formula: $\begin{array}{cc}y=A+\sum _{i=1}^{n}f\left(\mathrm{Xi}\right)+\mathrm{Error}\mathrm{where}f\left(X_i\right)=\sum _{k=1}^{H_i}f\left(Z_{\mathrm{ik}}\right)\left\{\begin{array}{c}{Z}_{\mathrm{ik}}=\mathrm{Xi};{\mathrm{req}}_{\mathrm{i0}}<Z_{\mathrm{i1}}<{\mathrm{reg}}_{\mathrm{i1}}\mathrm{else}{\mathrm{reg}}_{\mathrm{ik}-1}\le Z_{\mathrm{ik}}<{\mathrm{reg}}_{\mathrm{ik}}\\ Z_{\mathrm{ik}}=0\end{array}\right\}\mathrm{with}1\le H_i\le H_{\mathrm{Max}}& \left(\mathrm{eq}.12\right)\end{array}$
and where y is the dependent variable, X_i are the independent variables, k is the number of regions, H_i is the maximum number of regions for the i'th independent variable and Z_ik are the independent variables of specific regions where reg_ik determines the upper border of region k.

[0088] For each $f\left(Z_{\mathrm{ik}}\right)=\left\{\mathrm{select}\mathrm{one}\mathrm{of}\begin{array}{c}{B}_{\mathrm{ik1}}*Z_{\mathrm{ik}}^1\\ B_{\mathrm{ik1}}*Z_{\mathrm{ik}}^1+B_{\mathrm{ik2}}*Z_{\mathrm{ik}}^2\\ \dots \\ B_{\mathrm{ik1}}*Z_{\mathrm{ik}}^1+B_{\mathrm{ik2}}*Z_{\mathrm{ik}}^2+\dots +B_{\mathrm{ikM}}*Z_{\mathrm{ik}}^M\end{array}\right\}$
where A and B_ikj denote the unknown coefficients.

[0089] Partial Stepwise Polynomial Regression determines region specific regression polynomial, i.e. the fitting curve (eq. 3), in such a manner, that for each region from all potential regression polynomials up to a maximum degree M a specific regression polynomial is selected. The combination of all region specific polynomials determines the regression function of an independent variable. The sum of all regression functions from the independent variables is the regression function for the dependent variable.

1.6.1 Determination of Initial Regions/Sub-Ranges

[0090] The initial region borders of the i'th independent variable are defined as $\begin{array}{cc}{\mathrm{ireg}}_{\mathrm{ik}}=\left\{\begin{array}{cc}-\infty & k<1\\ C_i-{\mathrm{dI}}_i/2+\frac{{\mathrm{dI}}_i}{H}*k& k\ge 1,k<H\\ \infty & k=H\end{array}\right\}& \text{(eq. 13)}\end{array}$
where H defines the maximum number of regions, dI_i determines the definition interval for the initial region borders and C_i is a specific center of the i'th independent variable.

[0091] A specific implementation of an initial region determination according to the current invention is based on the following features.

[0092] The most significant measures describing distributions of data are the mean value and the standard deviation.

[0093] The mean value is defined as $\begin{array}{cc}{Z}_{\mathrm{Mean}}=\frac{\sum _{i=1}^n{Z}_i}{n}& \text{(eq. 14)}\end{array}$
the number of observations is n and the standard deviation is the square root of the empirical variance s=√{square root over (S)} with the empirical variance: $\begin{array}{cc}S=\frac{\sum _{i=1}^n{\left(Z_i-Z_{\mathrm{Mean}}\right)}^2}{\left(n-1\right)}& \text{(eq. 15)}\end{array}$

[0094] Assuming the mean value of the i'th independent variable as center C_i the definition interval dI_i for the region borders as four times of its standard deviation s_i this results in a specific initial region determination of $\begin{array}{cc}{\mathrm{ireg}}_{\mathrm{ik}}=\left\{\begin{array}{cc}-\infty & k<1\\ Z_{i\mathrm{Mean}}-2s_i+\frac{4s_i}{H}*k& k\ge 1,k<H\\ \infty & k=H\end{array}\right\}& \text{(eq.13.1)}\end{array}$
1.6.2 Determination of the Final Regions/Sub-Regions

[0095] The final region borders of the i'th independent variable are defined as $\begin{array}{cc}{\mathrm{reg}}_{\mathrm{ik}}=\mathrm{select}\mathrm{one}\mathrm{of}\left\{\begin{array}{c}{\mathrm{ireg}}_{\mathrm{ij}}\\ {\mathrm{ireg}}_{i\left(j+1\right)}\\ \dots \\ {\mathrm{ireg}}_{{\mathrm{iH}}_{\max }}\end{array}\right\}\left\{\begin{array}{c}j=k;k=1\\ j=j^{\prime };{\mathrm{reg}}_{i\left(k-1\right)}={\mathrm{reg}}_{{\mathrm{ij}}^{\prime }};k>1\end{array}\right\}& \text{(eq. 15)}\end{array}$
that the interval reg_i(k−1), reg_ik contain at least Np disjoint observations of the independent variable X_i.

[0096] In verbal representation eq. 15 defines a method wherein the sub-ranges (regions) and the corresponding local-data-mining-variables Z_ik are of variable size determined by an iterative procedure comprising the following steps:

• • [0097] a. an initial step of dividing the global-range into maximum number H of equidistant sub-ranges (regions),
  • [0098] b. an iteration step of selecting a certain sub-range for which the number of the training data falling into a certain sub-range is below a third threshold Np and joining said certain sub-range with a neighbor sub-range forming a larger sub-range, and
  • [0099] c. a termination step of terminating the iteration step if for each sub-range the number of the training data falling into said each sub-range is equal to or above the third threshold.

[0100] The result of this iterative approach to determine the number and sizes of the sub-ranges results in a situation wherein a size of a sub-range is on one hand large enough to comprise enough training information for calculating a corresponding prediction model, and on the other hand the sizes of the sub-ranges are small enough to take into consideration different functional dependencies in different value ranges of the independent variables.

[0101] Further attention should be given to eq. 13.1 with respect to the specific treatment of the two sub-ranges between −∞ and the lower limit of the global-range as well as between the upper limit of the global-range and +∞. As can be seen from this equation, it is suggested that the local-data-mining-variables are augmented by the following sub-ranges and corresponding independent local-data-mining-variables:

• • [0102] 1. a local-data-mining-variable representing a sub-range from −∞ up to the lower limit of the global-range; and/or
  • [0103] 2. a local-data-mining-variable representing a sub-range from the upper limit of the global-range up to +∞.

[0104] The advantage of such approach is that these specific sub-ranges characterized by comprising a limited number of training data only (it is remembered that due to the specific construction of the global-range based on the mean value and the standard deviation of the values of the independent variable as explained above most training data belong to the global-range) nevertheless can participate in a “lump-sum” treatment in the proposed determination procedures. Even though little knowledge on the functional dependency is available in these areas (due to the limited number of training data) the proposed teaching will provide stable prediction models in these sub-ranges.

1.6.3 Example of a Prediction Model Using Variable Number of Regions

[0105] In the following a comparison of the result of partial stepwise polynomial regression using a fixed number of regions and of the result of using a variable number of regions is presented. The regression model of the fixed number approach is shown in (sample 3) above.

[0106] Using the same training data for the variable number of region approach results in a R² of 0.9992, where R² is the squared correlation coefficient between observed and fitted values with the following prediction model: $\begin{array}{cc}\text{income/capital}=686734.967+315.724*\left(\mathrm{AGRI}<25.6\right)+10.062*{\left(\mathrm{AGRI}<25.6\right)}^2-0.41441*{\left(\mathrm{AGRI}<25.6\right)}^3+53667.040*\left(\mathrm{AGRI}>=25.6\right)-3053.65*{\left(\mathrm{AGRI}>=25.6\right)}^2+39.87*{\left(\mathrm{AGRI}>=25.6\right)}^3-17381.39*\left(\mathrm{INDUS}<31.58\right)-16136.22*\left(\mathrm{INDUS}>=31.58\mathrm{and}\text{<}43.45\right)-42322.28*\left(\mathrm{INDUS}>=43.45\right)+850.87*{\left(\mathrm{INDUS}>=4.45\right)}^2-5.62*{\left(\mathrm{INDUS}>=43.45\right)}^3-1214745.52*\left(\mathrm{SERVICE}<22.60\right)+77755.95*{\left(\mathrm{SERVICE}<22.60\right)}^2-938.51*{\left(\mathrm{SERVICE}<22.60\right)}^3+620.29*\left(\mathrm{SERVICE}>=31.15\mathrm{and}\text{<}39.69\right)-45.80*{\left(\mathrm{SERVICE}>=31.15\mathrm{and}\text{<}39.69\right)}^2+0.82*{\left(\mathrm{SERVICE}>=31.15\mathrm{and}\text{<}39.69\right)}^3+0.04*{\left(\mathrm{SERVICE}>=39.69\right)}^3& \text{(sample 4)}\end{array}$

[0107] The comparison between (sample 3) and (sample 4) shows that the dynamic solution uses only two regions for the variable AGRI versus 3 regions from the fixed region approach. Any reduction in a number of independent variables (directly related to a reduction of the number of sub-ranges) means an improvement with respect to computational efficiency, that is processing time.

[0108] In a further embodiment of the current invention its teaching may be readily extended to the calculation of a prediction model based on a multitude of regression functions as a generalization of regression polynomials. The only difference is that within step 106 for every independent local-data-mining-variable (relating to a certain sub-range) not yet reflected in said current prediction-model a multitude of partial regression functions are calculated. A partial regression-function within this context means a regression function (not necessarily a polynomial) which depends on one of the local-data-mining-variables only. The rest of the steps of the above described methodology, like the step of determining a significance value for a partial regression function, the step of selecting the most significant partial regression function and the step of adding the most significant partial regression function to the prediction model can be performed in absolute correspondence to above, regression polynomial based teaching.

[0109] The current invention proposes a data mining method for determining a prediction model that allows one to determine the actual polynomial degree of the unknown functional relationship of the dependent and the independent variables. The method can be exploited therefore at same time to generate prediction models for linear and non-linear functional relationships. It is no longer necessary to assume a functional relationship of a common predefined polynomial degree m for all independent variables. As the proposed method is able to determine the actual functional relationship, the prediction quality is improved at the same time. This advantage will even increase for values of the independent variable residing not in the close neighborhood of the training data areas.

[0110] Being able to determine the actual functional relationship between the dependent and the independent variable is of special importance in the area of data mining as it is its imperative target to show a user the functional dependencies in analytical terms.

[0111] Prediction models determined according to the current invention moreover are characterized by increased computational efficiency in terms of processing time and memory space (compared to prediction models with a comparable number of independent variables) as it avoids the need to compute polynomials for the independent variables with an assumed maximum polynomial degree m even if the functional relationship is of simpler form (for instance linear).

[0112] Finally the Stepwise Polynomial Regression is a highly scalable, high performance methodology for the determination of prediction models. It clearly outperforms other methodologies currently viewed as providing the best state of the art scalability behavior.

[0113] Further embodiments of the invention allow to determine prediction models with reduced number of independent variables (compared to prediction models wherein the number of independent variables has to be predefined and is not self-adjusted as in the current case), which at the same time improve the quality of the prediction models, as only those independent variables will become part of the prediction model, which improve the prediction quality. This property is of an important computational advantage within data mining as the frequency a prediction model is (re-) computed is low compared to frequency a given prediction model is used to compute the corresponding dependent variable. Finally a reduced number of independent variables provides exploiters of data mining technology a deeper insight in the unknown fundamental functional relationship between the variables. Once this knowledge is available it can be utilized for the provision of new data to be analyzed by data mining techniques: the scope of mining data to be sampled can be limited to those data records comprising the most relevant independent variables only. This leads to cost reduction when applying data mining techniques. Moreover the feature of the invention allowing to determine “prediction models with a reduced number of independent variables” can be expressed in other words as the capability to automatically determine from a large set of independent variables that subset which actually influence the dependent variable. This capability addresses a fundamental data mining problem: the typical starting point for the application of data mining techniques are huge data assets with a lot of different variables for which nobody knows which of the variables are functionally influencing a certain dependent variable. Current state of the art technology requires to rely on human “intuition” or human knowledge for selecting the (hopefully) “correct” independent variables to be included in the prediction model. Furthermore, once a certain set of independent variables has been selected as basis for the prediction model, one never could be sure on the correctness of the selection; i.e. important variables could have been ignored by the human expert or independent variables could have been added with a minor influence on the dependent variable only, which unnecessarily increases the computational complexity of the prediction model. Thus the teaching on the other hand provides a method for automatically determining from the huge set of potential variables those with the most significant functional influence on the independent variable. The prediction quality of a prediction model according to the proposed technology is therefore improved in two directions: first by determining regression polynomials with a polynomial degree adapted to the actual functional dependency; second by determining those independent variables with the most significant influence on the dependent variable. For experienced exploiters of data mining technology the proposed teaching helps to reduce the effort to identify those portions of the huge data assets being of relevance for the prediction models to be computed. For other addressees, not having the knowledge on the functional dependencies of variables, the current invention may represent the enabling technology for exploitation of data mining techniques altogether.