Computer-implemented method and device for evaluating a dataset based on a range of uncertainty

[0001] This application claims the priority, under 35 U.S.C. § 119, of European patent application EP 23163429, filed Mar. 22, 2023; the prior application is herewith incorporated by reference in its entirety.

[0002] The invention relates to a computer-implemented method for evaluating a dataset, which relates to a physical process, in particular in the rail transportation technology sector. Furthermore, the invention relates to a data processing apparatus and a computer program product.

[0003] The invention can hence relate to the technical field of the evaluation of datasets, in particular with regard to rail transportation technology.

[0004] A plurality of technical functions and physical processes can be described by way of datasets. Artificial intelligence (AI) algorithms or machine learning (ML) algorithms (hereinafter, ML is considered a form of AI) are used more and more frequently to evaluate (but also to create) these datasets. Accordingly, such algorithms should also be considered for safety applications, for example in the rail transportation technology sector. However, especially in this sector, the safety requirements are particularly high for understandable reasons.

[0005] At present, it is still challenging to use an AI algorithm for safety applications, since known algorithms (substantially) do not achieve a corresponding safety certification. However, here it could be particularly advantageous to evaluate the datasets (or even the training datasets) so that meaningful evidence of accuracy or misclassification would be available.

[0006] However, conventionally, substantially only purely statistical approaches are available for this purpose. The correspondingly determined probabilities (and thresholds) for misclassification errors are currently too imprecise to be of practical use since they only converge with the rate √{square root over (n)} (the rate n would be desirable) (when n is the number of data points). However, these previous estimates do not utilize the properties of the respective problem, such as, for example, distribution shapes, or make any particular assumptions regarding the problem. One consequence of this would be to adapt or lower the safety standards for AI applications. This may not be desirable, however, with regard to safety risks.

[0007] The document “JENS BRABAND ET AL: “On Safety Assessment of Artificial Intelligence”, ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, Feb. 29, 2020 (2020-02-29), XP081611032” discusses how systems with artificial intelligence (AI) can be subjected to a safety assessment.

[0008] It is accordingly an object of the invention to provide a method for evaluating a dataset which overcomes a variety of disadvantages of the heretofore-known devices and methods of this general type and which provides for a method to evaluate a dataset in an efficient and reliable way.

[0009] With the above and other objects in view there is provided, in accordance with the invention, a computer-implemented method comprising:

• • [0010] performing a physical process and acquiring data from the physical process;
  • [0011] creating a dataset based on the physical process;
  • [0012] evaluating a probability of a misclassification of the dataset by the following steps:
  • [0013] providing the dataset, which includes:
    • [0014] a plurality of first data points, which can be assigned to a first label, wherein the first data points at least partially form at least one first local cluster area; and
    • [0015] a plurality of second data points, which can be assigned to a second label, wherein the second data points at least partially form at least one second local cluster area; and
    • [0016] wherein the first data points and the second data points are separable;
  • [0017] establishing a range of uncertainty between the first local cluster area of the first data points and the second local cluster area of the second data points; and
  • [0018] evaluating the dataset based on the range of uncertainty by determining a limit range of the range of uncertainty based on a stochastic distribution.

[0019] The above and other objects of the invention are achieved by the novel method, a data processing apparatus and a computer program product, as described below.

[0020] According to a first aspect of the invention, a (computer-implemented) method is described for the evaluation (assessment, checking, in particular the probability of misclassification) of a dataset. The method comprises:

• • [0021] i) providing the dataset, which relates to a technical/physical process or was generated means of a technical/physical process (for example, acquiring and/or simulating signals from rail transportation technology), and has:
  • [0022] a) a plurality of first data points, which can be assigned to a first label (or marking, characteristic, parameter) (for example, red traffic light signal), wherein the first data points (at least partially) form at least one first local cluster area, and
  • [0023] b) a plurality of second data points, which can be assigned to a second label (for example, green traffic light signal), wherein the second data points at least partially form at least one second local cluster area,
  • [0024] wherein the first data points and the second data points are separable (or do not overlap);
  • [0025] ii) creating a range of uncertainty (for example, an area in the two-dimensional problem space with a length L and a width w or, in the general m-dimensional case, a hypersurface with an (m−1) dimensional volume and a distance) at least between the first local cluster area of the first data points and the second local cluster area of the second data points; and
  • [0026] iii) evaluating the dataset based on the range of uncertainty created.

[0027] According to a second aspect of the invention, a method is described with the following steps:

• • [0028] i) performing a physical process (for example, acquiring signals),
  • [0029] ii) creating a dataset (in particular by means of an artificial intelligence (AI) algorithm based on the physical process, and
  • [0030] iii) evaluating the dataset according to the above-described method.

[0031] According to a third aspect of the invention, a data processing apparatus is described which comprises at least one processor and is configured to execute the above-described method.

[0032] According to a fourth aspect of the invention, a computer program product is described which comprises instructions which, when the program is executed by a computer, cause the computer to execute the above-described method.

[0033] In the context of this document, the term “dataset” can in particular relate to a plurality of data items (data points). The data items (for example, signals, time series, etc.) can also be multidimensional. In particular, these data items are related to a technical function or a physical process. In one example, the data items are acquired or measured as part of the physical process. For example, signals are acquired or an axle counter measures the number of axles passing through. In a further example, the dataset is simulated/modeled, but still relates to a technical function, for example the signals or axle progression are simulated. The dataset can be a measurement result or also, for example, verification data or training data.

[0034] According to one exemplary embodiment, the invention can be based on the idea that a dataset can be evaluated in an efficient and reliable manner if several assumptions (see below) hold true for the dataset (or the dataset is transformed such that the assumptions hold true) and the evaluation is performed based on a range of uncertainty between (local) cluster areas of data points of the same label.

[0035] The assumptions mentioned will be clarified below. Here, n is the size of the dataset (or the number of data points) and m is the dimension of the data items or the number of dimensions of the problem space of the dataset. In particular, the following assumptions are made for the dataset (or should be fulfilled (also by data transformation)):

[0036] Assumption A1: the n data points are separated in the m-dimensional space by (m−1) dimensional sets of hyperareas or hyperplanes (L). In a two-dimensional space, for example, the hyperarea would be a length, while the hyperarea in a three-dimensional space would be a surface, for example.

[0037] Assumption A2: the hyperarea and/or the limit area (w) (or surface area/surface volume) of the hyperarea (L) can be defined/measured and is finite (in particular no unusual edges such as Mandelbrot (fractals) etc.).

[0038] Assumption A3: the data points that can be assigned to a label have a tendency to cluster or form a local cluster area. In other words, a data point of another label (substantially) only occurs in the immediate neighborhood of the separating hyperareas (A3 is hence a consequence of A1).

[0039] Assumption A4: in the vicinity of the limit range of the hyperarea, the distribution of the data points is locally homogeneous (by the same continuous distribution F). When x approaches zero, the limit of F⁻¹(x) exists and is finite.

[0040] In contrast to known statistical methods, in the present case, the distribution of the data points in the environment (vicinity) of the limit range (of the hyperarea) is considered. This approach can provide a surprisingly efficient evaluation method, which, for example, provides a measure for the reliability of misclassification. In a preferred exemplary embodiment, this is achieved such that rate n is used to converge instead of rate √{square root over (n)} (see statistical method above). This is extremely surprising, but it can be justified by the assumptions (A1-A4), which differ from conventional statistical learning theory.

[0041] Furthermore, a particular advantage can be that the result is very general and that it could, for example, enable the number of training data items to be significantly reduced.

[0042] The following describes exemplary embodiments and implementations of the inventive concepts:

[0043] According to one exemplary embodiment, the evaluation comprises: estimating the probability that a data point is assigned to the wrong label. In other words, that one of the first data points is assigned to the second label and/or one of the second data points is assigned to the first label. In a practical example, for example, a red traffic light signal (as the first label) could be confused with or misclassified as a green traffic light signal (as the second label). A misclassification of this kind (for example, by an AI System) would obviously have significant consequences. Therefore, it can be particularly important to determine the probability of misclassification as accurately as possible.

[0044] It has surprisingly been shown that the described evaluation method can (at least partially) accomplish this more efficiently and reliably than established statistical methods. In other words, a measure for the probability of misclassification can be determined. Overall, the described evaluation method can be used to efficiently and reliably assess datasets (in particular from the rail transportation sector, where safety requirements play a prominent role), but also training datasets.

[0045] According to a further exemplary embodiment, the creation of the range of uncertainty furthermore comprises: determining (defining) of a finite hyperarea (L) of the range of uncertainty. Such a hyperarea exists and can be clearly determined (see assumptions A1 and A2 above) between cluster areas. In the present context, the term “hyperarea” is used to be independent of the number of dimensions of the respective problem space. The term hyperarea substantially refers to a known hyperplane, but is not intended to restrict dimensionality; this depends on the respective problem space of the dataset. While the hyperarea in a two-dimensional problem space would therefore be a length, in a three-dimensional problem space, it would be a surface (i.e., hyperplane or hypersurface), in a four-dimensional problem space, it would be a volume, etc.

[0046] In one example, a plurality of established methods can be used to determine the hyperarea, for example Delaunay triangulation, limit (value) search, maximum margin hyperplane, etc. This list of examples is not intended to be exhaustive. As a result, established and efficient methods can be implemented directly.

[0047] According to a further exemplary embodiment, the creation of the range of uncertainty furthermore comprises: determining or estimating a limit range (w) of the range of uncertainty (or the hyperarea). In particular, here, the assumption is made that the distribution of the data points in the limit area follows a fixed, but unknown, stochastic distribution. For example, a (upper) quantile of the minimum distance distribution can be estimated as a safety measure. It has surprisingly been shown that such an estimation (based on a stochastic distribution) can function particularly efficiently and reliably. Detailed exemplary embodiments relating to the estimation of the limit range can be found below with reference to FIGS. 2 to 5.

[0048] According to a further exemplary embodiment, the dataset originates from the rail transportation technology sector. In particular, the dataset relates to at least one of the following technical functions or physical processes: signals, in particular light signals (further in particular traffic lights), axle counters, point mechanisms. This can have the advantage that datasets relating to economically relevant technical functions, in particular from the railroad sector, can be evaluated/characterized by means of the described method. As already mentioned above, the safety factor is of particular relevance in this sector.

[0049] According to a further exemplary embodiment, the dataset comprises at least 100,000 data points, in particular at least 500,000 data points, further in particular at least 1,000,000 data points, further in particular at least 10,000,000 data points. It has surprisingly been shown that the reliability of the evaluation (misclassification) increases enormously with a particularly high n (number of data points).

[0050] This can be illustrated by the following example. First, a dataset comprising n=10,000 data points is considered. In statistical learning theory, the limit of the evaluation is in the order of magnitude of 0.01, but with the present method it is in the order of magnitude of 0.0001 (both results would have to be multiplied by an unknown constant in the range of 10-100). However, both can be unusable for safety applications (in the railroad sector). However, if n is now increased to 1,000,000 data points, the result according to statistical learning theory would be in the order of magnitude of 0.001, i.e., still too unreliable for safety applications. However, with the method described, the limit would be in the order of magnitude of 0.000001, as a result of which, this example would already be suitable for basic safety arguments.

[0051] According to a further exemplary embodiment, the problem space of the dataset is defined in two or more, three or more, dimensions (m). This can have the advantage that the separation of the data points can be improved. In particular, in some cases, this enables the assumptions to be fulfilled particularly well or a high degree or separability of the data points can be achieved. The presence of local cluster areas can also be improved by this measure.

[0052] According to a further exemplary embodiment, the method comprises: reducing the applied dimensions. This can have the advantage that resources (in particular computing power) can be saved. The dimensions of the problem space can be reduced directly by means of established methods, for example principal component analysis. In other words, the dimensionality of the problem (the dimensions of the problem space) is reduced, for example by feature extraction in order to only consider the most important features of the data items.

[0053] According to a further exemplary embodiment the method comprises: increasing the applied dimensions. As already explained above, this can improve separability or clustering. This can also be implemented directly by means of established methods for transforming the data (points).

[0054] Even in sub-datasets, in some circumstances, it may not be easy to separate the data points. It can be advantageous to transform the data points, in particular in such a way that the number of dimensions is increased. This enables a more efficient separation of the data points to be achieved.

[0055] Transformations are known which also enable linear separation, wherein however, additional dimensions have to be introduced. This is known from statistical learning theory, for example, where it is referred to as “the kernel trick.”

[0056] According to a further exemplary embodiment, the method comprises: separating at least one sub-dataset (or one sub-space) from the dataset (or the problem space). In one exemplary embodiment, this is in particular performed in such a way that the distribution of the data points in the vicinity of the range of uncertainty is (substantially) locally homogeneous for the sub-dataset.

[0057] According to a further exemplary embodiment the method comprises: merging the evaluation of at least two sub-datasets. This enables a reliable overall result to be obtained at the end despite separation.

[0058] According to one exemplary embodiment, if at least A1 and A2 apply to the dataset (or are at least plausible), the following steps can be performed, for example:

• • [0059] if necessary, the problem space of the dataset can be separated into subsets (sub-datasets), to which A4 applies. The number of data points and the empirical distribution is determined for each separated sub-dataset.
  • [0060] for each sub-dataset (subset), the surface area (the surface volume) L of the limit of the hyperplane and the uncertainty width w can be calculated.
  • [0061] the sub-dataset information can be collected and summated to obtain the overall result for the entire problem space of the dataset.

[0062] According to a further exemplary embodiment, the evaluation comprises: establishing whether the dataset fulfils a predetermined safety criterion, in particular in the rail transportation technology sector. This can have the advantage that a dataset (in particular the probability of misclassification) can be evaluated efficiently and reliably. For this purpose, in particular in the rail transportation technology sector, it can be necessary for the measure of data reliability to be set so high that it cannot be met by conventional statistical methods (see above). However, surprisingly, this can be enabled with the evaluation method described here.

[0063] In one exemplary embodiment, a safety requirement of, for example, 10-5 is to be met. This would mean that training and validation results should not have a higher failure rate. Assuming an empirical result R_emp is in this order of magnitude. Then, in particular the upper limit R_emp+Δ can be of interest, so that the actual error rate R is below this threshold with a statistical confidence of at least p. Therefore, it may also be necessary to ensure that R_emp+Δ<s, so that it is known that the safety requirement is met with a confidence of p. The results for the length of the confidence intervals Δ can be considered for different approaches if it is assumed that the validation results are adequate.

[0064] In this context, the following table compares the described evaluation method with the established statistical approaches Chernoff, normal confidence and Vapnik. With the evaluation method described here, overall results are given because, unlike the case with the other approaches, Δ cannot be separated.

[0065]

		Normal-		Evaluation
n	Chernoff	confidence	Vapnik	method
100,000	0.0019	0.000025	0.029	0.00028
1,000,000	0.00059	0.000008	0.0092	0.000028
10,000,000	0.00019	0.0000025	0.0029	0.0000028
100,000,000	0.000059	0.0000008	0.00092	0.00000028
1,000,000,000	0.000019	0.00000025	0.00029	0.000000028

[0066] According to a further exemplary embodiment, the method comprises the use of an artificial intelligence (AI) algorithm. In the context of this document, the term “AI” can in particular refer to computer-based approaches for mimicking cognitive functions of a human mind, in particular learning and problem-solving. A plurality of different mathematical algorithms and computational models have been developed to implement AI functionalities, for example “machine learning”, “deep learning”, neural networks, genetic algorithms, kernel regression, etc. The main purpose of these can be seen as improving an existing algorithm by training it with training data so that a learning effect occurs, and the algorithm's problem-solving ability improves over time. This can be done with or without human intervention (for example, improvement).

[0067] According to a further exemplary embodiment, the dataset (evaluated, in particular considered to be reliable) can be used as training data for an AI algorithm. Training data can be of particular importance in order to enable an acceptable reliability of the algorithm to be trained, in particular an AI algorithm. According to a further exemplary embodiment, the training data is provided to an artificial intelligence (AI) algorithm.

[0068] It should be noted that embodiments of the invention have been described with reference to different types of subject matter. In particular, some embodiments have been described with reference to method claims, while other embodiments have been described with reference to apparatus claims. However, a person skilled in the art will understand from the foregoing and the following description that, unless otherwise indicated, in addition to any combination of features belonging to one type of subject matter, any combination of features relating to different types of subject matter is also deemed to be disclosed by this document. This in particular applies to features of the method claims and features of the apparatus claims.

[0069] The above-defined aspects and further aspects of the present invention result from the exemplary embodiments described below and will be explained with reference to the exemplary embodiments. The invention is described in more detail below with reference to embodiments to which, however, the invention is not restricted.

[0070] Other features which are considered as characteristic for the invention are set forth in the appended claims.

[0071] Although the invention is illustrated and described herein as embodied in evaluating a dataset based on a range of uncertainty, it is nevertheless not intended to be limited to the details shown, since various modifications and structural changes may be made therein without departing from the spirit of the invention and within the scope and range of equivalents of the claims.

[0072] The construction and method of operation of the invention, however, together with additional objects and advantages thereof will be best understood from the following description of specific embodiments when read in connection with the accompanying drawings.

[0081] The depictions in the drawings are schematic. Reference is made to the fact that, in different figures, similar or identical elements or features are provided with the same reference symbols or with reference symbols that only differ from the corresponding reference symbols in the first digit. In order to avoid unnecessary repetition, elements or features that have already been explained in relation to a previously described embodiment are not explained again at a later point of the description.

[0082] In addition, spatially relative terms such as “front” and “back,” “top” and “bottom,” “left” and “right” etc. are used to describe the relationship of one element to another element as depicted in the figures. Thus, the relative spatial terms can apply to orientations used that differ from the orientation shown in the figures. Obviously, these spatially relative terms only refer to a simplification of the description and the orientation shown in the figures and are not necessarily restrictive since an apparatus according to one embodiment of the invention may assume orientations other than those shown in the figures, in particular when in use.

[0083] FIG. 1. schematically shows a dataset 100 according to one exemplary embodiment of the invention. The dataset 100 was generated by means of a physical process that is associated with signals in rail transportation technology. A first label relates to a red signal, while a second label relates to a green signal. The dataset 100 has a plurality of first data points 101 assigned to the first label (here, black points) and a plurality of second data points 102 assigned to the second label (here, white points).

[0084] In other words, the dataset 100 comprises a number n of data points with simple labels (for example, red and green in the case of a problem with traffic light recognition). In this example, the problem space of the dataset 100 has two dimensions. It is assumed (or is plausible) that the first data points 101 and the second data points 102 are separable (assumption A1).

[0085] The dataset 100 is now to be evaluated in such a way that an error limit for the probability of misclassification is found. Such a misclassification would, for example, occur if a red point were classified as a green point.

[0086] It is furthermore assumed (or is plausible) that a limit line between the two sets (first and second data points) is sufficiently smooth, is measurable and has a finite length L. This excludes highly irregular shapes such as Mandelbrot sets (fractals) etc. (assumption A2).

[0087] Finally, it is assumed (or is plausible) that the data points 101, 102 cluster with the same label. In other words, there is a high probability that a data point with the same label will be located in the vicinity of a data point with a certain label. In one example, there is no data point with another label within a data points triangle with the same label. It is hence assumed that data points with the same label normally have neighbors with the same label (assumption A3). This is already implied by (A1) in theory, but is not always specified when recording the data. A3 implies that a sufficiently large and representative dataset is available.

[0088] FIG. 2 schematically shows a range of uncertainty U in the dataset 100 according to one exemplary embodiment of the invention. Assuming that the first and second data points are separated, there is no single optimal solution to the classification problem. However, there is a local area between cluster areas of first and second data points in which the classification is uncertain.

[0089] The example in FIG. 2 has a theoretical limit L which is known to exist (A1) and can be defined (A2). The aim is now to quantify this range of uncertainty U, for example to determine the area in the two-dimensional problem space. A corresponding hyperarea is then one-dimensional (a length). One approach would be to calculate the area, i.e., the length of the hyperarea L times the width w. According to A3, it is assumed that it is fairly unlikely that data points with different labels lie in the same cluster area. However, the distribution of the data points is uncertain.

[0090] FIG. 3 schematically shows a range of uncertainty U in a one-dimensional problem space of the dataset 100 according to one exemplary embodiment of the invention. Between a first cluster area of first data points 101 (left side) and a second cluster area of second data points 102 (right side), there is a limit 0. Of interest in this one-dimensional example is the range of uncertainty U between the limit 0 and the second cluster area.

[0091] It is assumed that the second data points 102 have a positive iid distribution (iid: independent and identically distributed). Hence, the data points are independently and identically distributed as random variables (each random variable has the same probability distribution). Furthermore, it is assumed that the minimum of the random variables is an appropriate measure for w/2.

[0092] Exponential distribution: for exponential iid random variables with parameter λ, the minimum is exponentially distributed with parameter nλ, so that the mean value is 1/(nλ). A one-side percentile for a significance level of p can hence be determined as (1):

[0093] $\begin{array}{cc}\frac{-\ln \left(1-p\right)}{n\lambda }& \left(1\right)\end{array}$

[0094] Hence, (1) quickly approaches zero at the rate n (instead of √{square root over (n)}). The other parameters are constant.

[0095] Uniform distribution: for comparison, it can be assumed that the distribution of the data points is uniform for a specific interval. It can then be concluded that the minimum (1,n) is distributed and the mean value is 1/(n+1). The following percentile can be derived from this (2):

[0096] $\begin{array}{cc}1-\sqrt[n]{\left(1-p\right)}& \left(2\right)\end{array}$

[0097] Although (2) differs significantly from (1), after a Taylor expansion a similar result can be obtained for small values of (1−p) as with (1).

[0098] General distributions: in general, the distribution of the minimum is not known, but it is known from extreme value theory that the limiting distribution for limited minimum problems is a Weibull distribution. This distribution depends on two parameters, wherein λ is a scaling and k is a shape parameter, for example k=1 gives an exponential distribution:

[0099] $F\left(t\right)=1-e^{-{\left(\lambda t\right)}^k}$

[0100] The percentiles of a Weibull distribution are given by:

[0101] $\frac{\sqrt[k]{-\ln \left(1-p\right)}}{\lambda }$

[0102] This appears similar to (1), but further assumptions regarding the shape of the distribution may be necessary.

[0103] However, since even the moments for the minimum of the N(0,1)-distribution do not have a closed shape solution, it can be difficult to obtain at least an approximate result for the expected minimum or the variance from the minimum of a general distribution.

[0104] However, the so-called probability integral transform can be helpful. It is known that any continuous probability distribution F can be generated from the uniform distribution by X=F⁻¹(U). This is, for example, used to generate general random numbers in the Monte-Carlo simulation. It is known that this relationship applies to the order statistics and in particular to the minimum X₍₁₎=F⁻¹(U₍₁₎). Therefore, this can be applied if sets are to be calculated for a general distribution. For the uniform distribution, the exact quantile from (2) and a useful approximation of (1) are known.

[0105] The uniform percentile is denoted by q_p. The related percentile for the general case is then given as F⁻¹(q_p). However, since q_p tends toward zero as n increases, the behavior F⁻¹ near zero is particularly interesting. For a continuous positive distribution, a series expansion can be assumed: F⁻¹(x)≈ax. If the distribution F covers the vicinity of zero, there should be no constant term. Then a is only given by the derivative (F⁻¹)′(0), which is equal to 1/F′(0)=1/f(0) if F has the probability density f. Finally, the following can be derived (3):

[0106] $\begin{array}{cc}{F}^{-1}\left(q_P\right)=\frac{q_p}{f\left(0\right)}& \left(3\right)\end{array}$

[0107] Equation (3), however, means that even for general distributions the percentile is similar to (1), or that (1) already has the general form when λ is replaced by f(0). A comparison shows that (3) is meaningful, since f(0)=0 for an exponential distribution would mean that the expected value would be infinite. For the actual uniform distribution, f(0)=1 and so (3) is equal to (2).

[0108] In general, f(0)=0 only means that the probability mass near zero is small and that the series expansion is not justified so that the approximation cannot be used, but the inverse of F has to be determined directly. However, if more is known about the behavior of F or f in the vicinity of zero is known, it is perhaps possible to derive a more meaningful result for distributions with f(0)=0, either by a series expansion of F or by a more detailed examination of limit properties, for example by l′Hospital's rule.

[0109] But, for all distributions with f(0)≠0, (3) and (1) demonstrated that the rate of convergence is in the range 1/n instead of 1/√{square root over (n)} (with statistical approaches). This in particular means that a significant improvement has occurred under the above assumptions. For example, much less training data is now required.

[0110] FIG. 4 schematically shows a range of uncertainty U in a two-dimensional problem space of the dataset 100 according to one exemplary embodiment of the invention. Compared to the example in FIG. 3, therefore, a further dimension is added, wherein once again only a range of uncertainty next to the limit 0 is examined. Assuming that the linear segment has a length L and w denotes an approximate percentile of the limiting distribution, Lw can be used as a local estimation of the range of uncertainty.

[0111] To extend the local estimation to the full problem, as depicted in FIG. 1, more assumptions may be necessary. First, it can be assumed that locally all limiting distributions are similar, not necessarily the same (A4). Then, as in the example in FIG. 2, it can be assumed that the area of the range of uncertainty is approximately Lw, wherein L is the length of the limiting curve. It can be questionable whether the range of uncertainty will be underestimated if all n data points are taken into account and the minimum of all data points is used.

[0112] FIG. 5 schematically shows a plurality of range of uncertainties, which in each case delimit a first cluster area 110 with first data points from an area 120 of second data points (with respect to the dataset 100 in FIG. 1). It can be assumed that the data points are identically and independently distributed and A4 is fulfilled. A1 to A3 also appear to be plausible so that the limiting curve can be estimated.

[0113] As in FIGS. 3 and 4, here once again only one side of the limiting curve can be of interest. One way to estimate the hyperarea (here the length L) can, for example be, by means of Delaunay triangulation (in particular automatically). Here, the data points are connected to form triangles, wherein two results are possible: all three corners of a triangle are assigned to the same label or one corner is assigned to a different label than the other two. Triangles with the same label are classified correctly, whereas the others are close to the limit. Based on the latter, L can be estimated (e.g. summation of the longest sides of these triangles).

[0114] FIG. 6 shows a dataset 100 with complex cluster areas 110, 120, which may be challenging to separate, according to one exemplary embodiment of the invention. Compared to the dataset in FIG. 1, the dataset in FIG. 6 shows higher complexity and there is no homogeneous overall distribution.

[0115] Although it can be assumed that the distribution is normal, in reality it can be much more complex because there is actually a mixture of normal distributions that are truncated at the limit. Therefore, assumption A4 would no longer hold true.

[0116] In order to tackle this higher complexity, the problem space can be partitioned. FIG. 7 shows a detailed view of FIG. 6, wherein a regular division into 16 subproblems is shown. This is by way of example, other types of partitioning, for example triangles, could be used instead.

[0117] It is known how many data points n_i,j there are in each partition, so that it is possible to estimate the probability of a new data point falling into a partition:

[0118] p_i,j=n_i,j/n. Accordingly, the overall probability (overall result) m of a misclassification can first be determined for each partition, for example:

[0119] $m={\sum }_{i,j}{p}_{i,j}{m}_{i,j}$

[0120] It can be assumed that the data items are representative of the true distribution, and partitionings with only one label can be sorted out, for example 3,4 or 3,3 in FIG. 7. For some partitionings, a simpler approach can be selected, for example 1,2 or 4,2, wherein for example 2,4 or 4,4 can be more challenging. Now, either more data can be acquired or it can be assumed that m_i,j is 1 in this partitioning. If the number of data points in this partitioning is low, this can be a possibility. For example, n=10,000 first data points (label red) can be present in the dataset 100 in FIG. 6, but only one is localized in partitioning 4,4.

[0121] Assuming an equal limit distribution (in particular almost the same f(0) in all sub-datasets), the overall result can be simplified to (with uniform or equal distributions):

[0122] $m={\sum }_{i,j}{p}_{i,j}{m}_{i,j}\sim \frac{-\ln \left(1-p\right)}{f\left(0\right)n}{\sum }_{i,j}{L}_{i,j}=L\frac{-\ln \left(1-p\right)}{f\left(0\right)n}$

[0123] FIG. 8 schematically shows a reduction in the dimensionality and a separation of a sub-dataset according to one exemplary embodiment of the invention. The previous figures dealt with one-dimensional and two-dimensional problem spaces. However, the number of partitionings increases exponentially with the number of dimensions. Therefore, it can be advantageous to only consider the most important features and extract them, for example by means of principal component analysis. This can reduce the dimensionality.

[0124] In the example in FIG. 8, there are initially multiple dimensions (MD). These are reduced to three dimensions. Then, sub-datasets 131, 132 (or sub-spaces) are separated (or partitioned). Then, each sub-space (here 131) is considered separately, and m (see above) is determined. Finally, the overall result is evaluated by summarizing the results of the sub-spaces.

[0125] Reference is had to the fact that the term “comprising” does not exclude other elements or steps and the use of the article “a” does not exclude a plurality. Elements described in connection with different embodiments can also be combined. Reference is also made to the fact that reference symbols in the claims should not be construed as limiting the scope of the claims. Further, irrespective of the grammatical gender of specific terms, they also include individuals with male, female or other gender identities.