Learning Method for a Neural Network

1. Field of the Invention

The present invention is directed to a learning method for the neural modeling of dynamic processes.

2. Description of the Prior Art

Neural networks are being introduced into a large variety of technical fields. Neural networks prove especially suitable wherever it is important to derive decisions from complex technical relationships and from inadequate information. For forming one or more output quantities, one or more input quantities, for example, are supplied to the neural network. To this end, such a network is first trained for the specific application. Neural networks prove especially suitable for many applications since they are universal approximators.

A problem that often arises in conjunction with the use of neural networks, however, is that the input data for the training or during operation of the network are often not complete. This situation, as well as the fact that the measured values which are supplied to the neural network for constructing a time series are often imprecise or noise-infested, can cause degraded training results of the networks. In the case of recurrent neural networks in particular, the problem arises that not all of the inputs can be charged with training data, because some of the inputs are occupied with the outputs that are fed back during training. Heretofore, no methods have been available to take this specific problem into account.

An object of the present invention is to provide a learning method with which the learning process for training a recurrent neural network can be improved.

The above object is achieved in accordance with the principles of the present invention in a learning method for training a recurrent neural network having a plurality of inputs and a plurality of outputs and at least one connecting line between an output of the neural network and an input of the neural network, including the steps of separating the connection line during training of the neural network and using the input connected to the connecting line as a training input together with the other inputs, interpreting input quantities supplied to the neural network for training as a time series of a variable input quantity representing respective values of the input quantity at discrete points in time, generating respective input values for the thus-produced additional training inputs by treating the input values to be generated as missing values in the time series, identifying a statistical noise distribution of an uncorrelated noise of finite variance that has a chronological average value of zero and which is superimposed on the measured values, and, for each additional input value in the time series, generating an input value therefor by calculating a statistical missing value noise distribution dependent on the statistical noise distribution from at least one of the measured values neighboring the missing value in the time series, and calculating the value to replace the missing value using at least two Monte Carlo samples of the missing value obtained according to the missing value noise distribution.

A particular advantage of the inventive method lies in exploiting the fact that the missing or inadequate values or the noise-infested values that are to be supplied to the neural network are a component of a sequence of values in the time series. Advantageously, the known error distribution probability of the remaining values can be utilized in order to be able to calculate an anticipated error distribution, and thus the anticipated value for the missing value, according to the inventive method.

Missing values that neighb or one another in the time series can als o b e determined according to the i nv entive method. An iterative procedure is provided for this purpose that first calculates one value and subsequently determines the other neighboring value with the data acquired from the one value. This iteration procedure can also be multiply implemented so that an adequate precision of the values to be determined is assured.

In particular, a neural network that is intended to simulate the time series can also be trained according to the inventive method since the learning step width is thereby advantageously referred to the number of Monte Carlo samples that are taken.

A Gaussian distribution is assumed for the deviations of the predicted values of the time series or is predetermined since this is a distribution that corresponds to the greatest possible extent to values that are close to practice.

Simple numerical methods are utilized in the inventive method in order to be able to determine missing values, or in order to be able to predict a future value in the time series with missing values.

Simple mathematical methods are indicated according to the inventive method in order to be able to correspondingly edit measured values, and thus in order to be able to train the neural network.

Especially advantageously, embodiments of the method are disclosed in order to edit noise-infested measured values, or in order to edit measured values that contain a known and an unknown noise component, in order thus to be able to train the neural network in a simple and efficient way.

FIG. 1 shows a neural network that is being trained in accordance with the inventive method.

FIG. 2 schematically illustrates an example for training a neural network in accordance with t he principles of the inventive method.

FIG. 3 schematically illustrates a time series for use in explaining the inventive method.

FIG. 4 shows a time series and a system behavior in accordance with the inventive method.

FIG. 1 shows an example of a recurrent neural network NN. For example, this network NN is trained with a system behavior Sy that is shown in greater detail in FIG. 4. Free inputs 10 and 20 and free outputs 60 and 70 are provided at the network NM. Given training methods hitherto employed for such recurrent networks, only the inputs 10 and 20 are available for supplying training data.

Returns 30 and 40 are also provided at the network which realize a quasi memory at the recurrent network. A sectioning line 50 is also indicated in FIG. 1 along which the returns (lines) between the outputs and the inputs 30 and 40 can be separated according to the inventive method. Advantageously, the inventive method achieves the effect that the inputs 30 and 40--which were otherwise closed--are also available for the training process during training. As a result of employing a larger number of inputs when training the network, a faster training outcome is achieved. When editing the training data for these additional inputs, it is inventively assumed that the network is to be charged with a time series of chronologically successively occurring measured values of a technical system and should simulate this after the training. Since--according to the inventive idea--the two additional inputs 30 and 40 are likewise a component of the time series of measured values, the value can be calculated over the remaining measured values and their error distribution density. This occurs in the inventive method with the assumed error distribution density for the inputs 30 and 40 and by taking Monte Carlo samples according to this error distribution density and averaging the samples that are taken. Additional training input values thus arise according to the inventive method. More precise details of this calculating method with the inventive method are presented in the explanations of FIGS. 3 and 4.

The inventive approach is illustrated again with reference to an example in FIG. 2. The same recurrent neural network as in FIG. 1 is shown in FIG. 2. In this version of the inventive solution, it is assumed--even though this is not the case in reality--that the two inputs 30 and 40 arising by separating the line 50 are charged with values of a time series that can be calculated from the input values of the inputs 10 and 20. The input 30, for example, is thus charged with the value M and the error distribution .epsilon.1 from FIG. 4 that occupies the position Y.sub.t-2 in the time series. The input 40, for example, is then charged with the value L from FIG. 4 and the error distribution density .epsilon.2 thereof. This occupies the position Y.sub.t-3 in the time series. Z1 and Z2 reference time elements that correspond to the different delay of the individual values within the time series. In this case, Z1 would correspond to a time delay of Y.sub.t-1 through Y.sub.t-2 and Z2 would correspond in this case to a time delay between Y.sub.t-2 and Y.sub.t-3 at the line 40. In an embodiment of the inventive method, missing values of a time series can be simulated with high precision via the error distribution density thereof and by taking Monte Carlo samples with subsequent averaging. The basic approach of the invention utilizes this idea by charging the artificially generated inputs of recurrent neural networks with artificially generated values calculated by the invention that, however, are extremely precise. As a result, a better training performance is advantageously achieved that is reflected in precise weightings at the individual neurons. For implementing the inventive method, it is not necessary to separate all lines, i.e., returns of the neural network; on the contrary, only individual lines can be separated on a case-bycase basis since substantially less calculating outlay then need be exerted when obtaining the Monte Carlo samples and when calculating the value based on the error distribution density of the remaining time series. It can also be meaningful given more complex procedures to separate more return lines of the neural network than shown here in FIGS. 1 and 2 and to determine the input values required therefore in a complicated way. Advantageously, the inventive method includes an iteration method that also enables a calculation of a number of values on the basis of the inventive method. A person skilled in the art implementing the invention can decide on a case-by-case basis what method is most beneficial for the implementation of the invention. Such a person will thereby consider whether adequate calculating capacity is present in order to be able to exactly model the network, or whether it is more important to achieve a rapid success when training the neural network.

FIG. 3 shows a time series of measured values that, for example, can be supplied to a neural network for training in accordance with the invention. In chronological succession, these measured values, are acquired, for example, from a technical system and are referenced as Y.sub.t through Y.sub.t-6, according to their chronological succession. The illustrated arrows between the individual boxes symbolize the dependencies of the various values on one another. For example, it is assumed in FIG. 1 that the value Y.sub.t-2 is missing. The relevant values in the Markov blanket, as neighboring values of this missing measured value, are Y.sub.t-4, Y.sub.t-3, Y.sub.t-1 and Y.sub.t. Such a missing measured value in a time series can, for example, arise because the measuring instrument for registering the values did not function at the point in time in question or, in order to train the neural network better, it seems beneficial between individual measured values to supply this neural network with a further value that, consequently, is yet to be identified, i.e. that is still to be generated according to the inventive method.

FIG. 4 shows the time series from FIG. 3 in conjunction with a neural network NN. It may be seen that y represents a time-dependent variable that represents the system behavior SY of a technical system. As may be seen, the values Y.sub.t through Y.sub.t-6 correspond to measured values that are taken from the system behavior SY. The dashed arrows at the respective points in time symbolize that these measured values are to be supplied to the neural network NN during operation or during training.

As in FIG. 3 as well, the questionable measured value M for the point in time Y.sub.t-2 is not present. The probability density .epsilon. is indicated for this measured value M. For example, this probability density .epsilon. can be back-calculated according to the inventive method from a predetermined, known error distribution density of the remaining measured values. What is thereby particularly exploited is that the missing measured value must be located between two known measured values and the error thereof is thus also limited by the errors of the neighboring values and the errors of the remaining measured values of the time series. The underlying time series can be described as follows:

wherein the function is either "known to" the neural network such as being stored therein or stored in a memory accessible by the neural network, or is adequately modeled by a neural network. The contribution .epsilon..sub.t denotes an additive, uncorrelated error with the chronological average 0. This error--and this is essential for the inventive method--comprises a known or predetermined probability density P .sub..epsilon. (.epsilon.) and typically symbolizes the unmodeled dynamics of the time series. For example, a future value is to be predicted for such a time series that is to be completed according to the inventive method. It should be noted that future values are to be understood as being relative to the time position selected at the moment. This means that for a point in time Y.sub.t-5, the point in time Y.sub.t-4 constitutes its future value. Under these conditions, the conditional probability density can be described as follows for a value of the time series to be predicted:

As already mentioned, the error distribution density must be known. This distribution density can either be calculated on the basis of the system behavior and known, other external quantities, /or can be predetermined. A typical error distribution that occurs in practice is the Gaussian distribution. The conditional probability density can be described as follows with such an assumed Gaussian error distribution:

wherein G(x;c,o.sup.2) is the notation for a normal density that is determined at x and with a center C and a variance o.sup.2. When it is assumed that the system to be described is presented in the form of a sequence of values on a time axis, then the individual values of Y.sub.t can also be interpreted as random variables in a probabilistic network. The invention is thereby based on the problem of predicting a value of the time series with the existing information from the remaining values being employed as completely as possible. Based on the assumptions that were made above, the entire probability density of the time series can be described as follows: ##EQU1##

It is thereby assumed that Y.sub.t-k, with k.ltoreq.N, is the missing value. Assuming that Y.sup.u ={Y.sub.t-k } and Y.sup.m ={Y.sub.t-1, . . . ,Y.sub.t-k-N }.backslash.{Y.sub.t-k } are valid, the anticipated value that is to be predicted in the time series can be described as follows:

The following conditions thereby apply: M.sub.t-1 stands for all measurements up to point in time t-1. The above equation is the basic equation for the prediction with missing data. It should be particularly noted that the unknown Y.sub.t-k is dependent not only on the values of the time series before the point in time t-k but also is dependent on the measurements following t-k. The reason for this is that the variables in Y.sup.m .uparw. Y.sub.t form a minimum Markov blanket of Y.sub.t-k. This minimum Markov blanket is composed of the direct predecessors and the direct successors of a variable and of all direct predecessors of variables of the direct successor. In the example under consideration in FIG. 2, the direct successors are Y.sub.t . . . Y.sub.t-k+1. The direct predecessors are:

and the direct predecessors of the successor of the variables are:

It is known from the theoretical fundamentals that a variable is independent of another variable of this network when the variables within the Markov blanket are known. The required, conditional density is therefore determined from Equation (5) as follows:

The case of a missing measured value described here can also be expanded to a plurality of missing measured values lying side-by-side without limiting the invention. When this is the case, the one value must first be determined, for example, according to the inventive method on the basis of its neighbors and successors and predecessors and the further value is then determined with this first-determined value. This can continue back and forth until and adequate precision is achieved. The following is valid for this case:

For all missing values of the time series between the points in time t-1 and t-N, the following is also valid:

which represents the number of all measured values up to point in time t-1. Also valid

the right side in (5e) being obtained from Equation (4). In general, these integrals in the preceding equations for the function f cannot be analytically solved in case this is a nonlinear function. Details for the numerical solution by means of taking Monte Carlo samples are recited in conjunction with FIG. 3.

FIG. 3 shows two representations of a neural network NN1 and NN2. First, this can be interpreted such that NN1 is present at one point in time and NN2 is present at another point in time; however, it can also be two completely different neural networks. Two data lines are shown with which the neural networks can communicate; in case of an identity of the two networks, chronologically succeeding exchange events are meant by these lines. NN2 forwards data to NN1 via the data link 100. NN1 forwards data to NN2 via the data link 150. The individual values of the time series, as shown in FIG. 1 and in FIG. 2, are no longer shown here for simplicity. It should be noted, however, that the conditions for FIGS. 1 and 2 are still valid.

For example, the following relationships are valid when training the network with missing data: in case Y.sub.1,. . . Y.sub.t is to represent possible values of the time series, then Y.sup.m .OR right.{Y.sub.1,. . . Y.sub.t } indicates all measured values and Y.sup.u ={Y.sub.1,. . . Y.sub.t }.backslash.Y.sup.m indicate all known values. Let the neural network that models the function f be parameterized, for example, with a set of weightings w. Then the following is valid:

Without limitation of the invention, however, another known parameterizable function approximator can be employed. The logarithmic probability function, also referred to as the log likelihood function, then reads:

whereby the common probability density is then approximated as ##EQU2## and the following relationship is valid for the neural network for the calculation of the error distribution density:

For learning with the assistance of back-propagation or other gradient-based learning algorithms, the gradient of the logarithmized probability function is now also required, this derived as: ##EQU3## It should be noted that one proceeds on the basis of known initial conditions for Y.sub.1, . . . ,Y.sub.N. a Gaussian distribution for the error distribution is present, it follows therefrom that: ##EQU4## whereby Y.sup.ul =Y.sup.u .andgate.{y,. . . Y.sub.-N } represent the missing values for the inputs of the network and (8a) shows that the integral disappears when all Y.sub.l. . . Y.sub.l-N are known.

When the measured values are overlaid with an additional noise, the following relationships derive. Again valid, for example, is:

In this version of the invention, however, there is not to be any direct access to Y.sub.t. Instead, the time series

is to be measured. The contribution .delta..sub.t denotes independent noise with an average value 0. Under the condition that z={z.sub.1 . . . z.sub.t-1 } and Y={Y.sub.1 . . . Y.sub.t } are valid, the overall probability density derives as: ##EQU5## The calculating rule for the anticipated, next value of the time series can thus be recited

The gradient for the probability function for the training can likewise be calculated. When a Gaussian distribution of the noise with

is present, it follows that ##EQU6##

In one version of the inventive method, for example, values that are noise-infested or that cannot be exactly determined are supplied to the neural network. Due to the approximation of the weightings in the neural network, new values of the time series can be determined by the function f that is simulated by the neural network. These new values of the time series are subsequently supplied via the data link 150 to the further neural network NN2 which determines new values of the time series therefrom, again by simulation of the function f. This iterative procedure is continued until an adequate precision of the values to be determined has been achieved.

The following fundamentals form the basis for the exact determination of missing values using the Monte Carlo method. It should be noted that all solutions here have the form

whereby u denotes the set of unknown variables and m denotes the set of known variables. An integral having this form can, for example, be solved by taking random samples of the unknown variables are taken according to P(u.vertline.m). For example, these samples are referenced u.sup.1, . . . ,u.sup.s. The following relationship for the approximation follows therefrom: ##EQU7##

It should be noted that u in this equation corresponds to the value Y.sub.t-k, which is missing. With this inventive solution, thus, the problem is reduced to taking samples from P(u.vertline.m). When only one variable is missing, the problem is thus reduced to taking samples from a one-variable distribution that can be done with the assistance of the sampling-importance-resampling or with other sampling techniques, Bernardo, J.M., Smith, A.F.M. (1994) Bayesian Theory, Wiley & Sons, pp. 349-355.

The situation becomes somewhat more complicated when more than one measured value is missing. The reason is that the unknown variables are usually dependent on one another and that sampling must be undertaken from the distribution of all unknown variables. A general solution thereof is specified by the Gibbs sampling, as also described in the above textbook. In Gibbs sampling, the unknown variables are initialized with random values or--better--with estimated initial values that, for example, can be derived from the neighboring values of the missing values. Thereafter, one of the unknown variables u.sub.i is selected and a sample of P(u,.vertline.m,u.backslash.u) is taken; u is then set to this value. The procedure then is repeated for the next unknown variable, etc. Apart from, for example, the first samples, the samples are preferably taken with the corrected error distribution density. This, however, means that samples must be taken for all unknowns that ever occurred in the time series. In practice, however, the time window can be drawn, for example, from the samples and can be limited to a reasonable size. For example, this size can correspond to the size of the Markov blanket for the missing values. It should thereby be noted that the coupling of the unknowns fails in that case wherein successive values are known between two missing values N and that further values of this time series therefore need not be considered.

Taking samples for future values is especially simple according to the inventive method, however, it must be noted that it does not function like this for deterministic systems. The inventive procedure presents an especially simple solution for these situations that seem especially complicated. For the prediction of the values of the time series, the values obtained according to the corresponding distribution are substituted and the prognoses thereof are averaged, this yielding the missing value according to the inventive method. When training the network, for example, the average of the error gradients is formed by employing the values of the time series, that were determined with the samples, for the calculation thereof. For example, one can proceed as follows when taking the samples according to the Monte Carlo method.

For example, K steps in the future of the time series are to be predicted. In conjunction with the equations discussed above, this means that the values Y.sub.t-1, . . . ,Y.sub.t-K+1 are missing and that Y.sub.t is to be predicted under these conditions. The Monte Carlo method is extremely simple under these conditions. First, for example, a sample must be taken from the distribution P(Y.sub.t-K+1 .vertline.Y.sub.t-K, . . . ,Y.sub.t-K-N). This sample is referenced Y.sup.s.sub.t-K+1. A further sample Y.sub.t-K+2 is taken from the distribution P(Y.sub.t-K+2 .vertline.Y.sup.s.sub.t-K+1-N, , , Y.sub.t-1-N with this sample and the preceding measurements, etc., until each sample was generated for each unknown. When this procedure is repeated S times, then one obtains ##EQU8##

The experimental findings have shown that good training performances are achieved with the inventive method.

Although modifications and changes may be suggested by those skilled in the art, it is the intention of the inventors to embody within the patent warranted hereon all changes and modifications as reasonably and properly come within the scope of their contribution to the art.