Internal Connection Method for Neural Networks

The present invention relates to an internal connection method for neural networks, having particular application to prediction tasks or to observation vector dimension reduction in, for example, the image processing and analysis fields. More generally, it can be applied to techniques employing multi-layer neural network modelling structures.

Over the last few years, a large number of neural network models have been proposed. Among these, the model that seems to be of most interest for industrial applications is the multi-layer Perceptron which is particularly discussed in an article by D. E. Rumelhart, G. E. Hinton and R. J. Williams "Learning internal representations by error back propagation" in the book "Parallel Distributed Processing" Cambridge: MIT Press, 1986. This structure is an assembly of individual units called "neurons" or "automata". The neurons are organized into layers, each layer receiving the outputs from the preceding layer at its inputs. A learning algorithm known as a gradient back-propagation algorithm has been proposed for this type of network. This algorithm is discussed in the abovementioned article.

Even though the multi-layer Perceptron's performance is satisfactory as regards classification tasks, the same does not apply to tasks involving prediction or, notably, space dimensional reduction. The multi-layer Perceptron performs badly in these fields because of its poor ability to handle non-linear problems.

The aim of the invention is to overcome the above-cited disadvantages. The invention hence provides an internal connection method for neural networks linking successive neuron layers, each layer receiving the outputs from the layer of immediately lower rank at its inputs, the method consisting in representing the output states of the neurons by a function, said function being obtained by the steps comprising: carrying out a weighted summation of functions, each of said functions representing the output state of a neuron from the preceding layer, then, in a second step, applying a saturating function to the result of said summation, and, in a third step, applying a distribution function to the result after application of the saturating function.

The principal advantage of the invention is that it improves the ability of the multi-layer Perceptron to handle non-linear problems with a notable improvement in the Perceptron's prediction and dimension-reducing performance.

Further features and advantages of the invention will become more clear from the description that follows in conjunction with the attached drawings in which:

FIG. 1a shows the organisation of neurons in layers in accordance with the multi-layer Perceptron structure.

FIG. 1b illustrates a neuron connection model.

FIG. 2a shows a neuron connection model in accordance with the invention.

FIG. 2b shows examples of distributed scalar functions.

FIG. 2c shows a neural model with sampling in accordance with the invention.

FIG. 3 shows the segmentation of a distributed scalar function.

FIG. 4a shows results obtained by application of the method according to the invention.

FIG. 4b illustrates the structure of the neural network used to obtain the results shown in FIG. 4a.

FIG. 5 shows a flow chart of the method according to the invention.

FIG. 1a shows how neurons are organized in layers in accordance with the structure of the multi-layer Perceptron. An input vector 1 supplies information to input units symbolized by arrows. There are no internal connections within the layers.

Each neuron receives the outputs from the layer of immediately lower rank at its inputs. Generally, a so-called threshold neuron, having no inputs, whose output is always 1, and which is symbolized by a triangle, is provided on each layer, except at output vector 2 level, in order to improve the network's capacity by thus adding a degree of freedom.

FIG. 1b shows one way of connecting neurons in association with the multi-layer Perceptron's structure, and proposed by Rumelhart et al. It consists of a summing element 3 and a non-linear transition function 4, of the tangential hyperbolic type, for example. O.sub.i stands for the output state of a neuron i from the preceding layer, O.sub.j is the output state of a neuron j forming part of the next layer, the three arrows at the input to summing element 3 symbolizing the connections to the neurons of the preceding layer. W.sub.ij stands for the weighting coefficient associated with the connection between neuron i and neuron j. X.sub.j, the result of the weighted sum 3, is called the potential of neuron j. If we use F to stand for the transition function 4 of the neuron, and pred (j) stands for the set of neuron j's predecessors, then, using this notation, we can write: ##EQU1## the symbol .epsilon. indicating that i belongs to the set pred (j).

FIG. 2a shows a connection model in accordance with the invention. The neuron output states are no longer indicated by scalar quantities as before, but by functions. The connection method according to the invention consists, in a first step 21, of providing the weighted sum of the functions f.sub.xi representing neurons i from the preceding layer. If the function W.sub.ij stands for a weight continuum characterizing the connection between the neurons i and j, then, at the end of this first step 21, the potential X.sub.j associated with neuron j is given by the relation: ##EQU2## pred (j) having been defined previously.

In a second stage 22, the result X.sub.j of this weighted sum is mathematically treated by a saturating function F.sub.s, the result from which is given by x.sub.j, x.sub.j =F.sub.s (X.sub.j), F.sub.s being, for example, of the tangential hyperbolic type a third step 23, a distribution function d.sub.h is applied to the value x.sub.j in order to obtain the output function that characterizes the output state of neuron J, f.sub.xj standing for this function. The distribution function d.sub.h employed in the third step is a function that applies the set of real numbers R into all the applications of R within R, and associates with every real number x the result of the convolution product of the Dirac delta function centered on x, represented by .delta..sub.x, with an even and aperiodic function represented by h. Using the above notations, we have, for each real number x:

where .delta..sub.x.sup.* h stands for the convolution product of the function .delta..sub.x by the function h. In this case, for all real numbers x and y,

The distribution function d.sub.h is hence determined entirely by knowledge of the function h. Below, the function f.sub.x will be called the distributed scalar representation of x. Function h can in practice be a smoothing filter, for example.

FIG. 2b shows several distributed scalar representations in the case where, for example, h is a function that generally follows a Gaussian curve. The y-axis is the set of real numbers, the axis marked f.sub.x (y) is the set of images of the real numbers resulting from application of the function f.sub.x. Curve 201 shows the shape of the function f.sub.x for x=0, curve 202 shows the shape of the function f.sub.x for x=0.3 and curve 203 shows the shape of function f.sub.x for x=0.7.

How the network according to the invention works will be better understood from the following examination of a connection between a neuron i and a neuron j. Starting from the expression for potential X.sub.j, it is possible to obtain the relation: ##EQU3## in accordance with relation (1).

This latter relation in fact represents the convolution product of the function h by the function W.sub.ij applied to the real number x.sub.i, in other words:

Thus, by grouping together the distribution function for neuron i with the weight continuum W.sub.ij characterizing the connection between neuron i and neuron j, this connection appears as the result of filtering the weight continuum by the distributed scalar representation generating function. If we write C.sub.ij =h*W.sub.j, it is possible to describe the connections that terminate at neuron j by the following sequence of operations, using the preceding notations: ##EQU4##

One way of expressing the effect of distributed scalar representation is to consider that the connection is an adaptive function and no longer a simple adaptive gain like in the case of a conventional neural model, thus making it possible to deal more effectively with problems of non-linearity.

It is still however possible to represent a weight continuum in practice. According to the invention, we thus represent the weights by a sampled function. This sampled function between neuron i and neuron j is defined and expressed in the following manner: ##EQU5## where n is a relative whole number, W.sub.ij,n is the value of the continuum of weight W.sub.ij at point n, and .delta. is the Dirac delta function.

From relation (3), using the previous notations, the function s.sub.ij that links the potential X.sub.j of a neuron j to the value x.sub.i of a neuron i of the preceding layer is defined in the following manner: s.sub.ij (x.sub.i)=(h*W.sub.ij *)(x.sub.i) giving ##EQU6## as W.sub.ij,n is constant: we obtain: ##EQU7## by setting: ##EQU8## we obtain: ##EQU9##

The neural model with sampling is shown in FIG. 2c. The summing function of step 21, the saturating function of step 22 and the distribution function of step 23 are the same ones as before, n and m being relative whole numbers,

W.sub.ij,m being the value of weight continuum W.sub.ij at sampling m. At the output from step 23, instead of a single arrow as in the previous cases, there are now two arrows separated by a dotted line that symbolizes sampled values obtained, of which O.sub.j,n is one particular value. The improvement in the neural network's internal connection can be clearly seen here as, instead of the neuron's output being represented by a single value, it is now represented by a sequence of sampled values with an accompanying improvement in accuracy and a higher suitability to the various types of processing involved.

As FIG. 2b shows, the filtering function h is generally a Gaussian function with only a small standard deviation.

Referring to FIG. 5, the method according to the invention is shown in the flow chart. Inputs as functions are received from the preceding neuron layer by the present neuron layer 66. A weighted summation 62 of the inputs is determined. Then a saturation function 62 is applied to the weighted summation. To the results of the saturation function, a distribution function 64 is applied. The distribution function can involve transforming the weighted summation by a convolution product of a smoothing filter and a Dirac Delta function. The result of this transformation can then be sampled 65, resulting in output functions representative of the output sate of the neuron layer.

From FIG. 3, it can be seen that outside the interval [n.sub.1, n.sub.2 ], the value of f.sub.x (y)=h(x-y)=h(y-x) is always small, meaning that weights W.sub.ij,n where n is outside the range [n.sub.1, n.sub.2 ] have little influence in the product W.sub.ij,n f.sub.xi,n of relation (5). For values of n outside the [n.sub.1, n.sub.2 ], the values of weights W.sub.ij,n can hence be set to zero. In practice, the limits of range [n.sub.1, n.sub.2 ] would be chosen so as to enclose asymptotic values of saturating function F.sub.s. In FIG. 3, F.sub.MIN and F.sub.MAX are the upper and lower saturation or asymptotic values of saturating function F.sub.s. For the tangential hyperbolic function for example, F.sub.MIN =-1 and F.sub.MAX =+1. In line with the notation established above, we can now write: ##EQU10##

Having now established the connection model, a learning algorithm needs to be defined. It is necessary to provide this learning phase in order that, regardless of the input to the net, the states of the output neurons are adapted to this input. The learning algorithm defined according to the invention is a gradient algorithm inspired by the back-propagation algorithm discussed in the article by Rumelhart et al cited above.

In the learning algorithm defined according to the invention, a mean value e.sub.QM for the squares of the errors is, for example, defined between the target outputs (t.sub.j) and the outputs actually obtained (x.sub.j):

The operator E{ }computes an average value for the values e.sub.Q defined by: ##EQU11## s identifying all the output neurons.

During the learning phase, all the examples are fed in several times in a random order. Each time an example is presented, the weights W.sub.ij,n vary in the opposite sense to the error gradient: denoting this variation by .DELTA.W.sub.ij,n : ##EQU12## .alpha. being a positive constant, and a reduction in error is thus ensured. The local gradient of the error of the squares will be written as g.sub.ij,n below. ##EQU13##

As e.sub.QM =E{eQ}, it is possible to write: ##EQU14## as differentiation is a linear operator. We consequently obtain: .DELTA.W.sub.ij,n =-.alpha.E{g.sub.ij,n}

In order to carry out the learning phase, it is hence necessary to know the gradients g.sub.ij,n of the error with respect to a weight: ##EQU15## from relations (2) and (6).

Thus, by setting ##EQU16## we obtain g.sub.ij,n =-.delta..sub.j O.sub.i,n

All that is needed now to obtain g.sub.ij,n is to define .delta.j. By applying elementary rules of differential calculus, it is seen that .delta.j can be expressed as a function of the terms: ##EQU17## relative to the neurons k of the layer following it. In effect: ##EQU18## can be expressed as ##EQU19## being the previously-defined interval. ##EQU20## can be written as ##EQU21## succ (j) identifying the neurons of the layer immediately following the layer where neuron j is situated. ##EQU22## from relations (2) and (6). Additionally, ##EQU23## In effect, O.sub.j,n =f.sub.xj,n, and from relations (1) and (4) ##EQU24## which gives the following expression for .delta.j in terms of the .delta.k's for the neurons k of the following layer: ##EQU25##

For the particular case of the last layer, .delta.j is readily expressed as a function of the type of error that is chosen, as, in effect: ##EQU26## Now, from relation (7), for the neurons j belonging to the last layer: ##EQU27##

We consequently have .delta.j=-(xj-tj)F.sub.s (Xj).

Thus, using iteration and making use of the successive quadratic errors between the target outputs and the actual outputs obtained, it is possible to determine the coefficients W.sub.ij,n, this constituting the learning phase.

The method according to the invention can advantageously be applied to space dimension reduction, notably in signal processing for example. Thus, in numerous signal processing applications, observation is effectively represented by a vector belonging to a space of dimension N. The components of this vector are very frequently not independent and are in reality limited to a hyper-surface of dimension M, M being less than N. Some methods of analysis do in fact allow this hyper-surface to be approximated to a hyper-plane. When the hyper-surface is highly curved however, this approximation is not satisfactory, and currently no satisfactory method is in existence. The method according to the invention enables satisfactory results to be obtained. The example that follows considers a two-dimensional case, but is representative of any other higher-dimension problem. In FIG. 4a, the hyper-surface S is represented by a quarter of a circle of unit radius 50.

The data to be processed are synthetic data represented by vectors with coordinates (a,b) chosen at random over the quarter S of the circle in accordance with a uniform law that is a function of, for example, angle. FIG. 4b shows the neural net employed in accordance with the invention, a and b being the input data. The d functions, 41 and 42, are distribution functions yielding the distributed scalar representation functions f.sub.a and f.sub.b defined as in relation (1), neurons 43, 44 and 45 being connected in accordance with the method of the invention. Function f.sub.x1 is the distributed scalar representation function representing the state of neuron 1, indicated by reference numeral 43. A and B are the values obtained at the output from the net. During the learning phase, the net is set up by a self-associative method, meaning that the target values required at the output from the network are equal to the data supplied at the network's input; in this case, if a and b are input data and A and B are the output data obtained, the error e.sub.Q of the squares is given by the following relation: ##EQU28##

As in this network, all information must of necessity pass via neuron 1 indicated by 43 in FIG. 4b, the problem to be computed is reduced to a dimension 1 problem, this giving a satisfactory result as the plot of the output coordinates (A, B) obtained, represented by curve S1 designated as 51, of FIG. 4a is a satisfactory approximation of the target output coordinates represented by curve S. Another solution that consisted of processing the same data with a conventional neural net that was not connected as per the invention, and even though it enabled the dimension to be reduced to 1, yielded an approximation represented by straight line P.

The method according to the invention can also be employed in the field of prediction, where it considerably improves performance. The good results obtained by this method in the processing of non-linear problems are principally due to the fact that the connections behave like adaptive functions and no longer like simple adaptive gains. For example, the smoothing function h of relation (1) employed to define the distribution function can be defined by the Gaussian function: ##EQU29## and adapted as a function of the particular problem that is to be processed by making changes to the standard deviation a which in general is of the order of unity.