Neutral Network With Plural Weight Calculation Methods and Variation of Plural Learning Parameters

The present invention relates to a learning machine for data processing with multi-input single-output circuits connected in a hierarchical structure.

An example of the conventional learning machine with multi-input single-output circuits connected in a hierarchical structure is disclosed in e.g. D.E. Rummelhart et al. "Learning representations by back-propagating errors", Nature Vol. 323 No. 9 (1986). In the conventional learning machine with multi-input single-output circuits connected in a hierarchical structure, each multi-input single-output circuit adds up input signals weighted,. and subjects the resulting signal to a conversion scheme having a saturation characteristic; the output signal thus provided is expressed by ##EQU1## where y[j]is an output signal from a j-the multi-input single-output circuit, y[i]is an output signal from an i-th multi-input single-output circuit in a previous layer, and w[i,j]is the weight charged on the output signal from the i-th multi-input single-output circuit in the previous layer when it is supplied to the j-th multi-input single-output circuit. fnc( ) is a function having a saturation characteristic which can be expressed by e.g. a sigmoidal function ##EQU2##

A learning machine is usually structured in such a manner that multi-input single-output circuits are connected in a hierarchical structure and learns to provide a desired output signal (hereinafter referred to as a supervising signal) in response to input signals. In the learning, an error E is acquired from the supervising signal and the actual output signal in accordance with Equation (2) ##EQU3## where y.sub.p >[j]is an output signal from a j-th multi-input single-output circuit in an output layer for a p-th input pattern, t.sub.p [j]is a supervising signal for Y.sub.p [j], .SIGMA..sub.p is a sum for all the input patterns, .SIGMA..sub.j is a sum for all the output signals in the output layer, and w is a vector including weight w[i,j]as a component (hereinafter referred to as a weight vector).

As shown in Equation (2), the error E can be expressed as a squared sum of a difference between a supervising signal and an actual output signal and so is a function of the weight vector w. A purpose of learning is to change the weight vector to minimize the difference between the supervising signal and the actual output signal, i.e. the error E. The amount of changing the weight vector is decided by ##EQU4## where .epsilon. is a positive constant referred to as a learning rate, .alpha. is a positive constant referred to as an accelerating parameter, .differential.E/.differential.W is a vector including as a component the differentiation of the error expressed by Equation (2) by the weight w[i,j], which is referred to as the steepest descent direction, and w is a vector representation of the weight changing amount in the previous learning. Such a learning algorithm is generally referred to as an error back-propagation method.

In the learning of the conventional learning machine with multi-input single-output circuits connected in a hierarchical structure, the learning rate .epsilon. and the accelerating parameter .alpha. are fixed; they are decided through experience or trial and error and so are not necessarily optimum values. This lengthens the time required for learning. Moreover, the back-propagation method based on the steepest descent method, used for minimizing the error which is the purpose of learning, is not always optimum. Further, the learning may possibly be fallen into a non-effective state where further learning does not reduce any error.

In view of the problems mentioned above, one of the objects of the present invention is to provide a learning machine with multi-input single-output circuits connected in a hierarchical structure which can shorten the time required for learning by using, as a method of minimizing an error between supervising signals (desired output signals and actual output signals, a conjugate gradient method which is considerably improved over the steepest descent method.

Another object of the present invention is to provide a learning machine with multi-input single-output circuits connected in a hierarchical structure which can shorten the time required for learning by advancing the learning with the optimum value of a learning rate being dynamically set.

A further object of the present invention is to provide a learning machine with multi-input single-output circuits connected in a hierarchical structure which can shorten the time required for learning by reducing the amount of operation in calculating errors for plural learning rates relative to a predetermined wight changing direction.

Yet another object of the present invention is to provide a learning machine with multi-input single output circuits connected in a hierarchical structure which can shorten the time required for learning by readjusting the weight to restart the learning when detecting that the learning has been fallen into a non-effective state such that further iterations do not reduce the error.

In accordance with the present invention, a learning machine is provided with multi-input single-output circuits connected in a hierarchical structure comprising hidden layers including plural multi-input single-output circuits which subject a weighted sum of input signals to a non-linear processing by a characteristic function having a saturation characteristic to provide output signals; an output layer includes of plural multi-input single-output circuits which subjects a weighted sum of the output signals from the hidden layers to a non-linear processing by a characteristic function having a saturation characteristic to provide output signals; a supervising signal generating unit generates desired output signals for said output layer, a weight changing direction deciding unit selects and decides, from plural directions, the weight changing direction for minimizing errors between outputs from the supervising signal generating unit and outputs from the output layer; a line searching unit produces weight changing amounts for plural learning rates with respect to the weight changing direction; a weight changing unit selects weight changing amounts from the weight changing amounts acquired by said line searching unit, a hidden layer product-sum storage unit stores a product sum of input signals and weight of the hidden layer and a product sum of the input signals and the direction of changing the weight of the hidden layer; and a non-effective learning state detecting unit detects when the learning has fallen into a unsuitable state where further iterations do not effectively reduce the error.

By means of this construction, the present invention subjects the weighted sum of input signals to a non-linear processing to provide an output signal in both hidden layers and output layer; in order to minimize an error between a desired output signal generated by the supervising signal generating unit and an actual output signal, the invention selects and decides the weight changing direction from the plurality of directions in the weight changing deciding unit; the weight changing amounts for plural learning rates are acquired in the line searching unit; then, on the basis of the product sum of the input signals and the weight in the hidden layer and the product sum of the input signals and the direction of changing the weight in the hidden layer (stored in the hidden layer product sum storage unit), acquires, with less processing than the conventional device, an output from the hidden layer for the weight changing amount set in the line searching unit; and finally, checking in the non-effective learning state detecting unit if the learning has been fallen into a non-effective state or not, selects, in the weight changing unit, the weight changing amounts required by the line searching unit.

Thereafter, in the same manner, the operation of changing weights is repeated, using a learning rate of minimum error in a weight changing direction thereby reducing the error. In this manner, when the weight changing direction is decided, an optimum learning rate is dynamically set in that direction, and the weights are changed using this optimum learning rate to reduce errors in a short time.

In the present invention, in order to minimize error, the weight changing direction is decided by the above weight changing direction deciding unit. Regarding the weight changing direction, the steepest descent direction is "locally" the most effective direction in that the greatest amount of reducing the error can be given with the smallest amount of changing the weight in the neighborhood of the present weight value. On the other hand, a conjugate gradient direction is "globally" the most effective direction in that the weight is changed in a different direction from the weight changing in the previous learning.

The present invention, by storing, in the hidden layer product sum storage unit, the product sum of the input signals and the weights in the hidden layer, and the sum of the input signals and the direction of changing the weight in the hidden layer, greatly reduces the amount of time acquiring the outputs from the hidden layer in calculating the errors for plural learning rates when the weight changing direction is fixed, thereby shortening the time required for learning.

The present invention, when the non-effective learning state detecting unit detects that changing the weight toward the steepest descent conjugate gradient direction cannot reduce an error, decides that the learning has been fallen into the non-effective state and readjusts the weight to restart the learning accordingly, thereby preventing stalling or wasted processing time.

FIG. 1 is a block diagram showing the general arrangement of the conventional learning machine with multi-input single-output, circuits connected in a hierarchical structure;

FIG. 2 is a block diagram showing the detailed arrangement of the conventional learning machine with multi-input single-output circuits connected in a hierarchical structure;

FIG. 3 is a graph of the characteristic function of an adder having a saturation input/output characteristic in the conventional learning machine of FIG. 2;

FIG. 4 is a block diagram showing the arrangement of a learning circuit in the conventional learning machine with multi-input single-output circuits connected in a hierarchical structure;

FIG. 5 is a block diagram showing the details of the learning circuit of FIG. 4;

FIG. 6 is a block diagram showing the learning machine with multi-input single-output circuits connected in a hierarchical structure according to the first embodiment of the present invention;

FIG. 7 is a block diagram showing the arrangement of the learning circuit in the embodiment of FIG. 6;

FIG. 8 is a block diagram of the detailed arrangement of a conjugate gradient deciding unit in the embodiment of FIG. 6;

FIG. 9 is a block diagram showing the learning machine with multi-input single-output circuits connected in a hierarchical structure according to the second embodiment of the present invention;

FIG. 10 is a block diagram showing the arrangement of the learning circuit in the embodiment of FIG. 9;

FIG. 11 is a schematic view of contour lines of an error curve for explaining the operation of the embodiment of FIG. 9;

FIG. 12 is a cross-sectional view between the starting point and point p of the error curve for showing the operation of the embodiment of FIG. 9;

FIG. 13 is a PAD (Problem Analysis Diagram) showing the operation of the line searching unit in the embodiment of FIG. 9;

FIG. 14 is a block diagram showing the learning machine with multi-input single-output circuits connected in a hierarchical structure according to the third embodiment of the present invention;

FIG. 15 is a schematic view of contour lines of an error curve for explaining the operation of the embodiment of FIG. 14;

FIG. 16 is a block diagram showing the learning machine with multi-input single-output circuits connected in a hierarchical structure according to the fourth embodiment of the present invention;

FIG. 17 is a block diagram showing the detailed arrangement of a hidden layer product sum storage unit in the embodiment of FIG. 16; and

FIG. 18 is a block diagram showing the learning machine with multi-input single-output circuits connected in a hierarchical structure according to the fifth embodiment of the present invention.

Prior to explaining several embodiments of the present invention, the conventional learning machine with multi-input single-output circuits connected in a hierarchical structure will be further explained with reference to FIGS. 1 to 5.

FIG. 1 shows the general arrangement of the conventional learning machine with multi-input single-output circuits connected in a hierarchical structure. In FIG. 1, 201 and 202 are input terminals; 212 is an output terminal; 217 is a learning circuit; 218, 219 and 220 are multi-input single-output circuits; 221 is an output layer; and 222 denotes hidden layers. As seen from FIG. 1, the learning machine, through the construction in which multi-input single-output circuits are connected in a hierarchical structure, process input from input terminals 201, 202 and to provide output signals at the output terminals. Among the multi-input single-output terminal circuits, the layer consisting of the multi-input single-output circuits generating the output signals is referred to as the output layer, and the layers consisting of the other multi-input single-output circuits are referred to as the hidden layers. The hidden layers may be a single layer of the multi-input single-output circuits or plural layers thereof.

FIG. 2 shows the detailed arrangement of the conventional learning machine with multi-input single-output circuits connected in a hierarchical structure. In FIG. 2, 203, 204, 205, 206, 207 and 208 are variable weight multipliers; 209, 210 and 211 are adders having a saturation input/output characteristic; 212 is an output terminal; 213 is a supervising signal generating unit; 214 is an error calculating unit; 215 is the steepest descent direction decision unit; and 216 is a weight changing unit. In comparison with the general arrangement of the learning machine of FIG. 1, the learning machine shown in FIG. 2 comprises the output layer consisting of one multi-input single-output circuit and the hidden layer of one layer consisting of two multi-input single-output circuits. As seen form FIG. 2, each multi-input single-output consists of two variable weight multipliers and one adder. Thus, the learning machine is constructed so that plural multi-input single=output circuits are connected in a hierarchical structure. FIG. 3 is a graph of the characteristic function of the adders 209, 210 and 211 having a saturation input/output characteristic. The adders 209, 210 and 211 have the saturation input/output characteristic shown in FIG. 3.

The conventional learning machine with multi-input single-output circuits connected in a hierarchical structure, in order to provide a desired output signal for input signals, changes the weight applied to the input signals by the variable weight multiplier. FIG. 4 shows the arrangement of the learning circuit 217 of this conventional learning machine. In FIG. 4, 223 is an input terminal of the output from the output layer; 224 is an input terminal of the output from the hidden layer; 225 is an input terminal of the input signals; 226 is an output terminal of the weight in the output layer; and 227 is an output terminal of the weight in the hidden layer. In the conventional multi-input single-output circuits connected in a hierarchical structure, the supervising signal generating unit 213 generates a supervising signal (desired output signal) t.sub.p ;[j]for the input signals. The error calculating unit 214 calculates an error E expressed by Equation 2 from the supervising signal t.sub.p [j]and an actual output signal y.sub.p [j]. The error calculating unit sends a difference signal between the supervising signal and the output signal t.sub.p [j]-y.sub.p [j], which si required to change the weight, to the steepest descent direction decision unit 215. On the basis of the above difference signal, the output signal from the output layer, the output signals from the hidden layer, the input signals and the weights in the output layer, the steepest descent direction of the error E in a weight space where the weights are expressed in vector. The steepest descent direction can be expressed by ##EQU5## The right side of Equation (4) is a vector expression of the differentiation of the error E by weights. The steepest descent direction decision unit 215 multiplies the steepest descent direction by a learning rate to provide an output to the weight changing unit 216. The weight changing unit 216 acquires the weight changing amount from Equation (3) and changes the weight to be multiplied in the respective variable weight multipliers 203, 204, 205, 206, 207 and 208. In this way, the operation of acquiring the weight changing amount is repeated by the steepest descent method to gradually reduce the error. When the error has been sufficiently decreased, decision that the output signal has approached the desired value is made to complete the learning.

FIG. 5 shows the details of the learning circuit 217 in the conventional learning machine with multi-input single-output circuits connected in a hierarchical structure. In FIG. 5, 228 is a learning completion decision unit; 229 and 230 are differentiation coefficient calculation units; 231 is an output differentiation coefficient storage unit; 232 is a .delta. storage unit; 233 is a .delta.* w buffer; 234 is a hidden layer differentiation coefficient storage unit; 235 is an input signal storage unit; 236, 237, 238, 239 and 240 are multipliers; 241, 242, and 244 are adders; 245 and 246 are learning rate multipliers; 247 is an output layer weight storage unit; and 248 is a hidden layer weight storage unit.

Referring to FIG. 5, the operation of the learning circuit will be explained. The weight changing amount supplied from the steepest descent direction decision unit 215 can be expressed by ##EQU6## where w[i,j]is the weight to be multiplied by the output signal from an i-the multi-input single-output circuit in the previous layer, which is supplied to a j-the multi-input single-output circuit, .DELTA.w.sub.g [i,j]is the weight changing amount of w[i,j]relative to the steepest descent direction, and .epsilon. is a learning rate. Equation (5) can be transformed into ##EQU7## where net.sub.p [i]=.SIGMA..sub.i (w[i,j]* y.sub.p [i]), and y.sub.p [i]is an output

signal from an i-the multi-input single-output circuit in the previous layer corresponding to the p-the input signals. Equation (6) can be further transformed into

where ##EQU8## The manner of transforming the equation depends on whether the j-the multi-input single-output circuit resides in the output layer or the hidden layer. * When the j-th multi-input single-output circuit resides in the output layer, ##EQU9## where fnc(net.sub.p [j]) is a characteristic function (=y.sub.p [j]) of each multi-input single-output circuit in the output layer. Using Equation (2), this equation can be further transformed into ##EQU10## Consequently, it can be expressed by ##EQU11## In FIG. 5, the adder 241 calculates (t.sub.p [j]-y.sub.p [j]) in Equation (9), the differentiation coefficient unit 229 calculates .delta.fnc(net.sub.p [j]) /.delta.net.sub.p [j]), and the multiplier 236 calculates (t.sub.p [j]-y.sub.p [j]) * .delta.fnc(net.sub.p [j]) / .delta.net.sub.p [j], through the output layer differentiation coefficient storage unit 231. This value, which is .delta.w.sub.p [j], is stored in the .delta. storage unit 232. The multiplier 237 calculates a product of .delta.w.sub.p [j]and the hidden layer output y.sub.p [i]. the multiplier 245 multiplies this product by a learning rate .DELTA.to provide .DELTA.w.sub.g [i,j]expressed by Equation (9), .alpha.times of the weight changing amount in the previous learning is added to .DELTA.w.sub.g [i,j]of Equation (9) to provide the weight changing amount .DELTA.w[i,j]expressed by Equation (3). The adder 243 calculates a sum of this weight changing amount and the weight before change thereby changing the weight in the output layer stored in the output layer weight storage unit 247.

*When the j-th multi-input single-output circuit resides in the hidden layer, ##EQU12## where net.sub.p [k]=.SIGMA.(w[j,k]*y.sub.p [j], and w [j,k]is a weight to be multiplied by the output y.sub.p [j]of the j-th multi-input single-output circuit in the next layer. Using Equation (8), this equation can be further transformed into ##EQU13## Consequently, it can be expressed by ##EQU14## In FIG. 5, the multiplier 238 calculates a product of the an output .delta.w.sub.p [k]from the .delta.storage unit 232 and the weight w[j,k]in the output layer, and the .delta.* w buffer 233 and the adder 242 calculates ##EQU15## The differentiation coefficient .delta.fnc(net.sub.p [j]) /.delta.net.sub.p [j]), is supplied to the multiplier 239 through the hidden layer differentiation coefficient storage unit 234, which calculates a product of this coefficient and the above ##EQU16## The multiplier 240 multiplies this product by the input signal y.sub.p [i]. The multiplier 246 multiplies this product by a learning rate .epsilon. to provide .DELTA.w.sub.g [i,j]expressed by Equation (10). .alpha.times of the weight changing amount in the previous learning is added to .DELTA.w.sub.g [i,j]of Equation (10) to provide the weight changing amount .DELTA.w[i,j]expressed by Equation (3). The adder 244 calculates a sum of this weight changing the weight storage unit 248.

In this way, in order to reduce the error between the output signals from the output layer and supervising signal, the amount of changing the weight in the output layer is acquired using difference signals between the output signals and the supervising signal and the weight changing amount is successively acquired form the output side towards the input side. Such a learning algorithm is referred to as a back-propagation method.

In the conventional learning machine having the construction described above, the learning rate .epsilon. and the accelerating parameter .epsilon. are fixed; they are decided through experience or trial and error and so are not always optimum values. This lengthens the time required for learning. Moreover, the weight changing direction using the differentiation of the error E by the weights (i.e. the steepest descent direction) is not always optimum. Further, the learning may possibly fall into a non-effective state where further learning does not reduce any error. FIG. 6 shows the arrangement of a learning machine with multi-input single-output circuits connected in a hierarchical structure according to a first embodiment of the present invention. In FIG. 6, 1 and 2 are input terminals; 3, 4, 5, 6, 7 and 8 are variable weight multipliers; 9, 10 and 11 are adders having a saturation input/output characteristic; 12 is an output terminal; 13 is a supervising signal generating unit; 14 is an error calculating unit; 15 is a conjugate gradient direction decision unit; 18 is a weight changing unit; 17 is a learning circuit; 18, 19 and 20 are multi-input single-output circuits; 21 is an output layer; 22 is a hidden layer; and 61 is a learning rate multiplier.

In the learning machine according to this embodiment, multi-input single-output circuits are connected in a hierarchical structure; each circuit consists of variable weight multipliers and an adder having a saturation input-output characteristic. Each multi-input single-output circuit adds up input signals, after having been weighted, and subjects the sum to a non-linear processing to provide an output signal. In the learning of the learning machine according to this embodiment, in order that an output from the output layer 21 is equal to an output from the supervising signal generating unit 13, the weight to be multiplied in the multipliers 3, 4, 5, 6, 7 and 8 are changed. If the vectors having weights as components are referred to as weight vectors, the amount of changing the weight vectors can also be expresses in vector form. The direction of this weight changing vector is referred to as a weight changing direction. In this embodiment, a conjugate gradient direction is used as the weight changing direction. The conjugate gradient direction is given by ##EQU17## where g is the steepest descent direction expressed by Equation (4), and B is a constant expressed by ##EQU18## d' is a conjugate gradient direction in the previous learning, and .parallel.g.fwdarw.".parallel. is the norm of the vector in the steepest descent direction g.fwdarw.' in the previous learning. Incidentally, the weight changing direction in an initial learning is defined as the steepest descent direction.

FIG. 7 shows an arrangement of the learning circuit 17. In FIG. 7, 23 is an input terminal of the output from the output layer; 24 is an input terminal of the output from the hidden layer; 25 is an input terminal of the input signals; 26 is an output terminal of the weight of the output layer; 27 is an output terminal of the weight of the hidden layer; 28 is the steepest descent calculation unit; 29 is a conjugate gradient direction calculation unit; 30 is a weight changing direction decision unit; and 31 is a learning iteration counting unit.

Now referring to FIG. 7, explanation will be given for the operation of learning circuit 17 in the learning machine with multi-input single-output circuits connected in a hierarchical structure. The supervising signal generating unit 13 provides a desired signal for the input signals as a supervising signal to the error calculation unit 14 which produces a difference signal between the supervising signal and an actual output signal. The conjugate gradient direction decision unit 15 acquires the weight changing direction from the difference signal. Further, the weight changing unit 16 changes the weight in accordance with the weight changing amount acquired by the learning rate multiplier 61. The conjugate gradient direction decision unit 15 acquires the weight changing direction in the following manner. The steepest descent direction calculation unit 28, on the basis of the difference signal provided from the error calculation unit 14, the output signal from the output layer, the input signal and the weight of the output layer, acquires the steepest descent direction expressed by Equation (4). The conjugate gradient direction calculation unit 29 acquires a conjugate gradient direction from the steepest descent and the weight changing direction in the previous learning. The learning iteration counting unit 31 counts the iteration of learning to provide a selection signal of which the steepest descent and the conjugate gradient direction is to be used as a weight changing direction.

Generally, in the conjugate gradient method, using the steepest descent for the learning iteration for each weight accelerates the convergence of the error. Therefore, as before, the learning iteration counting unit 31 produces the selection signal so that the steepest descent direction is used for the learning iteration every number of the weights. The weight changing direction decision unit 31 changes the weight changing direction between the steepest direction and the conjugate gradient direction in accordance with the selection signal produced from the learning iteration counting unit 31. The learning rate multiplier 61 multiplies the weight changing direction thus decided by a learning rate to provide a weight changing amount. Finally, the weight changing unit 16 changes the weight.

The details of the conjugate gradient decision unit 15 are shown in FIG. 8. In FIG. 8, 32 is an input terminal of the weight of the output layer; 33 is an output terminal of the direction of changing the weight of the output layer; 34 is an output terminal of the direction of changing the weight of the hidden layer; 35 is an input terminal of the difference signal; 36 and 37 are differentiation coefficient units; 38 is an output layer differentiation coefficient storage unit; 40 is a .delta. storage unit; 41 is a unit for storing the steepest descent direction in the output layer at this time; 42 is unit for storing the steepest descent direction in the output layer at the previous time; 43 is a .beta. calculation unit; 44 is a unit for storing the direction of changing the weight of the output layer at the previous time; 45, 46, 47, 48, 49, 50 and 51 are multipliers; 52 is an adder; 53 is a .delta. * w buffer; 54 is an input signal storage unit; 55 is a unit for storing the steepest descent direction in the hidden layer in real time; 56 is a unit of storing the steepest descent direction in the hidden layer at a previous time; 58 is a unit of storing the direction of changing the weight of the hidden layer at the previous time; 59 and 60 are weight changing direction switching unit; and 63 and 64 are adders.

With reference to FIG. 8, detailed explanation will be given for the operation of the conjugate gradient direction decision unit 15. The steepest descent direction calculation unit 28 has substantially the same arrangement as that of the steepest descent calculation unit 215 of the conventional learning machine with multi-input single-output circuits as shown in FIG. 5. The multiplier 46 provides the steepest descent direction relative to the weight of the output layer, and the multiplier 51 provides the steepest descent direction relative to the hidden layer. The signals indicative of these steepest descent directions are sent to the weight changing direction switching units 59 and 60 in the weight changing direction decision unit 30 and also to the real time output layer steepest direction storage unit 41 and the real time hidden layer steepest descent steepest descent direction storage unit 55 in the conjugate gradient direction calculation unit 29.

Using Equation (12), the .beta. on the basis of the steepest descent directions stored in the learning at real time store in the storage units 41 and 55 and the steepest descent directions stored in the learning at the previous time stored in the previous time output layer steepest descent direction storage unit 56. The multipliers 48 and 51 acquire products, of .beta. thus obtained and the weight changing directions d in the previous learning. The adders 63 and 64 acquire sums of these products and the steepest descent directions at this time. These sums are sent, as conjugate gradient directions, to the weight changing direction switching units 59 and 60.

On the other hand, the learning iteration counting unit 31 counts the iteration of learning and provides such selection signals as "select the steepest descent direction" as the weight changing direction for the learning in every number of iterations equal to the number of the weights and "select the conjugate gradient direction for the learning carried out in every number of other iterations." The weight changing direction switching units 59 and 60 switch the weight changing direction between the steepest descent direction and the conjugate gradient direction in accordance with these selection signals.

In this way, in the learning machine according to this embodiment, the conjugate gradient direction decision unit 15 decides whether the steepest descent direction or the conjugate gradient direction is the weight changing direction. If the conjugate gradient direction is selected as the weight changing direction, then the weight is changed to a direction different from the weight changing direction in the learning up until the previous iteration; this conjugate gradient direction is generally the most effective direction. In accordance with this embodiment, the learning is completed in a shorter time than in the conventional learning machine which adopts the steepest descent direction as the weight changing direction.

Incidentally, although in this embodiment, the steepest descent direction is adopted as the weight changing direction for the learning carried out in every number of iterations equal to that of the weights, the conjugate gradient direction may be adopted in all the iterations other than first time learning, or the steepest descent direction may be adopted for the learning to be carried out in every number of iterations equal to half of the number of the weights. Moreover, although in this embodiment, the hidden layer 22 includes of the multi-input single-output circuits, each of which processes two input signals to provide an output signal, as in the general arrangement of the conventional learning machine shown in FIG. 1, the hidden layer 22 may consist of any number of the multi-input single-output circuits; and it may consist of the multi-input single-output circuits connected in a hierarchical structure.

FIG. 9 shows an arrangement of the learning machine with multi-input single-output circuits connected on a hierarchical structure according to the second embodiment of the present invention. In FIG. 9, 65 is a weight changing unit; 66 is a learning rate initialization unit; 67 is a searching unit; 68 is a parabola approximation unit; and 69 is a learning circuit.

As seen from comparison between FIGS. 2 and 9, this embodiment differs from the conventional learning machine in that the learning rate initialization unit 66, the line searching unit 67 and the parabola approximation unit 68 are provided. Specifically, in the conventional learning machine, the learning parameter is fixed whereas, in this embodiment, the weight changing amounts for plural learning rates are acquired by the line searching unit 67, the error for each learning rate is acquired by the hidden layer 22, the output layer 21 and the error calculation unit 14, and the error curve is parabola-approximated from the errors for the learning rates with smaller errors by the parabola approximation unit 68 so as to provide the errors at the top of the parabola; using the learning rates of these errors the weights are changed by the weight changing unit 65.

FIG. 10 shows an arrangement of the learning circuit according to this embodiment. In FIG. 10, 70 and 71 are multipliers; 72 is a unit of calculating the learning rate at the top of a parabola; 73 and 74 are adders; and 75 is a weight storage unit. The supervising signal generating unit 13 generates a supervising signal for the input signals, and the error calculation unit 14 produces a difference signal and error between the actual output signal and the supervising signal. The steepest descent direction calculation unit 28 acquires the steepest descent direction as the weight changing direction on the basis of the above error, the output signal from the output layer, the output signals from the hidden layer, the input signals, and the weight of the output layer.

As expressed in Equation (2), the error E, which is the function of a weight vector, can be represented by a curve in the weight space where the weights in the variable weight multipliers 3, 4, 5, 6, 7 and 8 are represented in vector form. This curve is referred to as an error curve. FIG. 11 schematically shows contour lines in the error curve for explaining the operation in this embodiment. In FIG. 11, w[i ,j.sub.1 ]and w[i.sup.2,j.sub.2 ]are weights to be multiplied by any two multipliers of the variable multipliers 3, 4, 5, 6, 7 and 8; w.sub.o is vector representation of the value of the weights initialized by a random number; g.sub.o is the steepest descent direction in the first learning; p.sub.1 is an error minimum point in the first iteration of the learning; g.sub.1 is the steepest descent direction at point p.sub.1 ; and p.sub.2 is an error minimum point in the second iteration of the learning. In the iteration by the learning machine according to this embodiment, the weights in all the multipliers are first initialized by random numbers; this means that the starting point (w.sub.o) on the error curve in FIG. 11 is set for the random numbers. The object of the learning is, by changing the weights to the direction where the error decreases from the starting point on the error curve in FIG. 11, to repeat the search of the error minimum point to reach the global error minimum point. In the first learning, the steepest descent direction g.sub.o expressed by Equation (4) is set as the weight changing direction, and the point where the error in the direction of g.sub.o is minimum is acquired. This is referred to as line searching in the direction of g.sub.o. The learning rate initialization unit 66 produces a positive value which is suitable as an initial value of the learning rate in the line searching.

FIG. 12 shows the cross-section between the starting point and point p.sub.1 on the error curve for explaining the operation in this embodiment. In FIG. 13, .epsilon.1 is an initial value of the learning rate which can be expressed by

E.sub.k (k.gtoreq.0) is an error for the learning rate .epsilon..sub.k which can be expressed by

.epsilon..sub.v is a learning rate at the top of the parabola-approximated error curve; and E.sub.v is an error for the learning rate .epsilon..sub.v.

FIG. 13 is a PAD (Problem Analysis Diagram) for explaining the operation of line searching in the learning machine with multi-input single-output circuits connected in a hierarchical structure according to this embodiment. In the line searching, first, an error for the initial learning rate .epsilon..sub.1 is acquired. Specifically, the line searching unit 67 multiplies the initial value .epsilon..sub.1 by the steepest descent direction g.sub.0 to provide a weight changing amount for the initial value .epsilon..sub.1 ; this weight changing amount is supplied to the weight changing unit 65 (FIG. 10). In the weight changing unit 65, the adder 73 acquires a sum of the weight changing amount in the output layer and the weight of the output layer, and the adder 74 acquires a sum of the weight changing amount in the hidden layer and the weight of the hidden layer. Thus, the weight changing unit 65 produces the weights for the initial value .epsilon..sub.1 of the learning rate from the weight output terminal 26 for the output layer and the weight output terminal 27 for the hidden layer. Using these weights, the output signals from the hidden layer 22 and the output layer 21 are acquired, and the error is acquired by the error calculation unit 14 by comparing the supervising signal with the output signal from the output layer 21. In this way, the initial error E.sub.1 for the initial learning rate is acquired.

As seen from the PAD FIG. 13, the line searching unit 67 compares the initial error E.sub.1 with the error E.sub.0 at the starting point on the error curve. If E.sub.1 is smaller than E.sub.0, it can be expected that the error for the larger learning rate is smaller; the operation of doubling the learning rate is repeated until the error starts to increase. In the first learning shown in FIG. 12, E.sub.1 .gtoreq.E.sub.0 so that the operation of doubling the learning rate .epsilon. is repeated to provide the corresponding error. Then E.sub.0 >E.sub.1 >E.sub.2 >E.sub.3 so that the line searching unit 67 acquires the weight changing amount for up to the learning rate .epsilon..sub.3. On the other hand, if E.sub.1 is larger than E.sub.0, it can be expected that the error for the smaller learning rate is smaller; the operation of decreasing the learning rate by half is repeated until the error becomes smaller than the error at the starting point. Next, the line searching unit 67, in order for the parabola approximation unit 68 to approximate the neighborhood of the error minimum point, calculates the errors at the points on the weight space in FIG. 12 where the weight varies at regular intervals. Specifically, assuming that

and so .epsilon..sub.2 -.epsilon..sub.1 =.epsilon..sub.2.5 -.epsilon..sub.2 =.epsilon..sub.3 -.epsilon..sub.2.5, the error .epsilon..sub.2.5 can be expressed by

In FIG. 12, E.sub.2.5 <e.sub.2 <E.sub.3 <E.sub.1, so that errors E.sub.2, E.sub.2.5, E.sub.3 give the three minimum errors and the learning rates E.sub.2, E.sub.2.5 and E.sub.3 are sent to the unit 72, of calculating the learning rate at the top of the parabola approximation unit 65. The unit 72 approximates the error curve in the neighborhood of the error minimum point by the parabola passing the three minimum error points to provide the error near the top. Namely, the calculation unit 72 calculates the learning rate at the top of the parabola-approximated error curve using ##EQU19## The multipliers 70 and 71 multiplies the steepest descent g.sub.o by .epsilon..sub.v to provide a weight changing amount; the weight changing unit 65 sets the weight for the learning rate .epsilon..sub.v ; the hidden layer 22 and the output layer 21 acquire the output signals (FIG. 9); and the error calculation unit 14 acquires the error E.sub.v. The parabola approximation unit 68 compare the errors thus obtained E.sub.v, E.sub.2, E.sub.2.5, and E.sub.3, and acquires the weight changing amount .epsilon..sub.2.5 *g.sub.o for the learning rate .epsilon..sub.2.5, which provides the smallest error E.sub.2.5. The adder 73 and 74 acquire a sum of the weight changing amount and the weight w.sub.o, and store it as a new weight, respectively. Thus, the weight vector is changed as expressed by

The explanation hitherto made relates to the first iteration of the learning.

In the second iteration of the learning, the steepest descent direction calculation unit 28 acquires, as a weight changing direction, the steepest descent direction g.sub.1 at point P.sub.1 expressed by w.sub.1 in the weight space (FIG. 11). The learning rate initialization unit 66 sets the initial learning rate .epsilon..sub.1 for the first iteration of the learning and the larger value between .epsilon..sub.1 and the rate used in the previous for the second iteration of the learning et seg. That is, in the second iteration of the learning et seg., if an optimum value of the previous learning rate is larger than .epsilon..sub.1, the optimum value of the previous learning rate is adopted as an initial learning rate; if the former is smaller than the latter, .epsilon..sub.1 is adopted as an initial learning rate. Thus, the optimum value of the previous learning rate can be used as an initial learning rate in the present learning to effectively set the learning rate which is optimum for the present learning. Also, if the optimum value of the previous learning rate is smaller than .epsilon..sub.1, .epsilon..sub.1 is used as an initial learning rate for searching the error minimum point in the present learning. This permits the error minimum point to reach a global error minimum point. The line searching unit 67 varies the learning rate by twice or half in the direction of g.sub.1 to provide a weight changing amount decreasing the error. The parabola approximation unit 68 parabola-approximates the error curve to acquire the weight giving a minimum error of g.sub.1, thus providing an error minimum point P.sub.2 in the second iteration of the learning.

In this way, in the learning machine according to this embodiment, by repeating the searching in the steepest descent direction, the optimum learning rate is dynamically set to decrease the error and to approach the global error minimum point. In accordance with this embodiment, the line searching until 65 makes the learning rate variable when the weight is changed by the weight changing unit 65, and sets the learning rate dynamically giving the minimum error in the weight changing direction. Thus, the time required for the learning is shortened.

Incindentially, although in this embodiment, the learning initialization unit 66 sets the initial learning rate .epsilon..sub.1 for the first iteration of the learning and the larger value between .epsilon..sub.1 and the learning rate used in the previous iteration for the second iteration of the learning et seq., it may fix the initial value to .epsilon..sub.1. Moreover, although in this embodiment, the hidden layer 22 includes multi-input single-output circuits each of which processes two input signals to provide an output signal. As in the general arrangement of the conventional learning machine shown in FIG. 1, the hidden layer 22 may consist of any number of the multi-input single-output circuits and it may consist of the multi-input single-output circuits connected in a hierarchical structure.

FIG. 14 shows an arrangement of the learning machine with multi-input single-output circuits connected in a hierarchical structure according to a third embodiment of the present invention. As seen from comparison with FIG. 9, a learning circuit 76 in this embodiment differs from the second embodiment of FIG. 69 in that the steepest descent direction calculation unit 28 is replaced by the conjugate gradient direction decision unit shown in FIG. 6. In this embodiment, the error calculation unit 14 calculates the error from the difference between supervising signals and actual output signals. The conjugate gradient direction decision unit 15 sets the weight changing direction to the steepest descent direction expressed by Equation (4) for the first iteration of the learning to be carried out for every number of iterations to the number of weights, and sets to the conjugate gradient direction for the learning to be carried out in every other number of iterations. The learning initialization unit 66 sets the initial learning rate .epsilon..sub.1 for the first iteration of the learning, and the larger value between .epsilon..sub.1 and the learning rate used in the previous iteration for the second iteration of the learning et seq. The line searching unit 67, if the error for an initial learning rate is smaller than the error at the starting point, repeats an operation of doubling the learning rate .epsilon. until the error starts to increase, or the line searching unit 67, if the error for an initial learning rate has exceeded the error at the starting point. FIG. 15 schematically shows contour lines in the error curve for explaining the operation in this embodiment. In FIG. 15, w[i.sub.1, j.sub.1 ]and w[i.sub.2, j.sub.2 ]are weights to be multiplied by any two multipliers of the variable multipliers 3,4,5,6,7 and 8; w.sub.o is vector representation of weights initialized by random numbers; g.sub.o is the steepest descent direction in the first iteration of the learning; g.sub.1 is the steepest descent direction at point P.sub.1 ; B.sub.1 is a constant obtained by Equation (12) in acquiring the conjugate gradient direction at P.sub.1 expressed by Equation (11); and P.sub.2 is an error minimum point in the second iteration of the learning.

In this embodiment, the error minimum point P.sub.1 in the first iteration of the learning is acquired in the same way in the second embodiment. Thereafter, the weight changing direction in the second iteration of the learning is set to the conjugate gradient direction expressed by Equation (11) using the steepest descent direction g.sub.1 at P.sub.1 and to the weight changing direction g.sub.o (=d.sub.o) in the first iteration of the learning. The error minimum point P.sub.2 for the direction of d.sub.1. Because the conjugate gradient direction is adopted as the weight changing direction in FIG. 15, the error can be minimized in less iterations than in the case of FIG. 11 where the steepest descent direction is adopted as the weight changing direction. The reason is that the conjugate gradient method, in which the weight changing directions in a weight space at respective iterations of learning are set to be different from one another, can change the weight more effectively than the steepest descent method wherein the weight is changed on the basis of the local gradient on a weight curve. In this way, in accordance with this embodiment, a weight changing direction is set to the conjugate gradient direction by the conjugate gradient direction decision unit 15, the learning rate is acquired by the line searching unit 67 and the parabola approximation unit 68 so that the error can be minimized in the weight changing direction, and optimum learning rate is used to change the weight by the weight changing unit 65. Thus, the time required for learning is shortened.

Additionally, in this embodiment, the conjugate gradient direction decision unit 15 sets the weight changing direction to the steepest descent direction expressed by Equation (4) for the first iteration of the learning, and for the second iteration of the learning et seq., sets to steepest descent direction for the learning to be carried out in every number of times equal to the number of weights, and the conjugate gradient direction for the learning to be carried out in learning et seq., to the steepest descent direction for the learning to be carried out in every half number of iterations equal to the number of weights, and sets to the conjugate gradient direction for the learning to be carried out in every other number of iterations. Moreover, if the error does not become smaller than the error at the starting point although the line searching in the conjugate gradient direction has been done to make the learning rate smaller than a predetermined value .epsilon..sub.min, the weight changing direction to the steepest descent direction when the conjugate gradient direction is not suitable as the weight changing direction, the global error minimum point can be reached in a shorter time. Moreover, in this embodiment, the hidden layer 22 includes the multi-input single-output circuits each of which processes two input signals to provide and output signal, as in the general arrangement of the conventional learning machine shown in FIG. 1, the hidden layer 22 may consist of any number of the multi-input single-output circuits and it may consist of the multi-input single-output circuits connected in a hierarchical structure.

FIG. 16 shows an arrangement of the learning machine with multi-input single-output circuits connected in a hierarchical structure according to a fourth embodiment of the present invention. In FIG. 17, 22 is a hidden layer product-sum storage unit; 78 is a line searching unit; and 79 is a learning circuit. In this embodiment, the error calculation unit 14 calculates the error from the difference between supervising signals and actual output signals. The conjugate gradient direction decision unit 15 set the weight changing direction to the steepest descent direction expressed by Equation (4) for the first iteration of the learning, and for the second iteration of the learning et seq., sets to the steepest descent direction for the learning to be carried out in every number of iterations equal to the number of weights, and sets to the conjugate gradient direction for the learning to be carried out in every other number of iterations.

The learning initialization unit 66 sets the initial learning rate, and the larger value between .epsilon..sub.1 and the second iteration of the learning et seq. The straight line searching unit 78 in this embodiment is a combination of the line searching unite 67 and the parabola-approximation unit 68 in FIG. 14. The line searching unit 78, therefore, if the error for an initial learning rate is smaller than the error at the starting point, repeats the operation of doubling the learning rate .epsilon. to provide a weight changing amount until the error starts to increase; or the line searching unit 78, if the error for an initial learning parameter has exceeded the error at the starting point, repeats an operation of decreasing learning rate by half to provide a weight changing amount until the error becomes smaller than the error at the starting point. Thereafter, the line searching unit 78 approximates the error curve in the neighborhood of the error minimum point by the parabola passing the three minimum error points to provide the error near the top. In this way the weight changing direction is fixed, the line searching unit 78 varies the learning rate, and acquires a weight changing amount for minimum error; this embodiment is different from the third embodiment in that the output signal from the hidden layer is acquired through the hidden layer product-sum storage unit.

FIG. 17 shows the details of the hidden layer product-sum storage unit 77 in a block form. in FIG. 17, 80 is the input terminal or input signals; 81 is the input terminal of changing directions; 82 is the input terminal of weights; 83 is the input terminal of a learning parameter; 84 is the output terminal of hidden layer output signal; 85, 86, 87 are multipliers; 88 is a unit of storing the product-sum of the input signals and the changing directions; 90 is an adder; and 91 is a non-linear conversion unit. The hidden layer output signal y.sub.p [i]can be expressed by

where w[i,j]is the weight multiplied by the input signal y.sub.p [i]when it is supplied to the j-th multi-input single-output circuit, and fnc () is a characteristic function of the adders included in the hidden layer; this characteristic function is a non-linear function having a saturation characteristic. Assuming that the component of the weight changing direction decision unit 15 which corresponds to the changing amount of w[i,j], is d[i,j], hidden layer output signal y.sub.p [j]for learning rate .epsilon. is expressed by

Therefore, it is necessary to carry out the multiplication twice the number of the input signals whenever y.sub.p [j]is to be acquired. In this embodiment, by previously storing, in the storage unit 89 for the product-sum of the input signals and the weights, ##EQU20## and in the storage unit 89 for the product-sum of the input signals and the weight changing directions, ##EQU21## the hidden layer output signal y.sub.p [j]for the learning rate .epsilon.can be acquired by ##EQU22## Namely, the multiplier 87 multiplies the learning rate .epsilon.by the output B.sub.p [j]from the storage unit 88 for the product-sum of input signals and weight changing direction, and the adder 90 acquires the sum (A.sub.p [j]+.epsilon.* B.sub.p [j]thus obtained, and the output A.sub.p [j]from storage unit 89 for the product-sum of input signals and weights. The non-linear conversion unit 91 subject the output from the adder 90 to the non-linear conversion equivalent to the characteristic function of the adders in the hidden layer to provide y.sub.p >[j]expressed by Equation (22). In this way, in the line searching along for the weight changing direction .sub.d, multiplication is required the same number of times as in the third embodiment to provide the hidden layer output signal y.sub.p [j]for the initial learning rate; and the multiplication is required only once to provide the hidden layer output signal y.sub.p [j]for the other learning rate. This greatly reduces the number of operations in the line searching thereby shortening the time required for learning.

Additionally, although in this embodiment the weight changing direction is set to the conjugate gradient direction decision unit 15, it may be set to the steepest descent direction. Moreover, although in this embodiment, the hidden layer 22 consists of the multi-input single-output circuits each of which processes two input signals to provide an output signal, as in the general arrangement of the conventional learning machine shown in FIG. 1, the hidden layer 22 may consist of any number of the multi-input single-output circuits, and it may consist of the multi-input single-output circuits connected in a hierarchical structure.

FIG. 18 shows an arrangement of the learning machine with multi-input single-output circuits connected in a hierarchical structure according to the fifth embodiment of the present invention. In FIG. 18, 92 is non-effective learning state detection unit, and 93 is a learning circuit. In the learning machine according to this embodiment, the conjugate gradient direction decision unit 15 decides a weight changing direction; the searching unit 78 sets plural learning rates in the weight changing direction to provide a weight changing amount giving a minimum error; and the weight changing unit 65 changes the weight. The non-effective learning detection unit 92 compares the error before the weight changing with the result after the weight changing; then the weight changing direction is steepest descent direction and also the decreasing rate of the error is smaller than a certain value, the non-effective learning detection unit 92 detects that the learning has fallen into a state where further learning cannot effectively reduce the error, and initializes the weights by random numbers to restart the learning. In this way, in accordance with this embodiment, when the learning has stalled it is detected by the non-effective state detection unit 92, and the weight is readjusted to restart the learning. This prevents the learning from stalling and shortens the time required for the learning.

Additionally, although in this embodiment, the non-effective state detection unit 92 detects the non-effective learning state by detecting that the weight changing direction is the steepest descent direction and also the error decreasing rate is smaller than a certain value, the non-effective state may be detected by detecting that the absolute value of the weight is larger than a certain value. Moreover, although in this embodiment, the weights are adjusted by initializing all the weights by random number, they may be adjusted by equally compressing all the weights or only by compressing only the maximum weight. Further, although in this embodiment, the hidden layer 22 consists of the multi-input single-output circuits each of which processes two input signals to provide and output signal, as in the general arrangement of the conventional learning machine as shown in FIG. 1, the hidden layer 22 may consist of any number of the multi-input single-output circuits and it may consist of the multi-input single-output circuits connected in a hierarchical structure.