US 5,490,236 AGrant
Method of Assigning Initial Values of Connection Parameters to a Multilayered Neural Network
Issue Date:1996-02-06
•22 Claims
•7 Drawing Sheets
Abstract
A generator in a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled the input and output layers forms initial values for connection parameters. A first generator produces an initial value W10 of a weight coefficient of each connection parameter of the intermediate layer from in-class covariant data SW and inter-class co-variant data SB over data inputted to the input layer. The produced values are set into respective synapses of the intermediate layer as connection parameters.
Metadata
Inventor
- Toru Niki
Application Information
Application Number:US 2869144
Filing Date:1994-08-08
Priority Date:1989-05-22
Art Unit:239
Classifications
IPC:
G06F 1518
Field of Search:
39522;23;21
Patent Drawings (7 sheets)
Description
Background of the Invention
The present invention relates to a neural network which can be used, for example, for character recognition and, in particular, to a method of assigning initial values for learning the connection parameters of a neural network by utilizing statistical information on input data.
For example, as shown in FIG. 1, in a back propagation type neural network, synapse elements of an input layer, an intermediate layer and an output layer are generally connected in a network form. The connection between synapse elements is represented by synapse parameters (or a weight coefficient W and a bias .theta.). Adjusting these parameters to suitable values is called learning. If the network is used for character recognition, learning is achieved using learning data for character recognition. If it is used for picture image processing, learning is achieved using learning data for picture image processing; if such learning is sufficient to determine parameters appropriately and synapse parameters are set to these determined values, thereafter, this neural network performs character recognition, picture image processing and the like in accordance with the learning.
In a neural network, therefore, setting the above-mentioned synapse parameters suitably, i.e., setting efficiently and suitably, is very important.
In learning of a neural network, a back propagation method is generally performed. In the learning of a weight coefficient W and a bias .theta. of the synapse elements of this back propagation type neural network, regarding the weight coefficient W and bias .theta., correction is made to the weight coefficient W (n) and bias .theta.(n) beginning with initial values W(O) and .theta.(O) in the learning process. However, there are at present no theoretical grounds for preferring any particular set of initial values as the starting point for the learning of the weight coefficient W and bias .theta.. Under the existing circumstances, a small random number is generally used for each element as the initial value.
The problems regarding a weight coefficient that arise in a conventional learning method will be explained in detail.
In a back propagation algorithm, a weight coefficient w.sub.ij between two synapse elements i, j is successively corrected by using a correction amount .increment.w.sub.ij . That is,
In the above equation, i denotes elements of the input side; j denotes elements of the output side. The correction amount .increment.w.sub.ij is calculated from the following equation. ##EQU1## where .eta. is a positive constant and E is an error of the entire network given by the following equation. ##EQU2## where t.sub.j is a teacher signal, and y.sub.j is the output signal of an output-side synapse element. .differential.g/.differential.w.sub.ij is calculated using outputs of each layer, but the explanation of a concrete calculation method is omitted.
The correction of a weight coefficient W.sub.ij (this step is called learning) is performed as described above and it is expected that the weight coefficient will converge on a suitable value through repetition of the learning steps. At this time, to make the value converge on a suitable value quickly, it is important to start with the right initial value w.sub.ij (O).
The learning of bias .theta.j for an element j is performed by means of a learning process similarly to w.sub.ij such that the bias .theta.j is regarded as a weight coefficient for input data that may be assigned a "1" at any time as a value.
As described above, since a back propagation type neural network is dependent on the initial values mentioned above regarding a weight coefficient and a bias in the learning process, the above-mentioned conventional technology, in which random numbers are used as initial values for a weight coefficient W={w.sub.i,j } and a bias .theta..dbd.{.theta..sub.j }, entails the possibility that (1) the learning time will be inordinately long and (2) a parameter will fall into a minimum value which is not optimum in the midway of learning and this minimum value will be erroneously taken as a suitable value.
Summary of the Invention
According to the present invention, in a back propagation type neural network having the three layers --an input layer, an intermediate layer and an output layer, a calculation means is provided for calculating a weight coefficient matrix of an intermediate layer from a in-class covariant matrix and a inter-class covariant matrix of input data of an input layer and initial values of a weight coefficient matrix of an intermediate layer are determined from the input data.
According to the present invention, in a back propagation type neural network having the three layers--an input layer, an intermediate layer, and an output layer, a calculation means is provided for calculating a vector representing the bias of an intermediate layer from a total average vector of input data of an input layer and a weight coefficient matrix of an intermediate layer and an initial value of a vector representing the bias of an intermediate layer is determined from the input data.
According to the present invention, in a back propagation type neural network having three layers--an input layer, an intermediate layer, and an output layer, a calculation means is provided for calculating the weight coefficient matrix of an output layer from an average vector of each class of input data of an input layer, a weight coefficient matrix of an intermediate layer and a vector representing the bias of an intermediate layer and initial values of a weight coefficient matrix of an output layer are determined from the input data.
According to the present invention, in a back propagation type neural network having three layers--an input layer, an intermediate layer, and an output layer, a calculation means is provided for calculating a vector representing the bias of an output layer from a weight coefficient matrix of an output layer and an initial value of the bias of the output layer is determined from the input data.
Other objects and advantages besides those discussed above shall be apparent to those skilled in the art from the description of a preferred embodiment of the invention which follows. In the description, reference is made to accompanying drawings, which form a part hereof, and which illustrate an example of the invention. Such example, however, is not exhaustive of the various embodiments of the invention, and therefore reference is made to the claims which follow the description for determining the scope of the invention.
Brief Description of the Drawings
FIG. 1 is a block diagram showing the configuration of the neural network in one embodiment of the present invention;
FIG. 2 is a flowchart showing an initial value determination process;
FIG. 3 is a view showing the distribution of input data x;
FIG. 4 is a view showing the distribution of vector W.sub.1.x;
FIG. 5 is a view showing the distribution of vector W.sub.1.x-.theta..sub.1 ;
FIG. 6 is a view showing the distribution of vector f(W.sub.1.x-.theta..sub.1); and
FIG. 7 is an view for explaining a method of determining W.sub.20 (K) and bias .theta..sub.20 (K) .
Description of the Preferred Embodiments
One embodiment of the present invention will be explained in detail hereinunder with reference to the drawings. This embodiment is made up of an input layer, intermediate layers, and an output layer, in this embodiment, one layer is formed as an example of intermediate layers.
FIG. 1 is a block diagram of a neural network of this embodiment. In this figure, the neural network consists of one input layer 1, one intermediate layer 2, and one output layer 3.
Where the parameters of the synapse elements of this network are suitably set, input data X is received by the synapses of the input layer 1 and sent to the synapses of the intermediate layer 2. The intermediate layer 2 multiplies the input data X sent from the input layer 1 by a weight coefficient W.sub.1, adds a bias .theta..sub.1 to operate a transfer function (in this embodiment, a sigmoid function is selected as a transfer function), and outputs the resulting data Y to the output layer 3. The output layer 3 multiplies the input data Y sent from the input layer 2 by a weight coefficient W.sub.2, add a bias .theta..sub.2 to operate a transfer function, and outputs the resulting data Z.
The above is the explanation of the flow of data in general data processing, after learning, by the neural network of this embodiment.
Shown in FIG. 1 are weight coefficient calculation sections 4 and 6 for calculating weight coefficient matrices W.sub.1 and W.sub.2, respectively, in a learning process. Also shown are bias calculation sections 5 and 7 for calculating biases .theta..sub.1 and .theta..sub.2, respectively. The method of calculating the learning values of W.sub.1, W.sub.2, .theta..sub.1 and .theta..sub.2 by these calculation sections follows a back propagation method. The method is well known and therefore an explanation thereof is omitted. Next, a method of assigning the initial values of these parameters W.sub.10, W.sub.20, .theta..sub.10 and .theta..sub.20, a feature of this invention, will be explained.
The procedure of assigning initial values in this embodiment will be explained with the flowchart in FIG. 2 with reference to FIG. 1. In this embodiment, data input for learning are represented by x, y and z to discriminate the above-mentioned X, Y and Z. From the statistical properties of the learning data x, the calculation sections 4, 5, 6 and 7 assign initial values W.sub.10, W.sub.20, .theta..sub.10 and .theta..sub.20. Because the learning data x, y, z are represented as vectors, these are also called vectors x, y, z in this specification.
First, in step S1, the weight coefficient calculation section 4 calculates the initial value W.sub.10 for a weight coefficient matrix of the intermediate layer from the in-class covariant matrix and the inter-class covariant matrix of the input data x. Next, in step S2, the bias calculation section 5 calculates the bias initial value .theta..sub.10 from the total average vector x.sub.tot of input data and the weight coefficient matrix W.sub.10 of the intermediate layer. The indicates an average value. Next, in step S3, the weight coefficient matrix calculation section 6 calculates the initial value W.sub.20 of the weight coefficient matrix of the output layer from an average vector x.sup.(k) of each class of input data, the weight coefficient matrix W.sub.10 of the intermediate layer and the bias .theta..sub.10 of the intermediate layer. In step S4, the bias calculation section 7 calculates the bias .theta..sub.20 of the output layer from the weight coefficient matrix W.sub.20 of the output layer.
Each calculation in the above procedure will be explained more in detail below using equations.
When the neural network of the present invention is used, for example, in a character recognition apparatus, input data becomes a feature vector representing the feature of a character, extracted from a character image. All input data is represented by an M dimensional vector and it is assumed that the data belongs to any one of K sets of classes.
If the number of data contained in each class is set at n and the j-th input data of class k (k=1, 2, . . . , K) is represented by x.sub.j.sup.(k) the following statistical quantities can be calculated from the distribution of input data; namely, an input vector average x(k) of each class, an average value x.sub.tot over all the classes, a covariant matrix S.sup.(k) of each class, an in-class covariant matrix S.sub.W, and an inter-class covariant matrix S.sub.B. ##EQU3## In the above equations, T in ().sup.T represents a transposition.
A signal processing in a back propagation type neural network made up of three layers--the input layer 1, the intermediate layer 2, and the output layer 3--can be represented by the following equations if the dimension of the input vector x is set at M, the number of elements of the intermediate layer, at N, and the number of classes to be classified, at K. That is,
where
x: input data (X .epsilon.R.sup.M)
y: output from intermediate layer (y .epsilon.R.sup.N)
z: output from output layer (z .epsilon.R.sup.K)
W.sub.10 : weight coefficient matrix of intermediate layer (N x M matrix)
.theta..sub.10 : bias of intermediate layer (.theta..sub.10 .epsilon.R.sup.N)
W.sub.20 : weight coefficient matrix of output layer (K x N matrix)
.theta..sub.20 : bias of output layer (.theta..sub.20 .epsilon.R.sup.K).
f indicates that a sigmoid function, ##EQU4## is operated on the components of the vector.
As learning is performed according to a back propagation algorithm and the values of the weight coefficient W and the bias .theta. have come to converge on an ideal value, an output such that ##EQU5## is made from the j-th synapse element of the output layer to the input vector x.sup.(k) belonging to class k.
In the present invention, to assign the initial value W.sub.10 of the weight coefficient of the intermediate layer, "The method of linear discriminant mapping" is used. (For example, refer to "Handbook of Multivariate Analysis", by Yanai and Takagi, Gendai Suugaskusha.)
Linear discrimination mapping intends to select, as the evaluation criterion for discrimination, a linear mapping A (y=A. x: x .epsilon.R.sup.M, y .epsilon.R.sup.N) such that "the inter-class covariation becomes as large as possible and, at the same time, the in-class covariation becomes as small as possible".
When this method of linear discrimination mapping is applied to determine the initial value W.sub.10 of a weight coefficient of the intermediate layer, a problem of finding a mapping A such that an inter-class covariant S.sub.B becomes maximum while keeping an in-class covariant S.sub.W constant, reduces to an eigen value problem of finding an eigen value .lambda..sub.i and an engine vector a.sub.i that satisfy the relation
When the eigen value .lambda..sub.i and the eigen vector a.sub.i are found by solving an eigen problem, an eigen vector matrix, in which eigen vectors (a.sub.1, a.sub.2, . . . , a.sub.N) corresponding to N pieces of (.lambda..sub.i, .lambda..sub.2, . . . , .lambda..sub.N) that are taken from such an eigen value .lambda..sub.i in a descending order, becomes a linear discrimination mapping A to be determined. That is,
A transposed matrix of this eigen vector matrix is given as the weight coefficient matrix W.sub.10 of the intermediate layer. That is,
The above results will be explained in detail using FIGS. 3 and 4.
The distribution of x when the dimension M of the input vector x is set at 2 and the classes are set at 3 of k.sub.1, k.sub.2 and k.sub.3, is shown in FIG. 3. As shown in FIG. 3, for example, it is assumed that an input vector of each class has a tendency such that it is distributed in a lateral direction in classes k.sub.1 and k.sub.2. Thus, if, keeping the inclass covariant S.sub.W constant, a transposed matrix of a mapping A such that the inter-class covariant S.sub.B becomes maximum is defined to be the initial value W.sub.10 of the weight coefficient of the intermediate layer, the input vector of each class shows the distribution shown in FIG. 4 by the mapping of W.sub.10 x. That is, a variance is maximum among input vectors in the different classes by the mapping of W.sub.10. x and the extension of the variance is small among the input vectors in the same class. In other words, if W.sub.10 (=A.sup.T) is taken as the initial value of a weight coefficient matrix, convergence in a learning process is quickened.
Assignment of the initial value .theta..sub.10 of a bias is explained next.
where x.sub.tot is a total average vector obtained by taking the average of vector x over all the classes. FIG. 4 shows W.sub.10 x.sub.tot by mapping W.sub.10. The assignment of .theta..sub.10 as in equation (16) causes the distribution of the vector obtained by the conversion of W.sub.10 x-.theta..sub.10 to the input vector x to become as shown in FIG. 5, and the center of the distribution moves to the origin.
Further, since the distribution of the output y=f(W.sub.10 x-.theta..sub.10) from the intermediate layer is obtained by operating the sigmoid function of equation (12) on the components of the vector, the region of the values of each component is suppressed to a section [-1/2].
Next, a method of assigning the initial value W.sub.20 of the weight coefficient matrix of the output layer and the bias .theta..sub.20 will be explained.
Among the synapse elements of the output layer, element Zk is assumed to output 1/2 for the input of data belonging to solely class k, and output -1/2 for the other inputs. And if the initial value of the weight coefficient vector for the Zk is denoted by W.sub.20.sup.(k) and the initial value of the bias is denoted by .theta..sub.20.sup.(k). Then the output zk from such elements becomes as follows:
T represents transposition.
The set of vectors that satisfy (W.sub.20.sup.(k)T.sub..y-.theta..sub.20.sup.(k))=0 represents a hyperplane that includes f(W.sub.10 x.sup.(k) -.theta..sub.10), which is an image of an average vector x.sup.(k) of class k on the positive area side and includes the images of the other classes on the negative area side. To be specific, in FIG. 7, the set of f(W.sub.10 x.sup.(k) -.theta..sub.10) for the input vector x of class k.sub.1 is included in a positive area and the set of f(W.sub.10 x.sup.(k) -.theta..sub.10) for the input vector x of classes k.sub.2 and k.sub.3 (.noteq.k.sub.1) is included in a negative area.
Thus, the following are defined:
.alpha.is a positive constant less than 1 and, for example, may be set to 0.5.
In such a case, as shown in FIG. 7, the set of vectors that satisfy z.sub.k =0 passes a midpoint between an image y.sup.(k) of x.sup.(k) and the origin and represents a K-1 dimensional hyperplane perpendicular to y.sup.(k).
Thus, those in which initial values W.sub.20.sup.(k) and .theta..sub.20.sup.(k) of a parameter for class k are arrayed become a matrix W.sub.20 representing the initial values of the weight coefficient of the output layer and a matrix .theta..sub.20 representing the bias initial value of the output layer, respectively. That is,
From the above procedure, W.sub.10, .theta..sub.10, W.sub.20, and .theta..sub.20 are determined.
The initial values obtained from the above-mentioned method are good estimate values obtained by the theory of linear mapping. Ordinary input data is nonlinear. However, input data handled by the neural network of the above-mentioned embodiment is thought to have a small non-linearity and therefore they are values more proper than those obtained by a conventional method of adopting a random number as an initial value.
Various modifications are possible within the scope not departing from the spirit of the present invention.
For example, a suitable normalization may be performed on the weight coefficient matrices W10 and W20 and biases .theta..sub.10 and .theta..sub.20 used in the above-mentioned embodiment to change the norm.
A transfer function f is not limited to the format of equation (12). For example, any function may be used if it is differentiable, monotone increasing and a suppression type.
Some of W.sub.10, .theta..sub.10, W.sub.20, and .theta..sub.20 may be assigned with a random number as in the prior art rather than all values of these being determined according to the above-mentioned embodiment.
Accordingly, in the neural network of the present invention, estimate values obtained on the basis of statistical information (average value, covariant matrix) of input data are adopted in place of a conventional method of assigning a random number as an initial value of a weight coefficient of each element and/or an initial value of a bias. As a result, learning time is considerably shortened over the conventional art which uses a random number as an initial value, and the advantage can be obtained that the possibility of falling to a minimum value which is not optimum is lessened.
The present invention is not limited to the above embodiments and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, to apprise the public of the scope of the present invention the following claims are made.
Claims
What is claimed is:
1. An apparatus for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have similar characteristics, said apparatus comprising a neural network comprising an input layer, an output layer, and an intermediate layer coupled between the input layer and the output layer, and determining means for determining connection parameters between the layers of said neural network by using training data prior to classifying the input data, the connection parameters being represented at least by a weight coefficient, wherein the input layer includes means for inputting data; the intermediate layer includes means for receiving data from the input layer and processing the received data by means of the connection parameters; the output layer includes means for receiving data processed by the intermediate layer, processing the received data by means of the connection parameters, and outputting signals indicating one of the predetermined k classes; and the determining means comprises: a first generating means for generating initial value data W.sub.10 of the weight coefficient of each connection parameter of the intermediate layer from in-class dispersion data S.sub.W and inter-class dispersion data S.sub.B of the training data over all classes into which the training data has been partitioned, said in-class dispersion data S.sub.W representing a dispersion degree of the training data within a class and said inter-class dispersion data S.sub.B representing a dispersion degree of the training data over classes; setting means for setting the generated initial value data into the respective synapses of the intermediate layer, as connection parameters; and correction means for correcting the connection parameters by using a back-propagation learning method.
2. The apparatus according to claim 1, wherein said first generating means calculates the matrix data of said inclass covariant S.sub.W and the matrix data of interclass covariant S.sub.B when input data is represented as a vector, calculates the data of a matrix representing a mapping A such that the inter-class covariant matrix data S.sub.B becomes maximum while keeping the in-class covariant matrix data S.sub.W constant and calculates said initial value data W.sub.10 by calculating the transposed matrix of the calculated matrix representing the mapping A.
3. A method for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have a similar characteristic, by using a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled between the input and output layers, each layer comprising a network of synapse elements and connections between the synapse elements, each connection being represented by a connection parameter which includes at least a weight coefficient, the method comprising the steps of: (a) partitioning a plurality of training data x into k classes; (b) generating data representing an in-class dispersion matrix S.sub.W and data representing an inter-class dispersion matrix S.sub.B of the training data over all classes into which the training data has been partitioned, each class designating a set in which a plurality of training data which have similar characteristics are included, said in-class dispersion data S.sub.W representing a dispersion degree of the training data within a class and said inter-class dispersion data S.sub.B representing a dispersion degree of the training data over classes on the basis of the plurality of input data; (c) determining initial values W.sub.10 of the weight coefficients of the synapses of the intermediate layer on the basis of the data representing said inter-class dispersion matrix S.sub.W and said in-class dispersion matrix S.sub.B ; and (d) determining final values of said weight coefficients of the synapses of the intermediate layer beginning with the initial values W.sub.10 according to back propagation type learning method in which training is input via said input layer, and setting connection parameters including the determined final weight coefficients into the respective synapses of the intermediate layer; (e) inputting input data to the input layer; (f) processing the input data through the intermediate layer having the determined weight coefficients; and (g) further processing the data processed through the intermediate layer by means of the output layer, and outputting from the output layer signals indicating one of the predetermined k classes to which the input data belongs.
4. The learning method according to claim 3, wherein said step (c) further comprises the steps of: (c1) forming a mapping A, such that the inter-class covariant matrix S.sub.B becomes maximum while keeping said in-class covariant matrix S.sub.W constant; and (c2) generating said initial value W.sub.10 by calculating a transposed matrix of a matrix representing the mapping A.
5. An apparatus for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have similar characteristics, said apparatus comprising: a back propagation type neural network including an input layer, an output layer, and an intermediate layer coupled between the input layer and the output layer; and determining means for determining connection parameters between the layers of said neural network by using training data prior to classifying the input data, each connection parameter being represented by a weight coefficient and a bias .theta., wherein the neural network input layer includes means for inputting data; the neural network intermediate layer includes means for receiving data from the input layer and processing the received data by means of the connection parameters; the neural network output layer includes means for receiving data processed by the intermediate layer, processing the received data by means of the connection parameters and outputting signals indicating one of the predetermined k classes; and the determining means comprises: generating means for generating data representing an initial value .theta..sub.10 of the bias of each connection parameter of the intermediate layer from a total average x of input data x to be input to synapses of the input layer and a weight coefficient W.sub.10 of each connection parameter of the intermediate layer; and setting means for setting the generated initial values .theta..sub.10 and W.sub.10 into the respective synapses of the intermediate layer, as beginning connection parameters for the back propagation type neural network.
6. The apparatus according to claim 5, wherein said generating means generates a vector representing the bias .theta..sub.10 of the intermediate layer on the basis of .theta..sub.10 =W.sub.10 x.sub.tot wherein x.sub.tot denotes a total average vector of the input data x.
7. The apparatus according to claim 5, wherein said network accepts input data x as vector input, further includes a third generating means for forming the initial value W.sub.10 of the weight coefficient of the intermediate layer from the in-class covariant matrix S.sub.W and the inter-class covariant matrix S.sub.B of the vector x, and said second generating means generates the bias .theta..sub.10 of said intermediate layer by using the initial value of the weight coefficient of the intermediate layer formed by the third generating means.
8. A method for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have a similar characteristic., by using a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled between the input and output layers, each layer comprising a network of synapse elements and connections between the synapse elements, each connection being represented by a connection parameter which includes at least a weight coefficient and a bias .theta., the method comprising the steps of: (a) accepting plural input data x as vectors; (b) generating an average vector x.sub.tot of all vectors x by performing a statistical processing on input data; (c) forming an initial value .theta..sub.10 of the intermediate layer from the average vector x.sub.tot and the weight coefficient W.sub.10 of the intermediate layer; (d) determining said connection parameters beginning with the initial value .theta..sub.10 according to a back propagation method; (e) inputting input data to the input layer; (f) processing the input data through the intermediate layer having the determined weight coefficients; and (g) further processing the data processed through the intermediate layer by means of the output layer, and outputting from the output layer signals indicating one of the predetermined k classes to which the input data belongs.
9. The method according to claim 8, wherein said step (c) comprises generating a vector representing the bias .theta..sub.10 of the intermediate layer on the basis of .theta..sub.10 =W.sub.10 x.sub.tot wherein a total average vector of the input data x is denoted by x.sub.tot.
10. The method according to claim 8, wherein said step (a) accepts the input data x as a vector, further including a step for generating an initial value W.sub.10 of the weight coefficient of the intermediate layer from an in-class covariant matrix S.sub.W and an inter-class covariant matrix S.sub.B of vector x, and a step of generating the initial value .theta..sub.10 of the bias of the intermediate layer from the initial value W.sub.10 and the average vector x.sub.tot.
11. An apparatus for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have similar characteristics, said apparatus comprising: a back propagation type neural network including an input layer, an output layer and an intermediate layer coupled between the input layer and the output layer; and determining means for determining connection parameters between the layers of said neural network by using training data prior to classifying the input data, the connection parameters being represented by at least a weight coefficient, wherein, the neural network input layer includes means for inputting data; the neural network intermediate layer includes means for receiving data from the input layer and processing the received data by means of the connection parameters; the neural network output layer includes means for receiving data processed by the intermediate layer, processing the received data by means of the connection parameters and outputting signals indicating one of the predetermined k classes; and the determining means comprises generating means for generating data representing a weight coefficient matrix W.sub.20 of the output layer from data representing an average vector x.sup.(k) of each class of input data of the input layer, an initial value W.sub.10 of a weight coefficient matrix of the intermediate layer, and a bias .theta..sub.10 representing an initial value of a bias of the intermediate layer as beginning values for the back propagation type neural network.
12. The apparatus according to claim 11, wherein said generating means comprises means for generating an average vector x.sup.(k) of each class k when the input data is input as a vector, class by class; means for generating a weight coefficient vector W.sub.20.sup.(k) of each class in accordance with W.sub.20.sup.(k) =f(W.sub.10 x.sup.(k) -.theta..sub.10), where f is a transfer function; and means for generating the weight coefficient matrix W.sub.20 of the output layer by combining the weight coefficient vector W.sub.20.sup.(k) of each class.
13. The apparatus according to claim 11, wherein said generating means further includes: means for generating an in-class covariant matrix S.sub.w and an inter-class covariant matrix S.sub.B from said input vector x; means for generating a mapping A such that the inter-class covariant matrix S.sub.B becomes maximum while keeping the in-class covariant matrix constant; means for generating said initial value W.sub.10 by forming a transposed matrix of a matrix representing this mapping A, and means for generating the bias .theta..sub.10 of the intermediate layer from the total average value x.sub.tot of the input data x and the weight coefficient W.sub.10 of the intermediate layer.
14. A method for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have a similar characteristic, by using a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled between the input and output layers, each layer comprising a network of synapse elements and connections between the synapse elements, each connection being represented by a connection parameter which includes at least a weight coefficient and a bias .theta., the method comprising the steps of: (a) partitioning input data x to classes as vectors; (b) generating an average vector x.sup.(k) for each class of input data of the input layer, an initial value W.sub.10 of a weight coefficient of the intermediate layer, and a vector .theta..sub.10 representing an initial value of a bias of the intermediate layer by performing a statistical process on the input data; (c) generating a weight coefficient matrix W.sub.20 of the output layer from the average vector x.sup.(k) the initial value W.sub.10 of a weight coefficient matrix of the intermediate layer, and the initial value vector .theta..sub.10 of the bias of the intermediate layer; (d) determining the connection parameters beginning with these initial values according to a back propagation method; (e) inputting input data to the input layer; (f) processing the input data through the intermediate layer having the determined weight coefficients; and (g) further processing the data processed through the intermediate layer by means of the output layer, and outputting from the output layer signals indicating one of the predetermined k classes to which the input data belongs.
15. The method according to claim 14, wherein said step (c) further comprises the steps of (c1) forming an average vector x.sup.(k) for each class when input data is input class k by class k as a vector (c2) forming the weight coefficient vector W.sub.20.sup.(k) of each class in accordance with from the average vector x.sup.(k) and said initial values W.sub.10 and .theta..sub.10, where f is a transfer function; and (c3) combining the weight coefficient vector W.sub.20.sup.(k) of each class to generate the weight coefficient matrix W.sub.20 of the output layer.
16. The method according to claim 14, wherein said step (b) further comprises the steps of: (b1) forming an in-class covariant matrix S.sub.W and an inter-class covariant matrix S.sub.B from said input vector x; (b2) forming a mapping A such that the inter-class covariant matrix S.sub.B becomes maximum while keeping the in-class covariant matrix S.sub.W constant and generating said initial value W.sub.10 by calculating a transposed matrix of a matrix representing this mapping A; and (b3) forming the bias .theta..sub.10 of the intermediate layer from the total average value x.sub.tot of input data x and the weight coefficient W.sub.10 of the intermediate layer.
17. An apparatus for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have similar characteristics, said apparatus comprising: a back propagation type neural network including an input layer, an output layer, and an intermediate layer coupled between the input layer and the output layer; and determining means for determining connection parameters between the layers of said neural network by using training data prior to classifying the input data, the connection parameters being represented at least by a bias, wherein, the neural network input layer includes means for inputting data; the neural network intermediate layer includes means for receiving data from the input layer and processing the received data by means of the connection parameters; the neural network output layer includes means for receiving data processed by the intermediate layer, processing the received data by means of the connection parameters and outputting signals indicating one of the predetermined k classes; and the determining means comprises means for generating a vector .theta..sub.20 representing an initial value of a bias of the output layer from an initial value W.sub.20 of a weight coefficient matrix of the output layer as a beginning value for the back propagation type neural network.
18. The apparatus according to claim 17, wherein said generating means comprises: means for forming the average vector x.sup.(k) of each class when input data is input class k by class k as a vector; means for calculating the weight coefficient vector W.sub.20.sup.(k) of each class in accordance with W.sub.20.sup.(k) =f(W.sub.10 x.sup.(k) -.theta..sub.10), from the average vector x.sup.(k) and said initial values W.sub.10 and .theta..sub.10 where f is a transfer function; means for calculating the bias vector .theta..sub.20.sup.(k) of each class in accordance with .theta..sub.20.sup.(k) =.alpha..parallel.W.sub.20.sup.(k) .parallel.from said weight coefficient vector W.sub.20.sup.(k) of each class and a predetermined constant .alpha.; and means for combining the bias vector .theta..sub.20 (k) of each class to generate the bias matrix .theta..sub.20 of the output layer.
19. The apparatus according to claim 17, wherein said generating means further comprises: means for forming an in-class covariant matrix S.sub.B and an inter-class covariant matrix from an input vector x; means for forming a mapping A such that the inter-class covariant matrix S.sub.B becomes maximum while keeping the in-class covariant matrix S.sub.B constant; and means for generating said initial value W.sub.10 by calculating the transposed matrix of the mapping A; means for calculating a bias .theta..sub.10 of the intermediate layer from a total average value x.sub.tot of input data x and the weight coefficient w.sub.10 of the intermediate layer, and means for forming the weight coefficient matrix W.sub.20 of the output layer from the average vector x.sup.(k) of each class of input data of the input layer, an initial value W.sub.10 of a weight coefficient matrix of the intermediate layer, and a vector .theta..sub.10 representing the initial value of a bias of the intermediate layer.
20. A method for classifying input data representing a physical quantity into predetermined k classes, each class designating a set including a plurality of data which have a similar characteristic, by using a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled between the input and output layers, each layer comprising a network of synapse elements and connections between the synapse elements, each connection being represented by a connection parameter which includes at least a weight coefficient and a bias .theta., the method comprising the steps of: (a) partitioning input data x for classes as vectors; (b) generating an initial value W.sub.20 of a weight coefficient of the output layer by performing a statistical process on the input data; (c) generating an initial value .theta..sub.20 of a vector representing a bias of the output layer from the initial value W20 of the weight coefficient of the output layer; (d) determining the connection parameters beginning with these initial values according to a back propagation method; (e) inputting input data to the input layer; (f) processing the input data through the intermediate layer having the determined weight coefficients; and (g) further processing the data processed through the intermediate layer by means of the output layer, and outputting from the output layer signals indicating one of the predetermined k classes to which the input data belongs.
21. The method according to claim 20, wherein said step (c) further comprises the steps of: (c1) forming an average vector x.sup.(k) of each class when input data is input class k by class k as a vector (c2) forming a weight coefficient vector W.sub.20.sup.(k) of each class in accordance with W20=f(W.sub.10 x.sup.(k) -.theta..sub.10) from the average vector x.sup.(k) and said initial values W.sub.10 and .theta..sub.10, where f is a transfer function; (c3) forming a bias vector .theta..sub.20 of each class in accordance with .theta..sub.20.sup.(k) =.alpha..parallel.W.sub.20.sup.(k) .parallel. from said weight coefficient vector W.sub.20 (k) of each class and a predetermined constant .alpha.; and (c4) combining the bias vector .theta..sub.20 (k) of each class to calculate an initial value .theta..sub.20 of a bias matrix of the output layer.
22. The method according to claim 20, wherein said step (b) further comprises the steps of: (b1) forming an in-class covariant matrix S.sub.W and an inter-class covariant matrix S.sub.B from said input vector x; (b2) forming a mapping A such that the inter-class covariant matrix covariant matrix SB.sup.becomes maximum while keeping the in-class covariant matrix S.sub.W constant; (b3) forming an initial value W.sub.10 by calculating a transposition matrix of a matrix representing the mapping A; (b4) forming the bias .theta..sub.10 of the intermediate layer from a total average value x.sub.tot of input data x and the weight coefficient W.sub.10 of the intermediate layer; and (b5) forming the weight coefficient matrix W.sub.20 of the output layer from the average vector x.sup.(k) of each of input data of the input layer, the initial value W.sub.10 of a weight coefficient matrix of the intermediate layer, and the vector .theta..sub.10 representing the initial value of a bias of the intermediate layer.
Patent Citations (5)
Non-Patent Literature (5)
- Rumelhart et al., Parallel Distributed Processing, vol. 1, Foundations, MIT Press, 1986, pp. 318-362.
- White, H., "Learning in Artificial Neural Networks: A Statistical Perspective", Neural Computation 1, 1989, pp. 425-464.
- Lari-Najafi et al., "Effect of Initial Weights on Back-Propagation and its variations," 1989 IEEE Intl. Conf. on Systems, Man and Cybernetics, 14-17 Nov. 1989, 218-219.
- Yau et al., "Sigma-Pi Implementation of a Gaussian classifier," IJCNN '90, Jun. 1990, III-825-III-830.
- Partial translation of "Multivariate Analysis Handbook", Haruo Yanai et al. (1986) pp. 132-144.