Programmable Analog Neural Network

The present invention concerns a programmable analogue neural network.

Neural networks, in the techniques developed in the field of artificial intelligence, designate electric circuits capable of performing transformations on vectors having a large number of components. Numerous examples of realisation of these circuits are known, for example, the description of the principle in the article by J. J. Hopfield and D. W. Tank entitled "Neural computation of decision in optimization problems", published in the journal Biological Cybernetics 52, pp 141 to 152 (1985), or the article by H. Graf and P. de Vegvar "a CMOS Associative memory chip based on Neural Networks" published in the proceeding of the ISCC 1987.

The object of these networks is to perform operations of the form:

where [V.sub.e ] is an input vector with N components V.sub.ei such that V.sub.ei .epsilon.{-1+1}, [V.sub.s ] is an output vector with M components V.sub.sj such that V.sub.sj .epsilon.{-1,+1} and where [A.sub.ij ] is a matrix of N.times.M positive or negative coefficients a.sub.ij.

1. [A.sub.ij ] designates the operation which for an input vector [V.sub.e ] provides a corresponding output vector [V.sub.s ] such that: ##EQU1## i.e. a scalar product of two vectors, with the convention that

The coefficients a.sub.ij are called "synaptic coefficients" or "synaptic weights".

The advantage of performing the operation a.sub.ij * V.sub.ei analogically in accordance with the circuit diagram in FIG. 1 has long been known. In the circuit in FIG. 1, n input terminals B1 to Bn are connected to a common output S.sub.j each via a resistor of value 1/apj with p having the value of 1 to n respectively. The input signals on terminals B1 to Bn are respectively V.sub.e1 to V.sub.en. At the output S we get ##EQU2##

The resistors shown in this figure and in the following figures do not necessarily represent a linear resistance but all sorts of loads used in microelectronics, notably in CMOS techniques, for transistors whose grid is connected to the source or to the drain. It should be noted that if the common point is loaded at infinite impedance, the addition is performed in voltage, whereas if the common point is loaded at zero impedance, the addition is performed in current.

As only a rough part of the information will be retained, -1 or +1, very great precision is not necessary, on the other hand, the advantage of rapidity arising from the structure shown in FIG. 1 is interesting: the characteristic of analogue addition is that it enables all the terms of the sum to be added in parallel and not sequentially as in digital devices.

In addition, it has been shown by E. de Chambost, in a communication at n'EURO'88, "Linear classification with binary connection networks", in Paris in June 1988, that neural networks retained most of their properties if the resolution of each synaptic coefficient was reduced to a single bit of information, i.e. if two single values of a.sub.ij were permitted: -1 or +1.

FIGS. 2, 3, 4, 5 and 6, described below, concern this particular case which is particularly interesting for the construction of programmable analogue neural networks: a single binary cell is assigned to a coefficient a.sub.ij.

In the following part of the description, we work on the assumption that the operations of writing new tables of coefficients a.sub.ij are much less frequent than the operations of loading a new input vector [V.sub.e ].

The solution consisting in physically constructing the resistors in FIG. 1 on a microelectronic circuit is not advantageous, as it would require the masks used in the technological process to be modified each time the value of the synaptic coefficients needs to be modified, which would be too expensive.

Programmable Analogue Neural Networks are analogue neural networks whose synaptic coefficients can be modified electrically, in the same way as a binary word can be written in an electronic memory.

The conventional solution for the construction of programmable analogue neural networks is that represented schematically in FIGS. 2 and 3.

In FIG. 2, the rectangle of broken lines, referenced M.sub.ij, delimits a synaptic connection according to the state of the art. C.sub.ij is a memory cell constructed in accordance with known techniques. The cell C.sub.ij delivers a logical state a.sub.ij at a first output, and at a second output its complement a.sub.ij. These outputs are each connected to a column conductor Lj via an FET transistor (T.sub.1, T.sub.2 respectively) in series with a resistor (R.sub.1, R.sub.2 respectively). The grid of T.sub.1, T.sub.2 is respectively connected to a line V.sub.ei and a line V.sub.ei with a logical state complementary to that of V.sub.ei.

In FIG. 3 is shown a part of a matrix of synaptic connections such as those in FIG. 2. This matrix comprises lines V.sub.ei to V.sub.en and lines V.sub.ei to V.sub.en with respectively complementary logical states, and columns L.sub.1 to L.sub.n. At each synapse (intersection of a column with two complementary lines such as V.sub.ek and V.sub.ek) is placed a synaptic connection such as that in FIG. 2. Each column L.sub.1 to L.sub.n is connected by an operational amplifier A.sub.1 to A.sub.N to the input of a comparator C.sub.1 to C.sub.N whose other input receives an digital offset voltage V.sub.s1 to V.sub.sn, memorised in the form of a binary word, via a digital-to-analogue converter DAC.sub.1 to DAC.sub.n. The group D.sub.1 to D.sub.n of these elements connected to each column constitutes a detection device, performing signal current addition of the N input signals of the corresponding column.

"0" and "1" designate the two states found on electric circuits in binary logic, I.sub.ij being the current resulting from the elementary connection in FIG. 2, proportional to a.sub.ij * V.sub.ei, the truth table of the current I.sub.ij is as follows: ##STR1##

This truth table is isomorphic with that of multiplication in {-1, +1} ##STR2##

As it is often necessary for a neural network to offset the sum .SIGMA.a.sub.ij * V.sub.ei by a threshold VS associated with a set of coefficients (a.sub.1j, . . . a.sub.ij, . . . a.sub.nj), the complete structure of a programmable analogue network is that represented in FIG. 3. This displacement is performed in the comparators C.sub.1 to C.sub.N using the voltages V.sub.s1 to V.sub.sn.

The need to divide the lines V.sub.ei into two parts, as shown in FIG. 3, arises from the fact that a single line would enable an operation isomorphic with multiplication to be performed in the space {0, 1} and not in the space {-1, 1}.

The object of the present invention is a programmable analogue neural network enabling a multiplication truth table to be implemented in the space {-1, 1} using fewer components than in the known circuits cited above.

The programmable analogue neural network according to the invention, of the type with a Cartesian matrix with N inputs and M outputs, includes a first addition circuit connected to the N inputs, N lines and M columns, the N.times.M synapses being each constituted by a logic gate, the M columns each being connected to a column addition circuit, another input of each of these column addition circuits being connected to the output of said first addition circuit, and still another input of each of these column addition circuits being connected to a circuit producing adjustable voltage.

The present invention will be better understood on reading the detailed description of an embodiment, illustrated by the appended drawing, in which:

FIG. 1 is a diagram of a known circuit performing an analogue addition,

FIG. 2 is a general diagram of a synaptic connection according to the state of the art,

FIG. 3 is a simplified diagram of a synaptic connection matrix, according to the state of the art,

FIG. 4 is a diagram of a synaptic connection according to the present invention,

FIGS. 5a and 5b are a simplified diagram of a programmable analogue neural network according to the invention,

FIG. 6 is a diagram of a synaptic connection in EEPROM technology according to the state of the art, and

FIG. 7 is a diagram of a synaptic connection in EEPROM technology according to the invention.

FIGS. 1 to 3 have already been described above.

Throughout the present description the matrices are described as having input lines and output columns, but the roles of the lines and the columns are of course interchangeable, these designations being used only to facilitate the explanations.

On the diagram in FIG. 4 a single synaptic connection CS is shown forming part of a matrix of which only part of an input line conductor V.sub.ei and part of a column conductor L.sub.j have been shown.

The connection CS includes a memory cell C'.sub.ij constituted for example of a logic gate of type "OR" or "AND" whose terminal a.sub.ij is connected by the source-drain path of an FET transistor T1 in series with a resistor R1 to the conductor L.sub.j, the grid of T1 being connected to the conductor V.sub.ei. Thus we have saved a transistor with its drain resistance and a line conductor (V.sub.ei).

The connection CS enables the truth table of a multiplication to be implemented for the following reasons:

A synaptic connection physically performs the function: ##EQU3## where a.sub.ij .epsilon.{-1, +1}

and V.sub.ei .epsilon.{-1, +1}

If we put:

a.sub.ij =2 * a'.sub.ij -1

and V.sub.ei =2 * V'.sub.ei -1

we obtain: ##EQU4##

The 3 terms A, B, C can be implemented using logic gates simply due to the fact that:

In relation (2), the term A is a scalar product which can be implemented using a series of simple logic gates such as that shown in FIG. 4. The term B depends on the input vector but is identical for all the columns. It must be calculated for each input vector. The term C does not depend on the input vector, but only on the value of the coefficients of the corresponding column.

The matrix, shown in simplified form in FIG. 5, enables the various operations of the above-mentioned relation (2) to be performed. This matrix has N line conductors V.sub.e1 to V.sub.en and M column conductors L.sub.1 to L.sub.M. In practice, in the majority of cases M=N. At each intersection (or synapse) of the matrix, a logic gate, P.sub.11 to P.sub.MN, such as the gate in FIG. 4 is connected, which performs an operation isomorphic to a scalar product as soon as a'.sub.ij .epsilon.{0,1} and V'.sub.ei .epsilon.{0,1}. A column conductor L.sub.o is connected to each line by a resistor, R.sub.1 to R.sub.N. This column L.sub.o is connected to the inverting input of an operational amplifier A.

Each of the other columns L.sub.1 to L.sub.M is connected to a first input, E.sub.11 to E.sub.1M, of an addition device, D'.sub.1 to D'.sub.M. The output from the amplifier A is at the same time connected to a second input, E.sub.21 to E.sub.2M, of all the devices D'.sub.1 to D'.sub.M.

All the devices D'.sub.1 to D'.sub.M are identical. In FIGS. 5a and 5b we have shown in detail only the device D'.sub.j. In this device D'.sub.j the input terminal E.sub.1j is directly connected to the input of an operational amplifier A.sub.j, and the terminal E.sub.2j is connected to the same input of the amplifier via a resistor R.sub.j. The output from the amplifier A.sub.j is connected to a first input of a comparator C.sub.j, a second input of which is connected to the output of a digital-analogue converter DAC.sub.j receiving a binary word V.sub.sj.

In the matrix in FIG. 5a, the amplifier A with the column L.sub.o and the resistors R.sub.1 to R.sub.N constitute a circuit implementing the term B of the equation (2) above, i.e. performing the addition of the N input signals. The output signal from this circuit is sent in offset to each circuit D'.sub.1 to D'.sub.N (via the terminal E.sub.21 to E.sub.2N respectively).

As specified above, the different gates P.sub.11 to P.sub.MN perform the operations isomorphic to scalar products (term A of the relation (2)).

Term C of the relation (2), which depends only on the value of the coefficients affecting the input vectors, is determined for each column in the circuits D'.sub.1 to D'.sub.M, by the value V.sub.S1 to V.sub.SM (in the form of a binary word, therefore easy to modify by program, converted into an analogue value by the converters DAC.sub.1 to DAC.sub.M). This digital value converted into an analogue value constitutes an offset of the signal of each column in the comparators C.sub.1 to C.sub.M.

Of course, the coefficients 2 and 4 of the terms A, B and C are easily obtained in the circuits D'.sub.1 to D'.sub.M by corresponding adjustment of the resistors of the operational amplifiers A.sub.1 to A.sub.M.

In FIG. 6 is shown the diagram of a synaptic connection CS1 in EEPROM technology with a floating grid, according to the state of the art. The connection CS1 includes two transistors Q.sub.ij and Q.sub.ij whose control grids are connected respectively to lines V.sub.ei and V.sub.ei, whose drains are connected via a resistor, RA, RB respectively to a column L.sub.j and whose sources are connected to a reference potential V.

On the other hand, the synaptic connection CS2 in FIG. 7, according to the invention, still using EEPROM technology, needs only one line V.sub.ei and one transistor Q.sub.ij whose drain is connected via the resistor RA to the column L.sub.j and whose source is connected to the reference potential V.

Although the device of the invention can be constructed using various techniques such as SRAM-CMOS, it is more profitable to make it using EEPROM technology with a floating grid, since in this case each memory cell normally has no transistor storing the state complementary to that stored in the form of loads in the grid of the single transistor (i.e. no state a.sub.ij, see FIG. 2). The space occupied by a matrix is thus reduced by half, compared with CMOS technology.

Of course dynamic memories cannot be used to construct the matrices of the invention, since their refresh principle makes impossible any collective reading, which is necessary in neural networks.