Digital Neural Network Executed in Integrated Circuit Technology

The present invention is directed to digital neural networks executed in integrated circuit technology.

Artificial neural networks (referred to below as ANN) are proposed for the parallel processing of extremely large data sets with the goal of pattern recognition and processing (for example, of speech or images). Known ANN of this species are composed of a plurality of non-linear processor elements (neurons) that are networked with one another via variable "weighting factors".

The following non-linear transfer characteristics have been proposed for modeling the neurons, c.f. R. P. Lippmann, "An Introduction to Computing with Neural Nets", IEEE ASSP Magazine, Apr. 1987, pages 4-22:

binary decision,

linear ramp function with saturation characteristic,

sigmoidal function,

tangent-hyperbola function.

Further, a plurality of different structures for the networking of the neurons exist (for example "Hopfield net", "Hamming net", "Perceptron", c.f. R. P. Lippmann as well). A direct digital realization of large nets of this type seems impossible with the current technologies for manufacturing integrated circuits because of the plurality of neurons (>1000) that are required. The main problem is thereby that the number of connections and, thus, the number of variable weighting factors quadratically increases with the number of neurons. For example, 1,000,000 weights are required given 1000 neurons in a completely meshed network.

A few realizations of ANN having, for example, 54 or 256 neurons are disclosed in the literature for a programmable or non-programmable network, respectively, c.f. H. P. Graf, P. De Vegvar, "A CMOS Associative Memory Chip based on Neural Networks, Proc. 1987 IEEE Int. Conf. on Solid State Circuits, pages 304, 305, 437; H.P. Graf et al, "VLSI implementation of a neural network memory with several hundreds of neurons", AIP Conference Proceedings 151, "Neural Networks for Computing", pages 182 through 187, Snowbird, Utah, 1986; W. Hubbard et al, "Electronic Neural Networks", AIP Conference Proceedings 151, "Neural Networks for Computing", pages 227 through 234, Snowbird, Utah, 1986. Proposed realizations have also been disclosed in J. P. Sage, K. Thompson, R. S. Withers, "An Artificial Neural Network Integrated Circuit based on MNOS/CCD principles", AIP Conference Proceedings 151, "Neural Networks for Computing", pages 381 through 384, Snowbird, Utah, 1986. These all involve analog realizations of analog ANNs. Compared to digital realizations, they have the advantage of a significantly smaller implementation surface area. What is disadvantageous, however, is the high power consumption caused by the required resistance matrix, this power consumption opposing a realization of larger, programmable networks (with more than a few hundred neurons). The behavior and the properties of the modelings of ANNs hitherto proposed are preeminently investigated by simulation on vector computers, work stations or special processor fields. The fundamental disadvantage of this method is that the (space) parallelism inherent in the neural network is completely or partially lost in the processing of the information and, thus, the calculating time of the simulated network increases to such orders of magnitude, particularly for large neuron composites, that a speedy or quasi-real-time processing of the jobs cited above is impeded or becomes impossible.

A significant shortening of the calculating time and a far greater pattern throughput is obtained, by contrast, with hardware emulators. In contrast to hardware simulators, they contain an artificial neural network having a small size with whose assistance a larger network can be emulated.

In addition to a higher throughput, the advantage of a digital realization of an ANN is comprised in the greater freedom with which the user can select and set network-typical parameters (for example, discriminator functions).

The demands that users currently make of the degree of complexity (>1000) for general neural networks currently exceed the possibilities of present VLSI integration technology.

In addition to these general, relatively large and complex neural networks, special solutions for specific jobs having smaller neural networks have already been proposed. These proposed, smaller networks are usually realized as complete solutions on a chip, being realized in analog technology with the disadvantages connected therewith.

The object of the present invention is to create digital neural networks executed in integrated circuit technology that are fashioned with a relatively small plurality of neurons that are completely networked with one another, whereby the limitations established in the known, analog solutions are to be accepted. As stated in the object, a circuit technology that exhibits a low surface area requirement and an adequately short cycle time should be employed.

The object underlying the invention is achieved by a digital neural network executed in integrated circuit technology according to the present invention.

The invention shall be set forth in detail with reference to a number of figures below.

FIG. 1 shows the basic structure of a neural network (Hopfield model) comprising four neurons;

FIG. 2 shows the block circuit diagram of a neural element having binary (0,+1) evaluation;

FIG. 3 shows the block circuit diagram of a neural element having ternary (-1, 0,+1) evaluation;

FIG. 4 shows the block circuit diagram of a neural element having expanded evaluation;

FIG. 5 shows the block circuit diagram of a neural network composed of a single neuron having serial processing;

FIG. 6 shows the block circuit diagram of a neural network having serial accumulation in the neurons of this network;

FIG. 7 shows a fundamental circuit diagram of a discrete module comprising four neurons; and

FIGS. 8a, b, c show different principles of the arrangement and of the interconnection of discrete modules for the formation of neural networks of different sizes.

As already set forth, FIG. 1 shows the basic structure of a neural network comprising four neurons N1 . . . N4. All of these neurons are completely meshed with one another. At the neurons N1 . . . N4, individual adjustment possibilities are present at the respectively appertaining evaluation stages B and at a neuron-associated decision stage E, whereby a threshold and an individual, constant input signal can be applied to the decision stages and a weighting information can be applied to the evaluators. In order to eliminate terminal points and, thus, to save chip area, these individual signals are at least partially serially input into the network, whereby these individual signals can be completely stored on the chip that contains the neural network. The following assumptions form the basis for the realization of a digital neural network.

the input/output word width per neuron is defined at 1 bit (binary) (1);

the word width in the evaluation per neuron is defined at 1 bit (binary) (2).

It is provided in detail for the digital neural network that a plurality of neurons completely meshed with one another are provided, each of these comprising an evaluation stage having a plurality of evaluators B that is equal to the plurality of neurons, and comprises a decision stage having the decision unit E; that a pre-processing means can supply every decision unit E with an adjustment information INF.sub.E via an information input, this adjustment information INF.sub.E effecting a defined pre-adjustment of the decision unit E; that a pre-processing means can supply every evaluator B with a weighting information INF.sub.G via an individual information input; that an output information INF.sub.A can be output by every decision unit E, being output to a post-processing means via an individual information output; that the information outputs of the decisions units E are respectively connected to an individual processing input of all evaluators B allocated to the appertaining decision unit E; and that individual processing outputs of the evaluators B are connected to individual processing inputs of the decision unit E in the appertaining neuron, so that every output information INF.sub.A can be fed back to every neuron N in mediated fashioned, cf. FIG. 1.

FIG. 2 shows a structure that can be inventively applied to the assumptions (1) and (2) recited above. This figure shows a neuron that is composed of a neural element having binary (0,+1) evaluation, whereby the element contains a shift register 20 having a plurality of register cells REG corresponding in number to the plurality of neurons in the neural network, said shift register having a serial input for the serial input of evaluation data. Every register cell REG has a logic element U with two signal inputs allocated to it; one signal input is connected to an output of an allocated register cell REG, and the other signal input is connected to the information output of the decision unit of an allocated neuron. The signal output of each logic element is connected to an individual input of an adder tree comprising a plurality of adders ADD arranged in cascade fashion whose outputs are connected to corresponding inputs of a final adder FADD. From a threshold register 22 comprising register cells REG, REG into which serial threshold data can be input from the pre-processing means over line 24, the final adder FADD can be supplied via a further input with the threshold data stored in this fashion. The final adder FADD supplies the operational sign signal VZ as operational sign switch, this operational sign signal VZ, for example, having the signal value "0" for a positive sum and the signal value "1" for a negative sum dependent on the size of the data sum calculated in the final adder FADD. This operational sign switch is followed by a NOT element N. The output of the NOT element N forms the information output of the decision unit that is fed back onto the processing inputs of all evaluators allocated to the appertaining decision unit, namely onto the other signal inputs of the appertaining logic element U. The logic element U is preferably an AND element.

For example, 50 binary evaluations each comprising a single bit are stored in the register 20 having a serial input. The register contents are AND operated with the respective input signal of, for example, the 50 neurons that are provided and are then added up in the adder tree that, for example, is a Wallace adder. Given 50 (equivalent) inputs, the adder including the final adder comprises about 50 full adders. At the beginning of every calculation, the threshold is input into these adders in the form of a negative number. When the sum of the "positive" input signals is greater than this threshold, then the operational sign VZ of the final adder is set, for example, to "0" and, thus, the output of the neuron is switched to "1". The threshold word that is stored in the register 22 is preferably 6 bits wide. Given application of the 1.5 .mu.m technology, such a neuron has a space requirement of about 0.8 mm.sup.2 or, respectively, given application of the 1.0 .mu.m technology, has a space requirement of about 0.4 mm.sup.2 and an additional space requirement of about 5 mm.sup.2 or, respectively, 2.5 mm.sup.2 is required for the overall wiring of a network comprising 50 neurons. The possible cycle frequency of this arrangement amounts to about 40 MHz or, respectively, 80 MHz.

FIG. 4 shows the structure of a neural element having expanded evaluation. In detail, FIG. 4 shows that the neuron is composed of a neural element having binary output signals that contain n, preferably n=3, shift registers 41-43 arranged parallel to one another each respectively having a plurality of register cells REG corresponding in number to the plurality of neurons in the neural network, and respectively having a serial input for coded, serial input of evaluation data, so that 2.sup.n evaluation degrees, preferably the evaluation degrees 0, 1, 2, 4, 8, 16, 32, 64 can be input to the neural element (an evaluation with positive and negative operational signs, for example -4, -2, -1, 0,+1,+2,+4,+8 is thereby also possible). Every register cell REG has a logic unit 44 with first and second signal inputs allocated to it; the first signal inputs of each logic unit 44 are connected to outputs of a decoder DEC, respectively allocated to the appertaining register cells REG, and the second signal input of each unit is connected to the information output of the decision unit of the allocated neuron and whereby the signal outputs of the logic means are connected to individual inputs of an adder tree comprising a plurality of adders ADD arranged in cascade fashion whose outputs are connected to corresponding inputs of a final adder FADD. From a threshold register 22 comprising register cells REG, REG into which the preprocessing means can input serial threshold data, the final adder FADD can be supplied via a further input with the threshold data stored in this fashion. As an operational sign switch, the final adder FADD supplies the operational signal signal VZ dependent on the size of the data sum calculated in the final adder FADD. The operational sign switch is followed by a NOT element N. The output of the NOT element forms the information output of the decision unit that is fed back onto the processing inputs of all evaluators allocated to the appertaining decision unit, namely, onto the second signal inputs of the appertaining logic elements 44. The logic elements 44 are preferably executed as AND elements.

FIG. 3 shows the structure of a neural element having ternary (-1, 0,+1) evaluation that contains a shift register 31 having a plurality of register cells REG corresponding in number to the plurality of neurons in the neural network and having a serial input for the serial input of evaluation data. Every register cell REG has two logic elements 32 with respectively two signal inputs allocated to it; the first signal input of each is respectively connected to an output of the appertaining register 31 cell REG, and the second signal input of each is connected in common to the information output of the decision unit of the allocated neuron. The signal outputs of the logic elements 32 are respectively connected to individual inputs of a first adder tree 33 comprising a plurality of adders ADD arranged in cascade fashion for positively evaluated signals or, respectively, of a second adder tree 34 comprising a plurality of adders ADD arranged in cascade fashion for negatively evaluated signals, whereby the outputs of these adder trees are connected to corresponding inputs of a final adder FADD. From a threshold register 22 comprising register cells REG, REG into which the pre-processing means can input serial threshold data, the final adder FADD can be supplied via a further input with the threshold data stored in such fashion. As the operational sign switch, the final adder FADD supplies the operational sign signal VZ dependent on the size of the data sum calculated in the final adder FADD. The operational sign switch is followed by a NOT element N. The output of the NOT element N forms the information output of the decision unit that is fed back onto the processing inputs of all evaluators allocated to the appertaining decision unit, namely onto the other signal input of the appertaining logic elements U. The logic elements are preferably executed as AND elements.

FIG. 5 shows a neural network that contains a single neuron that is composed of a neural element having binary output signal that contains n, preferably n=3 shift register sections 51-53 arranged parallel to one another, having a plurality of register cells REG corresponding in number to the plurality of neuron functions in the neural network and each having a serial input for the coded, serial input of evaluation data, so that 2.sup.n evaluation degrees, preferably the evaluation degrees 0, 1, 2, 4, 8, 16, 32, 64 can be input into the neural element. The shift register sections allocated to the single neuron are multiply provided in accord with the plurality of neuron functions and are arranged in series circuit (with register sections 54-56, for example) forming a loop with one another. Each register cell REG has a logic means 58 comprising first and second signal inputs allocated to it, whereby the first signal inputs are connected to outputs of a decoder DEC allocated to their respectively appertaining register cells REG, whereby the second input is indirectly connected to the information output of the decision unit of the neuron, and whereby the signal outputs of the logic means are connected to individual inputs of an adder tree comprising a plurality of adders ADD arranged in cascade fashion whose outputs are connected to corresponding inputs of a final adder FADD. From a threshold register comprising register cells REG into which serial threshold data can be input by the preprocessing means, the final adder FADD can be supplied via a further input with the threshold data stored in such fashion, whereby the register cells REG of the threshold register are subdivided into a plurality of sections, 57a- 57d, this plurality corresponding the plurality of the neuron functions. As operational signal switch, the final adder FADD supplies the operational sign signal VZ dependent on the size of the data sum calculated in the final adder FADD.

This operational sign switch is followed by a NOT element N. The output of the NOT element N forms the information output of the decision unit that is indirectly fed back onto the processing inputs of all evaluators allocated to the appertaining decision unit, namely onto the second signal inputs of the appertaining logic elements 58, whereby a preferably two-stage register arrangement that, per register arrangement stage, comprises a plurality of register cells REG 59 and 60 corresponding in number to the plurality of neuron functions and that can be driven via a multiplexer MUX is inserted between the output of the NOT element N and the second signal inputs.

FIG. 6 shows a neural network having serial accumulation in the neurons. It is provided in this structure that the neuron is respectively composed of a neural element having a binary output signal that contains n, preferably n=3 shift register sections arranged parallel to one another that each have a plurality of register cells REG 63 corresponding in number to the plurality of neuron functions in the neural network and that each have a respective serial input for the coded, serial input of evaluation data, so that 2.sup.n evaluation degrees, preferably the evaluation degrees 0, 1, 2, 4, 8, 16, 32, 64 can be input in the neural element. The outputs of the shift register sections allocated to the neural are each fed back onto the inputs thereof. A logic means 64 comprising first and second signal inputs is allocated to all of the register cells REG allocated to the neural element; the first signal inputs are connected to outputs of a decoder DEC allocated to the register cells REG, and the second signal input is indirectly connected to the information output of the decision unit of the neuron, and the signal outputs of the logic means 64 are connected to individual inputs of an accumulator ACC whose outputs are connected to corresponding inputs of an adder ADD. From a threshold register comprising register cells REG into which serial threshold data can be input by the pre-processing means, the adder ADD can be supplied via a further input with the threshold data stored in such fashion, whereby the register cells REG of the threshold register are subdivided into a plurality of sections corresponding in number to the plurality of neural elements. As operational sign switch, the adder ADD supplies the operational sign signal VZ dependent on the size of the data sum calculated in the adder ADD. This operational sign switch is followed by a NOT element N. The output of the NOT element N forms the information output of the decision unit that is indirectly fed back onto the processing inputs of all evaluators allocated to the appertaining decision unit, namely onto the second signal inputs of the appertaining logic elements 1, 64, whereby an output register having a plurality of register cells REG corresponding in number to the plurality of neural elements is inserted between the output of the NOT element N and the second signal inputs. Whenever a new signal value is applied for the output memory to the second signal input of the logic elements, the information is cyclically shifted by one place in the register cells REG respectively allocated to the neural elements and this new value is decoded, so that every signal value can be evaluated with a different information.

FIG. 7 shows a fundamental view of a discrete module comprising four neurons. Every neuron NR contains a shift register 71 having a plurality of register cells corresponding in number to the plurality of neurons NR, namely four, whereby the output of the fourth register cell is connected to the input of the first register cell. An AND logic means 72 is also provided, the first signal input thereof being connected to the output of the fourth register cell, the second signal input thereof being indirectly connected to the output of a full adder VA and the signal output thereof being connected to an adder ADD whose sum output is connected to the input of an accumulator ACC. The output of the accumulator ACC is connected to a second input of the adder ADD. A parallel output of the accumulator ACC is connected to the setting inputs of a plurality of register cells T of a shift register 73, whereby this plurality of register cells T corresponds to the bit width of the accumulator ACC, said shift register containing a plurality of register units T corresponding to the plurality of neurons N, namely four, whereby the output of this shift register is connected to the input of the full adder VA. A second input 74 of the full adder VA is charged with a defined potential, preferably grounded potential equal to logical zero, insofar as only a single module is provided for a neural network.

FIG. 8a shows a schematic illustration of a neural network comprising four neurons that is composed of a single module according to FIG. 7.

FIG. 8b shows a neural network that is formed of four modules according to FIG. 7, whereby the modules are arranged in quadratic form and whereby the second inputs of the full adders VA of the two lower modules lie at a defined potential, preferably grounded potential equal to logical zero, and whereby the second inputs of the full adders VA of the two upper modules are each connected to the sum output of the full adder of the module arranged respectively therebelow.

FIG. 8c, finally, shows a schematic illustration of a neural network comprising 16 neurons that contains 16 modules arranged in quadratic form in accord with the principles shown in FIG. 8b.

The circuits of the neurons as well as the auxiliary circuits thereof are preferably executed in 1.5 .mu.m CMOS technology. Another preferred version provides that the circuits of the neurons and the auxiliary circuits thereof are executed in 1.0 .mu.m CMOS technology.

The circuits of the neurons and the auxiliary circuits thereof can be fashioned in a single chip, whereby a plurality of these chips are arranged on a motherboard with corresponding wiring or interconnect management for connecting the chips to one another in other to form large neural networks. It is also contemplated that all required circuits are fashioned on a single wafer.

It will be understood that variations and modifications may be effected without departing from the spirit and scope of the novel concepts of the present invention.