Generator of neuron transfer function and its derivative

[0002] 1. Field of the Invention

[0003] This invention relates to an artificial neural network (ANN), particularly to a neuron component and its activation function including the derivative.

[0004] 2. Description of the Related Art

[0005] One of the artificial neural networks' most important components is the neuron, whose performance and complexity greatly affect the whole net. In many literatures, its activation function is found to be the sigmoid. Most of the analog implementations have used differential pair for generating the required function. In some cases, such as on-chip back-propagation learning, both a non-linear function and its derivative are required. Neurons that can meet the requirement have been presented. Most of these have voltage inputs and the current outputs. Since most of the applications employ current output synapses and voltage output neurons in order to enable the summation of those currents by simply connecting them together at the input of the neuron and to diverge signals from a neuron to a large quantity of synapses, these circuits a little bit inconvenient. In addition, some circuits are biased in the subthreshold region, so its driving capability is quite limited.

[0006] Therefore, an object of the invention is to provide a novel neuron circuit, which has current input and voltage outputs and is built with strong-inversion biased transistors, thereby realizing both a neuron activation function and its derivative.

[0007] Another object of the invention is to provide a novel neuron circuit, which enables threshold and gain factor adjustability and has high noise immunity.

[0008] Another object of the invention is to provide novel neuron circuit, which is very beneficial to on-chip learning neural networks.

[0009] To realize the above and other objects, the invention provides a novel neuron circuit, which is very beneficial to on-chip learning neural networks. The neuron circuit comprises: a pair of transistors having a first common terminal connected to a first input voltage, a second common terminal connected to a first output, a first gate connected to a second input voltage, and a second gate connected to a third input voltage; a surface field-effect transistor having a first terminal and a third gate connected together to the first output, which is connected to a first current source, and a second terminal connected to a second current source; and a first differential pair of transistors, having a third common terminal connected to a third current source, a first gate connected to the first output, a second gate connected to a fourth input voltage, a first terminal connected to a first load to form a connecting point as a second output, and a second terminal connected to a second load; a second differential pair of transistors, having a fourth common terminal connected to the third current source, a third gate connected to the first gate of the first differential pair of transistors output, a fourth gate connected to a fifth input voltage, a third terminal connected to a third load to form a connecting point as a third output, and a fourth terminal connected to a fourth load.

[0010] In the neural network, a neuron circuit design is the key point for the performance. In the invention with simple neuron circuit design, it can easily reach a requirement of errorless effect and decrease the time delay to within 1 ns. The circuit also has a large dynamic range and a high noise immunity using the adjustable threshold and gain factor. Therefore, these features make it quite fit for hardware realization besides its programmability.

[0011] The invention will become apparent by referring to the following detailed description of a preferred embodiment with reference to the accompanying drawings, wherein:

[0012] FIG. 1 is a schematic diagram of a neuron circuit of the invention;

[0013] FIG. 2ais a diagram of comparing a simulate neuron transfer function curve to a fitted sigmoid curve of FIG. 1;

[0014] FIG. 2bis a diagram of a relative error curve of FIG. 2a;

[0015] FIG. 3ais a diagram of comparing a simulated derivative curve to the derivative curve of the simulated neuron of the invention;

[0016] FIG. 3bis a diagram of a relative error curve, of FIG. 3a; and

[0017] FIG. 4 is a diagram of simulated neuron transfer function curves with different thresholds of the invention;

[0018] FIG. 5 is a diagram of activation curves under different gains of the invention.

[0019] To realize the above and other objects, the invention provides a novel neuron circuit with adjustable threshold and gain factor and high noise immunity, which is very beneficial to on-chip learning neural networks.

[0020] Refer to FIG. 1, illustrating a neuron circuit of the invention. In FIG. 1, the neuron circuit comprises: a pair of transistors M1, M2, a surface field-effect transistor Ms, and at least one differential pair composed of identical transistors M3, M4.

[0021] As shown in FIG. 1, a pair of transistors M1, M2 having a first common terminal A and a second common terminal B, a surface field-effect transistor Ms having a first terminal S and a gate Gs concurrently connected to the output B of the pair M1, M2, and at least one differential pair composed of identical transistors M3, M4, which one gate G3 of identical transistors M3, M4 is connected to the output B of the pair M1, M2 and the other D is connected to a fixed input voltage V_ref2or V_ref2+ΔV, together with the active loads to realize the actual sigmoidal shaped non-linearity.

[0022] In the proposed neuron circuit schematic, as shown in FIG. 1, V_out(1) outputs the sigmoidal transfer function. (V_out(2)−V_out(1)) realizes its approximate derivative. In the dash frame 1, which functions as a current generator, V_ref1is carefully chosen so that both transistors M1 and M2 are working in their linear range. The formed linear resistor R_ABcan be controlled by the gate voltage of both transistors V_Nand V_p. A surface field-effect transistor Ms, which is a current-controlled transistor, is connected to the output of the frame 1, and its gate current source of the transistor Ms is used to adjust the output voltage of the neuron circuit, wherein the output voltage is the desired sigmoidal transfer function. In the case that the dash dot frame 2 outputs the desired sigmoidal transfer function while the dash dot frame 3 outputs its derivative, a simple differential pair composed of identical transistors, for example M3 and M4 of the frame 2, together with the active loads, for example Acs of the frame 2, realize the actual sigmoidal shaped non-linearity. One port of the differential pair is connected to point B and the other is connected to a fixed voltage V_ref2or V_ref2+ΔV. I_ref1and I_ref2are fixed current sources. The referenced current directions are shown as the arrows in FIG. 1.

[0023] Assuming that M3, M4 are operating in saturation and follow an ideal square law. V_sis the input differential voltage, i.e. V_d=V_B−V_D, then $\begin{array}{cc}{I}_{\mathrm{d3}}\left(V_d\right)=\left(I_{\mathrm{ref2}}/2\right)+\left(\left(\beta V_d\sqrt{\frac{4I_{\mathrm{ref2}}}{\beta }-V_d^2}\right)/4\right)& \left(1\right)\end{array}$

[0024] With $V_{\mathrm{d0}}\equiv \sqrt{\frac{2I_{\mathrm{ref2}}}{\beta }}\le V_d\le -\sqrt{\frac{2I_{\mathrm{ref2}}}{\beta }}\equiv V_{\mathrm{d1}}.$

[0025] Here β is the transconductance parameter for transistors M3 and M4. I_AB=I_in+I_ref1. When I_inis small, V_d>V_d0, V_out(1) remains the low saturation voltage. As I_inincreasing, V_Bdescends tardily and V_out(1) increases slowly. When V_d<V_d1, V_outreaches and remains the high saturation level.

[0026] To verify that it operates correctly, the proposed circuit is simulated with HSPICE (Highly Optional Simulation Program Integrated Circuit Emphasis) in the art, using level 47 transistor models for a standard 1.2 μm CMOS process. FIG. 2(a) shows the neuron transfer function and its fitted sigmoid curve. Their relative error is not more than 3% as shown in FIG. 2(b).

[0027] Using the forward difference method, the approximate derivative voltage V_derivis available by subtracting V_out(1) from V_out(2).

[0028] V_deriv≡ΔVV′_out≅V_out(V_B(I_in)−V_ref2+ΔV)−V_out(V_B(I_in)−V_ref2)=V_out(2)−V_out(1)  (2)

[0029] Here ΔV is a fixed small value. FIG. 3(a) shows the derivative found by simulation of the circuitry in FIG. 1 and the derivative of the simulated neuron transfer function. The solid line in FIG. 3(b) shows that the relative error between them is less than 5%.

[0030] The great power of an artificial neural network derives from its ability to be adapted to the unknown and changing environment. Therefore, good programmability is of fundamental importance. It is known that a sigmoid function can be given by

[0031] f(X)=1/(1+exp(−α(X·W)+Θ))

[0032] where X is the input matrix, W is the weight matrix, α is the gain factor and Θ is the bias weight vector. Different applications may need a difference between α and Θ. This can be realized by varying I_ref1, V_Nand V_p.

[0033] When I_ref1changes, the bias weight also changes. As I_ref1increases, the current I_inneeded to satisfy V_B−V_ref2>V_d0decreases, so the transfer curve shifts to the left. In the same way, when I_ref1decreases, the curve shifts to the right. FIG. 4 shows the simulated neuron transfer functions with different thresholds.

[0034] When both transistors M1 and M2 are working in their linear range and their sizes are chosen in such a way that β₁=β₂, the relation between the current I_ABand V_ABcan be written as

[0035] I_AB=I₁+I₂=β₁V_AB[(V_N−V_T1)−(V_p−|V_T2|)]  (4)

[0036] So,

[0037] R_AB=1/{β₁V_AB[(V_N−V_p)−(V_T1+|V_T2|)]}  (5)

[0038] Equation 5 shows that the bigger (V_N−V_p) is, the less R_ABis. That is, the less the slope of V_Bversus I_inis. This means that V_out1increases more slowly, i.e. the gain factor is smaller.

[0039] Different transfer functions with various gain factors are described in FIG. 5. Note that the saturation levels of the sigmoid remain constant for different gain values, in contrast to most implementations in the prior art. This ensures that for different gain values, the input range of synapse in subsequent layer is completely used.

[0040] By utilizing the proposed neuron circuit that generates both an adjustable sigmoid-like function and a good approximation of its derivative has only a delay of within 1 ns in operation and speediness. Besides, the simple circuit composed of transistors as mentioned above, which works in the strong-inversion range, also has a large dynamic range and a high noise immunity. This is quite fit for hardware realization.

[0041] Although the present invention has been described in its preferred embodiment, it is not intended to limit the invention to the precise embodiment disclosed herein. Those who are skilled in this technology can still make various alterations and modifications without departing from the scope and spirit of this invention. Therefore, the scope of the present invention shall be defined and protected by the following claims and their equivalents.