RATE STABILIZATION THROUGH PLASTICITY IN SPIKING NEURON NETWORK

[0001] This application is related to co-owned U.S. patent application Ser. No. 13/152,119, entitled “SENSORY INPUT PROCESSING APPARATUS AND METHODS”, filed on Jun. 2, 2011, co-owned and co-pending U.S. patent application Ser. No. 13/465,924, entitled “SPIKING NEURAL NETWORK FEEDBACK APPARATUS AND METHODS”, filed May 7, 2012, co-owned and co-pending U.S. patent application Ser. No. 13/465,903 entitled “SENSORY INPUT PROCESSING APPARATUS IN A SPIKING NEURAL NETWORK”, filed May 7, 2012, a co-owned U.S. patent application Ser. No. 13/465,918, entitled “SPIKING NEURAL NETWORK OBJECT RECOGNITION APPARATUS AND METHODS”, filed May 7, 2012, a co-owned U.S. patent application Ser. No. 13/488,106, entitled “SPIKING NEURON NETWORK APPARATUS AND METHODS”, filed Jun. 4, 2012, a co-owned U.S. patent application Ser. No. 13/488,144, entitled “SPIKING NEURON NETWORK APPARATUS AND METHODS”, filed Jun. 4, 2012, a co-owned U.S. patent application Ser. No. 13/541,531, entitled “CONDITIONAL PLASTICITY SPIKING NEURON NETWORK APPARATUS AND METHODS”, filed Jul. 3, 2012, co-owned U.S. patent application Ser. No. 13/660,923, entitled “ADAPTIVE PLASTICITY APPARATUS AND METHODS FOR SPIKING NEURON NETWORK”, Oct. 25, 2012, co-owned U.S. patent application Ser. No. 13/660,967, entitled “APPARATUS AND METHODS FOR ACTIVITY-BASED PLASTICITY IN A SPIKING NEURON NETWORK”, filed Oct. 25, 2012, co-owned U.S. patent application Ser. No. 13/660,982, entitled “SPIKING NEURON SENSORY PROCESSING APPARATUS AND METHODS FOR SALIENCY DETECTION”, filed Oct. 25, 2012, and co-owned U.S. patent application Ser. No. 13/660,945, entitled “MODULATED PLASTICITY APPARATUS AND METHODS FOR SPIKING NEURON NETWORKS”, filed Oct. 25, 2012, each of the foregoing incorporated herein by reference in its entirety.

[0002] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.

[0003] 1. Technological Field

[0004] The present disclosure relates generally to artificial neural networks, and more particularly in one exemplary aspect to computer apparatus and methods for plasticity implementation in a pulse-code neural network.

[0005] 2. Description of Related Art

[0006] Artificial spiking neural networks are frequently used to gain an understanding of biological neural networks, and for solving artificial intelligence problems. These networks typically employ a pulse-coded mechanism, which encodes information using timing of the pulses. Such pulses (also referred to as “spikes” or ‘impulses’) are short-lasting (typically on the order of 1-2 ms) discrete temporal events. Several exemplary embodiments of such encoding are described in commonly owned and co-pending U.S. patent application Ser. No. 13/152,084 entitled APPARATUS AND METHODS FOR PULSE-CODE INVARIANT OBJECT RECOGNITION”, filed Jun. 2, 2011, and U.S. patent application Ser. No. 13/152,119, Jun. 2, 2011, entitled “SENSORY INPUT PROCESSING APPARATUS AND METHODS”, each incorporated herein by reference in its entirety.

[0007] Typically, artificial spiking neural networks, such as the exemplary network described in owned U.S. patent application Ser. No. 13/541,531, entitled “CONDITIONAL PLASTICITY SPIKING NEURON NETWORK APPARATUS AND METHODS”, may comprise a plurality of units (or nodes), which can be thought of as corresponding to neurons in a biological neural network. Any given unit may be connected to many other units via connections, also referred to as communications channels, and/or synaptic connections. The units providing inputs to any given unit are commonly referred to as the pre-synaptic units, while the units receiving the inputs are referred to as the post-synaptic units.

[0008] Each of the unit-to-unit connections may be assigned, inter alia, a connection efficacy, which in general may refer to a magnitude and/or probability of input spike influence on unit output response (i.e., output spike generation/firing). The efficacy may comprise, for example a parameter—e.g., synaptic weight—by which one or more state variables of post-synaptic unit are changed. In one or more implementations, the efficacy may comprise a latency parameter by characterizing propagation delay from a pre-synaptic unit to a post-synaptic unit. In some implementations, greater efficacy may correspond to a shorter latency. During operation of a pulse-code network, neuron responses to input stimulus may be characterized by a firing rate (e.g., a number of spikes per unit time). Individual neurons of such network may be configured to respond to one or more features of the input (e.g., a vertical and/or a horizontal edge, an object of a certain color, and/or other feature). When some of the features (e.g., red circles) may be more prevalent within the input, the neurons with the receptive fields corresponding to the red circles may respond more frequently (e.g., with a higher firing rate) to the input. Such frequent responses may overwhelm or ‘drown-out’ neurons responding to less frequent/prominent features of the input, and/or lead to an excessive use of network communication, computational, and/or power resources.

[0009] Consequently, there is a salient need for improved network operation capable of, inter alia, equalizing firing adaptive plasticity mechanisms to enable a pulse-code (e.g., spiking) neuron network capable of operating in a wide variety of input and network dynamic regimes.

[0010] The present disclosure satisfies the foregoing needs by providing, inter alia, apparatus and methods for implementing plasticity in spiking neuron networks that is capable of, inter alia, stabilization of neuron firing rate.

[0011] In a first aspect, an apparatus configured to process input signals in spiking network is disclosed. In one embodiment, the network includes an excitatory and an inhibitory neuron and the apparatus includes a storage apparatus configured to store said spiking network; and logic, in data communication with said storage apparatus. The logic is in one variant configured to, based on an excitatory response by the excitatory neuron, (a) strengthen a connection communicating the excitatory response to the inhibitory neuron; and (b) strengthen a connection communicating an inhibitory response from the inhibitory neuron. In another variant, the logic is further configured to, (i) based on the inhibitory response by the inhibitory neuron, (a) weaken the connection communicating the excitatory response to the inhibitory neuron; and (b) weaken the connection communicating the inhibitory response from the inhibitory neuron.

[0012] In a second aspect, a non-transitory computer-readable storage medium having instructions resident thereon is disclosed. In one embodiment, the instructions are executable by one or more processors to implement activity-dependent plasticity in a spiking neuron network signal processing apparatus. The exemplary instructions are configured to, when executed: (i) operate an excitatory network portion configured to generate an excitatory output based at least on the signal; (ii) operate an inhibitory network portion configured to generate an inhibitory output based at least on the excitatory output; and (iii) for a given value of a parameter associated with the signal, adjust an efficacy of one or more connections associated with the inhibitory output. In one variant, the excitatory output is characterized by an excitatory firing rate, and the adjustment is performed so as to substantially equalize the excitatory firing rate of individual neurons of the excitatory network portion.

[0013] In a third aspect, a computerized spiking neuron apparatus is disclosed. In one embodiment, the apparatus includes one or more processors configured to execute computer program modules, and the computer program modules are configured to, when executed, cause the one or more processors to: (i) based on generating an output, increase an efficacy of one or more inhibitory connections, the one or more connections configured to provide an inhibitory input to a neuron; and (ii) based on receiving the inhibitory input, reduce the efficacy.

[0014] In a fourth aspect, a computerized method of processing input stimulus by a spiking neuron network is disclosed. In one embodiment, the network includes first selectivity and second selectivity neurons, and the method includes: (i) causing the first selectivity neuron to respond to a first feature of the input stimulus, the first selectivity neuron response being characterized by a first rate configured based on a frequency of occurrence of the first feature; (ii) causing the second selectivity neuron to respond to a second feature of the input stimulus, the second selectivity neuron response being characterized by a second rate configured based on a frequency of occurrence of the second feature; and (iii) if the frequency of occurrence of the first feature is greater than the frequency of occurrence of the second feature, applying a rate reduction mechanism configured to reduce the first rate based on one or more response of the second selectivity neurons.

[0015] In a fifth aspect, a spiking neural network apparatus is disclosed. In one embodiment, the neural network apparatus comprises, inter alia, excitatory and selectivity neurons.

[0016] In a sixth aspect, a input processing method for use in a spiking network is disclosed. In one embodiment, the method comprising adjusting one or more responses based at least in part on, inter alia, excitatory or inhibitory neurons.

[0017] Further features of the present disclosure, its nature and various advantages will be more apparent from the accompanying drawings and the following detailed description.

[0039] All Figures disclosed herein are © Copyright 2012 Brain Corporation. All rights reserved.

[0040] Exemplary embodiments and implementations of the various aspects of the present innovation will now be described in detail with reference to the drawings, which are provided as illustrative examples so as to enable those skilled in the art to practice the disclosure. Notably, the figures and examples below are not meant to limit the scope of the present disclosure to a single embodiment or implementation, but other embodiments and implementations are possible by way of interchange of or combination with some or all of the described or illustrated elements. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to same or like parts.

[0041] Where certain elements of these embodiments or implementations can be partially or fully implemented using known components, only those portions of such known components that are necessary for an understanding of the present disclosure will be described, and detailed descriptions of other portions of such known components will be omitted so as not to obscure the innovation.

[0042] In the present specification, an embodiment or implementations showing a singular component should not be considered limiting; rather, the invention is intended to encompass other embodiments or implementations including a plurality of the same component, and vice-versa, unless explicitly stated otherwise herein.

[0043] Further, the present disclosure encompasses present and future known equivalents to the components referred to herein by way of illustration.

[0044] As used herein, the term “bus” is meant generally to denote all types of interconnection or communication architecture that is used to access the synaptic and neuron memory. The “bus” could be optical, wireless, infrared or another type of communication medium. The exact topology of the bus could be for example standard “bus”, hierarchical bus, network-on-chip, address-event-representation (AER) connection, or other type of communication topology used for accessing, e.g., different memories in pulse-based system.

[0045] As used herein, the terms “computer”, “computing device”, and “computerized device”, include, but are not limited to, personal computers (PCs) and minicomputers, whether desktop, laptop, or otherwise, mainframe computers, workstations, servers, personal digital assistants (PDAs), handheld computers, embedded computers, programmable logic device, personal communicators, tablet or “phablet” computers, portable navigation aids, J2ME equipped devices, cellular telephones, smart phones, personal integrated communication or entertainment devices, or literally any other device capable of executing a set of instructions and processing an incoming data signal.

[0046] As used herein, the term “computer program” or “software” is meant to include any sequence or human or machine cognizable steps which perform a function. Such program may be rendered in virtually any programming language or environment including, for example, C/C++, C#, Fortran, COBOL, MATLAB™, PASCAL, Python, assembly language, markup languages (e.g., HTML, SGML, XML, VoXML), and the like, as well as object-oriented environments such as the Common Object Request Broker Architecture (CORBA), Java™ (including J2ME, Java Beans), Binary Runtime Environment (e.g., BREW), and other languages.

[0047] As used herein, the terms “connection”, “link”, “synaptic channel”, “transmission channel”, “delay line”, are meant generally to denote a causal link between any two or more entities (whether physical or logical/virtual), which enables information exchange between the entities.

[0048] As used herein, the term “memory” includes any type of integrated circuit or other storage device adapted for storing digital data including, without limitation, ROM. PROM, EEPROM, DRAM, Mobile DRAM, SDRAM, DDR/2 SDRAM, EDO/FPMS, RLDRAM, SRAM, “flash” memory (e.g., NAND/NOR), memristor memory, and PSRAM.

[0049] As used herein, the terms “processor”, “microprocessor” and “digital processor” are meant generally to include all types of digital processing devices including, without limitation, digital signal processors (DSPs), reduced instruction set computers (RISC), general-purpose (CISC) processors, microprocessors, gate arrays (e.g., field programmable gate arrays (FPGAs)), PLDs, reconfigurable computer fabrics (RCFs), array processors, secure microprocessors, and application-specific integrated circuits (ASICs). Such digital processors may be contained on a single unitary IC die, or distributed across multiple components.

[0050] As used herein, the term “network interface” refers to any signal, data, or software interface with a component, network or process including, without limitation, those of the FireWire (e.g., FW400, FW800, etc.), USB (e.g., USB2), Ethernet (e.g., 10/100, 10/100/1000 (Gigabit Ethernet), 10-Gig-E, etc.), MoCA, Coaxsys (e.g., TVnet™), radio frequency tuner (e.g., in-band or OOB, cable modem, etc.), Wi-Fi (802.11), WiMAX (802.16), PAN (e.g., 802.15), cellular (e.g., 3G, LTE/LTE-A/TD-LTE, GSM, etc.) or IrDA families.

[0051] As used herein, the terms “pulse”, “spike”, “burst of spikes”, and “pulse train” are meant generally to refer to, without limitation, any type of a pulsed signal, e.g., a rapid change in some characteristic of a signal, e.g., amplitude, intensity, phase or frequency, from a baseline value to a higher or lower value, followed by a rapid return to the baseline value and may refer to any of a single spike, a burst of spikes, an electronic pulse, a pulse in voltage, a pulse in electrical current, a software representation of a pulse and/or burst of pulses, a software message representing a discrete pulsed event, and any other pulse or pulse type associated with a discrete information transmission system or mechanism.

[0052] As used herein, the term “receptive field” is used to describe sets of weighted inputs from filtered input elements, where the weights may be adjusted.

[0053] As used herein, the term “Wi-Fi” refers to, without limitation, any of the variants of IEEE-Std. 802.11 or related standards including 802.11 a/b/g/n/s/v and 802.11-2012.

[0054] As used herein, the term “wireless” means any wireless signal, data, communication, or other interface including without limitation Wi-Fi, Bluetooth, 3G (3GPP/3GPP2), HSDPA/HSUPA, TDMA, CDMA (e.g., IS-95A, WCDMA, etc.), FHSS, DSSS, GSM, PAN/802.15, WiMAX (802.16), 802.20, narrowband/FDMA, OFDM, PCS/DCS, LTE/LTE-A/TD-LTE, analog cellular, CDPD, RFID or NFC (e.g., EPC Global Gen. 2, ISO 14443, ISO 18000-3), satellite systems, millimeter wave or microwave systems, acoustic, and infrared (e.g., IrDA).

[0055] The present disclosure provides, in one salient aspect, apparatus and methods for implementing activity-based plasticity mechanisms in spiking neuron networks configured to, inter alia, to improve network response to input stimuli comprising a variety of features.

[0056] In some implementations, the input features may be unevenly distributed, so that some features may occur more frequently than others (e.g., a dark plane/bird statistically corresponding to a smaller area of the sky background). The activity-based plasticity mechanism of the disclosure is capable of reducing the response rate of neurons that develop receptive fields to more prevalent features. As a result, network output may become more evenly spread between different neurons responding to features of different frequency. Such functionality may advantageously provide network output such that rarely occurring features are not “drowned out” by more widespread stimulus.

[0057] In some implementations, the activity based plasticity may be implemented in a network comprising inhibitory units (e.g., artificial spiking neurons). Any given neuron of the network may provide an inhibitory signal to one or more neurons of the network via one or more outgoing inhibitory connections. The neuron may receive inhibitory signals from the one or more neurons of the network via one or more incoming inhibitory connections. The neuron may generate a (post-synaptic) output. The activity based plasticity rule may comprise a pre-synaptic rule corresponding to the network input occurring prior the neuron generating the output. The activity-based plasticity rule may further comprise a post-synaptic rule corresponding to the neuron generating the output after the occurrence of the network input. Pre and/or post-synaptic rules may be used to, inter alia, modify the efficacy of inhibitory connections.

[0058] In one or more implementations, the pre-synaptic and the post-synaptic rules may be configured to cause anti-correlated (with one another) efficacy adjustment. In some implementations, the pre-synaptic rule may potentiate the connection, and/or the post-synaptic rule may depress the connection.

[0059] In some implementations, the activity based plasticity may be implemented in a heterogeneous network comprising one or more classes of different neuron types characterized by different response properties and different plasticity rules. In some implementations, the heterogeneous network may comprise excitatory and inhibitory units. The excitatory units may receive a feed-forward (FF) sensory input and an inhibitory input from the inhibitory neurons. The FF input may comprise visual, audio, and/or other input. The excitatory units may generate output. The excitatory units may provide their output as e.g., feed-forward excitatory input to the inhibitory units via excitatory connections.

[0060] The heterogeneous network may be configured to implement separate activity-based plasticity rules for excitatory and inhibitory connections. The inhibitory connection plasticity rule may comprise a pre-synaptic rule corresponding to the inhibitory unit input occurring after the output generation by excitatory neuron the. The inhibitory connection plasticity rule may further comprise a post-synaptic rule corresponding to the excitatory unit generating the output after the occurrence of the inhibitory input.

[0061] The excitatory connection plasticity rule may comprise a pre-synaptic rule corresponding to the excitatory input arriving after the inhibitory unit has generated output. The excitatory connection plasticity rule may further comprise a post-synaptic rule corresponding to the inhibitory unit generating the output after the occurrence of the excitatory input.

[0062] The activity-based plasticity methodology described herein may advantageously improve learning of the neural network. The improved learning may be characterized by e.g., emergence of different receptive fields, requiring fewer neurons to represent a given input, and/or ability of the network to recognize a richer feature set. When the input comprises unevenly distributed (in frequency) features, the improved learning may be further characterized emergence of receptive fields that may reflect feature occurrence. By way of non-limiting example, consider an input comprising predominantly vertical (or near vertical features), and rare horizontal (or near horizontal) features. Such input may correspond for instance to video footage of a winter landscape with predominantly vertical live trees and some fallen horizontal trees. The activity based plasticity adjustment may cause additional selectivity between neurons that develop receptive fields for vertical orientation. That is, as the vertical features are prevalent in the input, individual neurons may develop progressively more selective (e.g., narrower in angular dimension) receptive fields, wherein some neurons begin to respond to only vertical features within, for example ±2° from the vertical, and some neurons begin to respond to only slightly slanted features within, for example 2-5° from the vertical. Conversely, as the horizontal features occur less frequently, the neurons with horizontal sensitivity may respond to horizontal features within ±20° of the horizontal orientation.

[0063] Implementations of the foregoing functionality of the present disclosure are useful in a variety of applications, including for instance a prosthetic device, autonomous robotic apparatus, and other electromechanical devices requiring visual or other sensory data processing functionality.

[0064] Detailed descriptions of the various embodiments and implementations of the apparatus and methods of the disclosure are now provided. Although certain aspects of the disclosure can best be understood in the context of the sensory (e.g., visual) information processing using pulse-code neural networks, the disclosure is not so limited, and implementations of the disclosure may also be used in a wide variety of other applications, including for instance in implementing connection adaptation in pulse-code neural networks.

[0065] Implementations of the disclosure may be for example deployed in a hardware and/or software realization of a neuromorphic computer system. In one such implementation, a robotic system may include a processor embodied in an application specific integrated circuit, which can be adapted or configured for use in an embedded application (such as a prosthetic device).

[0066] FIG. 1 illustrates a spiking neuronal network 100, configured to process sensory stimulus, in accordance with one or more implementations. The sensory stimulus (the input 104 in FIG. 1) may comprise, for example, an audio signal, a stream of video frames, and/or other input. In some implementations, the input stimulus 104 may comprise image frames received from an image sensor (such as a CCD or a CMOS sensor device). In one or more implementations, the input may comprise a pixel stream downloaded from a file, such as a stream of two-dimensional matrices of red green blue RGB values (e.g., refreshed at a 24 Hz or other suitable frame rate). It will be appreciated by those skilled in the art when given this disclosure that the above-referenced image parameters are merely exemplary, and many other image representations (e.g., bitmap, luminance-chrominance (YUV, YCbCr), cyan-magenta-yellow and key (CMYK), grayscale, etc.) are equally applicable to and useful with the various aspects of the present disclosure. Furthermore, data frames corresponding to other (non-visual) signal modalities such as sonograms, IR, radar or tomography images are equally compatible with the processing network of FIG. 1, or yet other configurations.

[0067] The network 100 of FIG. 1 may comprise excitatory units (depicted by triangles 102 in FIG. 1) and inhibitory units 112 (depicted by white circles 112). The excitatory units 102 may receive the sensory input 104 as feed-forward stimulus. The excitatory units 102 may receive feedback inhibitory input via inhibitory connections (depicted by the broken lines 112 in FIG. 1) from the inhibitory neurons 112. The inhibitory connections 114 may be characterized by inhibitor efficacy, depicted by solid circles 118 in FIG. 1.

[0068] Individual excitatory neurons 102 may be operated in accordance with a neuron dynamic process that may be characterized by a neuronal state parameter. In some implementations, the state parameter may comprise neuron excitability, neuron membrane potential, and/or one or more other parameters. In some implementations, the dynamic process may comprise a stochastic process, such as for example described in co-owned and co-pending U.S. patent application Ser. No. 13/487,533, entitled “STOCHASTIC SPIKING NETWORK LEARNING APPARATUS AND METHODS” filed Jun. 4, 2012, incorporated herein by reference in its entirety. In some implementations, connection efficacy may be used to describe a magnitude, and/or a probability of influence of pre-synaptic spike (e.g., the input 104 in FIG. 1) on response generation by a post-synaptic neuron (e.g., 102 in FIG. 1). In one or more implementations, connection efficacy may comprise for example a parameter such as synaptic weight, by which one or more state variables of the post-synaptic neuron(s) may be changed.

[0069] The excitatory neurons 102 may generate responses based on the sensory input 104 and the inhibitory input 114. In some implementations, the response may be generated in accordance with a deterministic spike response model, such as for example described in U.S. patent application Ser. No. 13/152,119, entitled “APPARATUS AND METHODS FOR TEMPORALLY PROXIMATE OBJECT RECOGNITION”, filed on Jun. 2, 2011, incorporated herein by reference in its entirety, In some implementations, the spike response process may comprise an adaptive threshold adjustment, such as e.g., described in U.S. patent application Ser. No. 13/623,820, entitled “APPARATUS AND METHODS FOR ENCODING OF SENSORY DATA USING ARTIFICIAL SPIKING NEURONS”, filed on Sep. 20, 2012 incorporated herein by reference in its entirety. In some implementations, the neuron may be operable in accordance with stochastic process, such as for example that described in U.S. patent application Ser. No. 13/487,499, entitled “STOCHASTIC APPARATUS AND METHODS FOR IMPLEMENTING GENERALIZED LEARNING RULES”, filed on Jun. 4, 2012, incorporated herein by reference in its entirety. Response generation in a stochastic neuron may be characterized by a probability of response generation.

[0070] The responses of the neurons 102 may be communicated via feed-forward connections 110, for subsequent processing by other parts of the network. In some implementations, such processing may comprise detection of higher-order features (e.g., corners), objects (e.g., rectangles), and/or other applications. Typically, the feed-forward stimulus 104 may cause an increase of a probability of the response by the neuron 102. Conversely, the feedback stimulus 114 may cause a decrease of the response probability. In some implementations, the probability decrease may be based on level of the inhibitory efficacy, e.g., element 118 in FIG. 1. The neurons 102 responses may be communicated to the neurons 112 as excitatory feed-forward stimulus via the excitatory connections 106.

[0071] In some implementations, corresponding to the inhibitory and/or excitatory units generating “bursts” of spikes, the inhibitory efficacy may suppress generation of some spikes within the burst. Such implementations may be referred to as the “soft” inhibition.

[0072] In some implementations, the inhibitory efficacy may be used to increment an inhibitory trace of a post-synaptic unit. The inhibitory trace may be configured e.g., in accordance with a “leaky” neuron process that may be configured to cause the inhibitory trace to decay with time in-between the increment events. In one or more implementations, the decay may be characterized by an exponentially decaying function of time, such as described in described in U.S. patent application Ser. No. 13/548,071, entitled “SPIKING NEURON NETWORK SENSORY PROCESSING APPARATUS AND METHODS”, filed Jul. 12, 2012, incorporated herein by reference in its entirety. One or more inputs from the inhibitory units may cause the inhibitory trace to rise above an inhibition threshold; the inhibitory trace that is above the threshold may cause for example a “hard” inhibition preventing subsequent post-synaptic unit activity for a period of time.

[0073] In some implementations (not shown), the neuron units may comprise logic configured to implement inhibitory trace mechanism. In one such case, the inhibitory connections (e.g., the connections 114 of FIG. 1) may comprise parameters that are internal to the respective neuron (e.g., the neuron 102 in FIG. 1), thereby advantageously obviating the need for a separate inhibitory unit and/or inhibitory connection (and hence simplifying the network).

[0074] The inhibitory neurons 112 may generate responses based on the excitatory input 104. The responses of the neurons 112 may be communicated back to the neurons 102 via feed-back inhibitory connections, 114. The combination of neurons 102, 112, and the connections, 106, 114 may form a negative feedback loop.

[0075] The excitatory connections 106 may be operable in accordance with activity-based plasticity rule, such as the exemplary rule 220 illustrated in FIG. 2. The plasticity rule 220 may be used to modify efficacy (denoted by the circle 108 in FIG. 1) of the connection 106. The plasticity rule 220 may comprise pre-synaptic portion 222 and post-synaptic portion 224, configured based on a time interval between time t_post of the post-synaptic response (neuron output) and time t_pre of the pre-synaptic input as: Δt=t_post−t_pre. It is noted that in FIG. 2, time is shown to increase from right to left, so that the area to the right of the y-axis (ordinate) corresponds to the past, relative the time of the post-synaptic response t_post. The pre-synaptic rule portion 222 may be invoked when the excitatory input (e.g., via the connection 106 in FIG. 1) arrives at the inhibitory unit (e.g., the neuron 112) after the unit 112 has generated post-synaptic response. The post-synaptic rule portion 224 may be invoked when the inhibitory neuron 112 may generate post-synaptic response after the occurrence of the excitatory input.

[0076] In one or more implementations, the pre-synaptic and the post-synaptic rules 222, 224 may be configured to effectuate opposing efficacy adjustments (e.g., as denoted by arrows 226, 228 in FIG. 2). For instance, the pre-synaptic rule 222 may cause efficacy increase (e.g., potentiate the connection 106) by amount 226, while the post-synaptic rule 224 may cause efficacy decrease (e.g., depress the connection 106) by amount 228. In some implementations, the depression magnitude 228 may be configured so as to be smaller than the potentiation (increase) magnitude 226. For instance, the depression magnitude 228 might comprise a 0.5 fraction (half) of the potentiation magnitude 226. It will be appreciated by those skilled in the arts that the exemplary excitatory connection plasticity rule shown in FIG. 2 is merely provided to illustrate the principles of the disclosure, and various other rules may readily be applied.

[0077] In some implementations, the excitatory connection 106 rule may be configured such that an integral of the pre-synaptic rule is positive while an integral of the post-synaptic rule is negative. Ratio of the magnitudes of the pre-synaptic rule and of the post-synaptic rule may be configured between 0.001 and 1.0.

[0078] The inhibitory connections 114 may further be operable in accordance with activity-based plasticity rule, such as for example the rules 300, 320, 340 illustrated in FIGS. 3A-3C, respectively. The plasticity rules of FIGS. 3A-3C may be used to modify efficacy (denoted by the solid circle 118 in FIG. 1) of the connection 114.

[0079] The plasticity rules 300, 320, 340 may comprise pre-synaptic portions 302, 322, 342, respectively, and post-synaptic portions 304, 324, 344, respectively, configured based on a time interval between time t_post of the post-synaptic response (neuron output) and time t_pre of the pre-synaptic input as: Δt=t_pre−t_post. It is noted that the time interval is shown in the Figures to increase from right to left, as indicated by the horizontal arrow in FIGS. 3A-3C. The pre-synaptic rule portion may be invoked when the inhibitory input arrives (e.g., via the connection 114 in FIG. 1) at the excitatory inhibitory unit (e.g., the neuron 102) after the unit 112 has generated a post-synaptic response. The post-synaptic rule portion may be invoked when the excitatory neuron 102 may generate a post-synaptic response after the occurrence of the inhibitory input (e.g., via the connection 114).

[0080] In one or more implementations, such as illustrated in FIG. 3A, the pre-synaptic and the post-synaptic rules 302, 304 may be configured to effectuate opposing efficacy adjustments (e.g., as denoted by arrows 306, 308 in FIG. 2) of the inhibitory connections (e.g., 114 in FIG. 1). For example, the pre-synaptic rule 302 may cause efficacy decrease (e.g., depress and/or weaken the connection 106) by amount 306, while the post-synaptic rule may cause efficacy increase (e.g., strengthen/potentiate the connection 114) by amount 308. In some implementations, the potentiation magnitude 308 may be configured to be greater than the depression magnitude 306. For example, the depression magnitude 306 may comprise a 0.01-0.5 fraction of the potentiation magnitude 308.

[0081] In one or more implementations, such as illustrated in FIG. 3B, the post-synaptic portion 324 of the plasticity rule 320 may comprise a potentiating region of width 330 and a region of width 332 where little (or no) efficacy modification is performed. When processing video input, comprising for example a 25 frames-per-second pixel stream, the width 330 of the potentiation region may be selected between 5 ms and 20 ms. As it will be appreciated by those skilled in the art, that sensory input of different temporal scales (e.g., ultrasonography imagery, and/or radar imagery for missile tracking) may be accompanied by plasticity rules, wherein the relevant scales are scaled in accordance with, for example, input stimulus refresh rate, or some other useful parameter or metric. In some implementations, the pre-synaptic rule portion 324 of the rule 320 may be characterized by depression magnitude 326 that may be configured to be e.g., comparable to the magnitude 306 of rule 300. In one or more implementations, the potentiation magnitude 328 of rule 320 may be configured greater than the potentiation magnitude 308 of rule 300 of FIG. 3A. In some implementations, an integral under the rule 324 may be configured equal (or on the order of) integral under the rule 304 of FIG. 3A.

[0082] FIG. 3C illustrates an exemplary plasticity rule 340 comprising a heterogeneous post-synaptic rule portion 344, in accordance with one or more implementations. The post-synaptic rule 344 may comprise a potentiation region 352 and depression region 350. When processing video input, comprising for example 25 frames-per-second pixel stream, the width 350 of the depression region may be selected between 5 ms and 20 ms. As it will be appreciated by those skilled in the art, sensory input of different temporal scales (e.g., ultrasonography imagery, and/or radar imagery for missile tracking) may be accompanied by plasticity rules where the relevant scales are scaled in accordance with, for example, input stimulus refresh rate. In some implementations.

[0083] FIG. 4 illustrates a homogeneous sensory processing spiking neuron network 400, in accordance with one or more implementations. The network 400 may comprise a plurality of neurons 402 that may receive sensory input 404. Individual neurons (e.g., the neuron 402_1) may generate post-synaptic response(s) that may be communicated via one or more inhibitory connections 406 to one or more other neurons 402_2 of the network, depicted by the dotted curve 412 in FIG. 4. The neurons of the network portion 412 may generate post-synaptic responses based on the feed-forward sensory input 404 and the inhibitory input 406. The responses of the network portion 412 may be communicated to the neuron 402_1 via one or more inhibitory connections 414.

[0084] The inhibitory connections 406, 414 of the network 400 of FIG. 4 may be operable in accordance with one or more activity-dependent plasticity rules, such as the rules 300, 320, 340 described above with respect to FIGS. 3A-3C, respectively.

[0085] In some implementations, the sensory input into network (e.g., the input 104 in FIG. 1, and/or the input 404 in FIG. 4) may comprise unevenly distributed (in frequency of occurrence) features, such as illustrated in FIG. 5A. According to one or more implementations, the input 500 of FIG. 5A may represent, for example, a photo and/or video footage of a winter landscape with predominantly vertical live trees 502, 504, 506 and fewer fallen horizontal trees 512, 514. In some prior art plasticity implementations, the input 500 may cause prior art network comprising, for example, 100 neurons to develop receptive field (RF) one configured to respond to vertical features 502, 504, 506; and another RF configured to respond to horizontal features 512, 514. By way of illustration, 60 neurons may develop vertical selectivity, 30 may develop horizontal selectivity while 10 may respond to other features, The network operated according to the prior art, the neuron selectivity may fall in the range between −45° to +45° s from the preferred orientation. Accordingly, when the input of FIG. 5A is provided to such network many more (30) vertical selectivity neurons will fire (due to a greater portion of near-vertical features) compared to the horizontal selectivity neurons.

[0086] The activity-based plasticity adjustment mechanism of FIGS. 2-3C implemented by networks 100, 400 of FIGS. 1, 4 described above, may advantageously provide for further selectivity between neurons that develop receptive fields for vertically orientated features. FIG. 5B illustrates receptive field selectivity obtained using activity-dependent plasticity in accordance with one or more implementations. The curves 532,536 denote neuron receptive fields configured to respond to features with a center axis oriented along the arrows denoted 530, 532, respectively in FIG. 5B. The receptive field 532 may correspond to features in the input 500 that are oriented within ±3° from the vertical (e.g., feature 502 in FIG. 5A). The receptive field 536 may correspond to features in the input 500 that are oriented within ±3° from orientation of 5° off vertical (e.g., feature 506 in FIG. 5A). That is, different neurons may develop to respond to different ‘vertical orientations’ in the input 500 when the network is operated in accordance with the activity dependent plasticity.

[0087] The curve 520 denotes neuron receptive field configured to respond to features oriented within a sector of orientations denoted by the arrow 526 and centered at the orientation denoted by the horizontal arrow 522 in FIG. 5B. The receptive field 520 may correspond to features in the input 500 that are oriented within ±45° from the horizontal (e.g., features 512, 514 in FIG. 5A). That is, a single neuron may develop to respond to different ‘horizontal orientations’ in the input 500.

[0088] The selectivity mechanism of the disclosure may further be illustrated as follows. While the network of the prior art may respond more frequently to vertical features, as described above, the selectivity mechanism of the disclosure equalizes responses occurrence between horizontally and vertically selective neurons when processing, for example, the input of FIG. 5A. The equalization may be attained by, at least partly, increasing selectivity (e.g., reducing the RF angular extent) of the neurons responding to more frequent features (e.g., vertical in FIG. 5A) so that these vertical neurons may be tuned to, for example, ±15°.

[0089] FIG. 5C illustrates equalization of neuron responses to input stimulus comprising features of different occurrence frequency using the plasticity mechanism described above with respect to FIG. 1 and/or FIG. 4. Panel 540 in FIG. 5C illustrates responses of a neuron network of prior art, comprising neurons 542, 544 configured to respond to vertical and horizontal features, respectively. As the vertical features (e.g., the features 504, 504) are more prevalent (compared to the horizontal features) in the input 500 of FIG. 5A, the response rate of the neurons 542 is greater than the response rate of the ‘horizontal’ neurons 544, as depicted by the taller ‘vertical’ rectangle 542 compared to the shorter ‘horizontal’ rectangle 544 in FIG. 5C.

[0090] Panel 550 in FIG. 5C illustrates responses of a neuron network configured in accordance with the activity-based plasticity mechanism configured to implement rate equalization. The rectangles 552, 554 in panel 550 depict responses of neurons to vertical and horizontal features, respectively. The exemplary embodiment of the plasticity mechanism of the disclosure is capable of reducing response rate of the neurons that respond to more prevalent features (e.g., the response 552 to vertical features in FIG. 5C), as compared to the network of the prior art (e.g., the response 552). In some implementations, the rate reduction between the responses 542 and 552 may be for instance in the range from 2-5 times. At the same time, the response rate of the neurons that respond to less frequent features (e.g., the horizontal features in FIG. SC) may not be affected substantially. In some implementations, the response rate of less frequent features (e.g., 554) may be for instance in the range between 0.8 and 1.2 of the response rate of the prior art 544. As a result of the rate reduction of the vertically responding neurons, the network selectivity may be increased, as described in detail with respect to FIG. 5B, above.

[0091] Referring now to FIGS. 6A-9B, exemplary implementations of adaptive plasticity methods according to the disclosure are described. In some implementations, the methods of FIGS. 6A-9B may be used, for example, for operating connections 106, 114 of FIG. 1A and/or connections 416, 414 FIG. 4. The exemplary methods of FIGS. 6A-9B may be implemented by neurons (e.g., the neurons 102, and/or 112 of FIG. 1 and/or neurons 402 of FIG. 4). The methods of FIGS. 6A-9B may be implemented in sensory processing apparatus, comprising one or more spiking neuron networks as described with respect to FIG. 10, infra, thereby advantageously aiding, inter cilia improving learning, potentially reducing a number of spikes required to represent a given input, thereby enabling data compression.

[0092] FIG. 6A illustrates an activity-dependent method of connection efficacy update, in accordance with one or more implementations. The method 600 of FIG. 6A may be utilized in, for example, a homogeneous network comprising inhibitory neurons (e.g., the network 400 of FIG. 4) for updating connections of a given neuron of the network (e.g., the neuron 402_1 of the network 400 of FIG. 4). The method 600 of FIG. 6A may be utilized in, for example, a heterogeneous network for updating connections of inhibitory units of the network (e.g., the units 112 of the network 100 of FIG. 1).

[0093] At step 602, a determination may be made whether the update is to be performed. In one or more implementations, the update may be based on a post-synaptic spike by the neuron. The update may be based for example on an external event (e.g., reinforcement signal); a timer event (e.g., for cyclic updates); or a buffer overflow event (e.g., indicative of a memory buffer, storing, for example, pre-synaptic and/or post-synaptic spike history) being full or nearly full. In some implementations, the history may comprise time data of pre-synaptic spikes stored in a synaptic memory buffer, such as described for example in U.S. patent application Ser. No. 13/239,259, entitled “APPARATUS AND METHOD FOR PARTIAL EVALUATION OF SYNAPTIC UPDATES BASED ON SYSTEM EVENTS”, filed on Sep. 21, 2011, incorporated supra.

[0094] When the update is to be performed, the method may proceed to step 608, where a post-synaptic update may be performed. In one or more implementations, the post-synaptic update may comprise e.g., one or more post-synaptic rule portions 304, 324, 344 illustrated in FIGS. 3A-3C.

[0095] When the update of step 602 is not to be performed, the method may proceed to step 604, wherein a determination may be made as to whether pre-synaptic activity has occurred. In one or more implementations, the pre-synaptic activity may comprise input from an inhibitory unit, such as the input 114_1 from the unit 112_1 in FIG. 1.

[0096] When the pre-synaptic activity is present, the method may proceed to step 608, where pre-synaptic update may be performed. In one or more implementations, the pre-synaptic update may comprise e.g., one or more pre-synaptic rule portions 302, 322, 342 illustrated in FIGS. 3A-3C.

[0097] FIG. 6B illustrates a method of connection update for use with inhibitory units of a network, in accordance with one or more implementations. The method 620 of FIG. 6B may be utilized in, for example, a homogeneous network comprising inhibitory neurons (e.g., the network 400 of FIG. 4). The method 620 of FIG. 6A may be utilized in, for example, a heterogeneous network for updating connections of the inhibitory units of the network (e.g., the units 112 of the network 100 of FIG. 1).

[0098] At step 622, a determination may be made whether the neuron response has been generated. In some implementations, the neuron and the network may correspond to the neuron 402_1 and the network portion 412 of FIG. 4. In some implementations, the neuron and the network may correspond to neurons 102 and 112 of FIG. 4, respectively.

[0099] When the post-synaptic response has been generated based, for example, on excitatory 404 and/or inhibitory input 414 in FIG. 4, a post synaptic plasticity rule may be evaluated at step 626. In one or more implementations, the post-synaptic rule may cause an increase of efficacy of connections providing inhibitory input into the post-synaptic neuron. In some implementations, the connections may correspond to connections 114 of FIG. 1 and/or connections 414 of FIG. 4. In one or more implementations, the post-synaptic Wile may be described by the curves 304, 324, 344 of FIGS. 3A-3C.

[0100] When the post-synaptic response has not been generated, the method may proceed to step 624 where a determination may be made as to whether pre-synaptic activity has occurred. In one or more implementations, the pre-synaptic activity may comprise input from an inhibitory unit, such as the input 114_1 from the unit 112_1 in FIG. 1.

[0101] When the inhibitory input activity is present (e.g., from another inhibitory unit), the method may proceed to step 628, where pre-synaptic update may be performed. In one or more implementations, the pre-synaptic rule may be configured to reduce inhibition onto the post-synaptic unit; e.g., the pre-synaptic rule may comprise reduction of inhibitory efficacy of the connection 114_1, and be described by the curves 302, 322, 342 of FIGS. 3A-3C.

[0102] FIGS. 7A-7B illustrate exemplary methods of implementing activity-dependent plasticity in a heterogeneous network, in accordance with one or more implementations. The approaches of methods 700, 730 may be utilized with the network 100 of FIG. 1, comprising excitatory (102) and inhibitory (112) portions.

[0103] FIG. 7A illustrates an exemplary plasticity update for an inhibitory neuron (e.g., the neuron 112_1 of FIG. 1), in accordance with one or more implementations. At step 732, a determination may be made whether the post-synaptic response has been generated by the inhibitory neuron.

[0104] When the post-synaptic response has been generated based, for example, on excitatory input 106_1 of FIG. 1, a post synaptic plasticity rule may be evaluated at step 736. In one or more implementations, the post-synaptic rule may cause (i) increase in efficacy of one or more connections providing inhibitory stimulus to excitatory neurons (e.g., the connection 114_1 in FIG. 1); and (ii) increase in efficacy of one or more connections providing excitatory stimulus input to the post-synaptic neuron (e.g., the connection 106_1 in FIG. 1). In some implementations, the inhibitory efficacy increase may comprise one or more rules 304, 324, 344 of FIGS. 3A-3C. In one or more implementations, the excitatory efficacy increase may comprise the rule 222 of FIG. 2.

[0105] When the post-synaptic response has not been generated, the method 700 may proceed to step 734, wherein a determination may be made as to whether pre-synaptic activity has occurred. In one or more implementations, the pre-synaptic activity may comprise an excitatory stimulus from an excitatory unit, such as the input 106_1 from the unit 102_1 in FIG. 1.

[0106] When the excitatory pre-synaptic activity is present, method 700 may proceed to step 738, where pre-synaptic update may be performed. In one or more implementations, the pre-synaptic update may be configured to cause: (i) a decrease of efficacy of one or more connections providing inhibitory stimulus to excitatory neurons (e.g., the connection 114_1 in FIG. 1); and (ii) decrease in efficacy of one or more connections providing excitatory stimulus input to the post-synaptic neuron (e.g., the connection 106_1 in FIG. 1). In some implementations, the inhibitory efficacy change may comprise one or more rules 302, 322, 342 of FIGS. 3A-3C. In one or more implementations, the excitatory efficacy change may comprise the rule 224 of FIG. 2.

[0107] FIG. 7B illustrates plasticity update for an excitatory neuron (e.g., the neuron 102_1 of FIG. 1), in accordance with one or more implementations. At step 752 of method 750, a determination may be made whether the post-synaptic response has been generated by the excitatory neuron.

[0108] When the post-synaptic response has been generated based, for example, on excitatory 104 and/or inhibitory input 114 of FIG. 1, a post synaptic plasticity rule may be evaluated at step 756. In one or more implementations, the post-synaptic rule may cause (i) a decrease of efficacy of one or more connections providing inhibitory input into the post-synaptic neuron (e.g., the connection 114_1 in FIG. 1); and (ii) a decrease in efficacy of one or more connections providing excitatory stimulus to inhibitory neurons (e.g., the connection 106_1 in FIG. 1). In some implementations, the inhibitory efficacy change may comprise one or more rules 302, 322, 342 of FIGS. 3A-3C. In one or more implementations, the excitatory efficacy decrease may comprise the rule 224 of FIG. 2.

[0109] When the post-synaptic response has not been generated by the excitatory neuron, method 750 may proceed to step 754, where a determination may be made as to whether pre-synaptic activity has occurred. In one or more implementations, the pre-synaptic activity may comprise input from an inhibitory unit, such as the input 114_1 from the unit 112_1 in FIG. 1.

[0110] When the inhibitory pre-synaptic activity is present (e.g., the inhibitory unit 112_1 has generated a response), the method 750 may proceed to step 758, where pre-synaptic update may be performed. In one or more implementations, the pre-synaptic update may be configured to cause: (i) an increase of efficacy of one or more connections providing inhibitory stimulus to the post-synaptic unit (e.g., the connection 114_1 in FIG. 1); and (ii) an increase in efficacy of one or more connections providing excitatory stimulus from the post-synaptic unit to inhibitory units (e.g., the connection 106_1 in FIG. 1). In some implementations, the inhibitory efficacy increase may comprise one or more rules 304, 324, 344 of FIGS. 3A-3C. In one or more implementations, the excitatory efficacy increase may comprise the rule 222 of FIG. 2.

[0111] FIG. 8 illustrates method of operating connections of a heterogeneous network (comprising for example excitatory 106 and inhibitory 114 connections of FIG. 1), in accordance with one or more implementations.

[0112] At step 802 of method 800, a determination may be made as to whether an inhibitory connection is to be updated.

[0113] If the inhibitory connection is to be updated, the method may proceed to step 804, wherein a determination may be made if a pre-synaptic rule is to be applied. In some implementations, the determination of step 802 may be based on an evaluation of the time of the most-recent post-synaptic response (t_post) by excitatory neurons 102 and time (t_pre) of pre-synaptic input from inhibitory neurons 112 preceding t_post.

[0114] When it may be determined at step 804 that the pre-synaptic input from inhibitory neurons arrives after the excitatory neuron has generated a response (t_pre>t_post), the method may proceed to step 806.

[0115] At step 806, a pre-synaptic rule may be evaluated. In one or more implementations, the pre-synaptic rule may be configured to decrease efficacy of the inhibitory connection.

[0116] When it may be determined at step 804 that the excitatory neuron has generated a response subsequent to the occurrence of pre-synaptic input from inhibitory neuron(s) (t_pre<t_post), the method may proceed to step 808.

[0117] At step 808, a post-synaptic rule may be evaluated. In one or more implementations, the post-synaptic rule may be configured to increase efficacy of the inhibitory connection. In one or more implementations, the pre-synaptic and/or the post-synaptic rule of the adjustment of steps 806, 808 may comprise one or more plasticity rules 300, 320, 340 of FIGS. 3A-3C. In one or more implementations, the magnitude of the inhibitory efficacy potentiation of step 808 may be greater that the magnitude of the efficacy reduction of step 806.

[0118] When it is determined at step 802 that the excitatory connection is to be updated, the method may proceed to step 810, wherein a determination may be made as to whether a pre-synaptic rule is to be applied. In some implementations, the determination of step 802 may be based on an evaluation of the time of the most-recent post-synaptic response (t_post) by inhibitory neurons 102 and time (t_pre) of the pre-synaptic input from excitatory neurons 112 preceding t_post.

[0119] When it may be determined at step 810 that the pre-synaptic input from exitatory neurons arrives after the inhibitory neuron has generated a response (t_pre>t_post), the method may proceed to step 812.

[0120] At step 812, a pre-synaptic rule may be evaluated. In one or more implementations, the pre-synaptic rule may be configured to increase efficacy of the excitatory connection.

[0121] When it may be determined at step 810 that the inhibitory neuron has generated a response subsequent to the occurrence of pre-synaptic input from excitatory neuron(s) (t_pre<t_post), the method may proceed to step 814.

[0122] At step 814, a post-synaptic rule may be evaluated. In one or more implementations, the post-synaptic rule may be configured to decrease efficacy of the excitatory connection. In one or more implementations, the pre-synaptic and/or the post-synaptic rule of the adjustment of steps 806, 808 may comprise the plasticity rule 220 of FIGS. 2. In one or more implementations, the magnitude of the excitatory efficacy potentiation of step 812 may be greater that the magnitude of the efficacy reduction of step 814,

[0123] FIG. 9A illustrates a method of equalizing rate of response by neurons of a spiking network via activity-dependent plasticity, in accordance with one or more implementations.

[0124] At step 902 of method 900, an analysis of network response to input stimulus may be performed. In one or more implementations, the input stimulus may comprise unevenly distributed (i.e., in frequency of occurrence) features, such as the input 500 of FIG. 5A. In some implementations, the analysis may comprise analysis of inhibitory neuron activity, such as the neurons 402 of the network 400 described above with respect to FIG. 4. In one or more implementations, the analysis may comprise analysis of activity of inhibitory and/or excitatory neurons, such as the neurons 112, 102, respectively, of the network 100 described above with respect to FIG. 1. In some implementations, the analysis may comprise analysis of activity of inhibitory and/or excitatory connections, such as the connections 112, 412 and 102 of FIGS. 1, 4. In some implementations, the response analysis may comprise an evaluation of the time of the most-recent post-synaptic response (t_post) by excitatory neurons 102 and time (t_post) of pre-synaptic input from inhibitory neurons 112 preceding t_post. In some implementations, the activity evaluation may comprise a comparison between the time of the most-recent post-synaptic response (t_post) by inhibitory neurons 112 and time (t_pre) of pre-synaptic input from excitatory neurons 102 preceding t_post.

[0125] At step 904, responses by individual neurons of the network may be equalized. In some implementations, activity of neurons responding to more frequently occurring features may be reduced using activity dependent plasticity. By a way of a non-limiting illustration, a neuron (e.g., the neuron 402_1 in FIG. 4) may respond with an output rate R1 to horizontal features 512 in FIG. 5A. One or more neurons (e.g., the neuron 402_2 in FIG. 4) may respond with an output rate R2 to vertical features 502 in FIG. 5A. When the neurons 402_1, 402_2 generate responses, efficacy of inhibitory connections 414, 406, respectively in FIG. 4, providing inhibitory input to 402_1, 402_2 gets increased. As the neuron 402_2 responds more frequently (due to a greater prevalence of vertical features in the input 500) the connection 406 becomes stronger compared to the connection 414 of the neuron 402_1. Therefore, the inhibitory input into the neuron 402_2 may exceed the inhibitory input into the neuron 402_1. When the inhibition into the ‘vertical’ neuron 402_2 becomes sufficiently strong (e.g., reaches inhibitory threshold, and/or reduces neuron excitability below firing threshold), the neuron 402_2 may not become inhibited and not respond to the input stimulus (e.g., the stimulus 404 in FIG. 4). Accordingly, the firing rate of the neuron 402_2 may become lower. In some implementations, the plasticity mechanism may cause reduction of the firing rate of the neuron 402_2 responding to vertical features, to the firing rate of the neuron 401_1 responding to horizontal features. In some implementations, the inhibition of the neuron 402_2 may enable other neurons of the partition 412 to begin responding to vertical features of slightly different orientation (e.g., the feature 506 oriented at 5°) compared to the feature 502 associated with the neuron 402_2 response. As a result, individual neurons of the partition 412 may develop further selectivity of their receptive fields to distinct vertical features, as illustrated by the receptive fields 532, 536 in FIG. 5B.

[0126] FIG. 9B illustrates a method of neuron firing rate equalization via activity-dependent plasticity, in accordance with one or more implementations. The method 920 of FIG. 9B may be utilized for instance in a network 400 comprising a plurality of inhibitory neurons and connections.

[0127] At step 922, the activity of a neuron may be compared to activity of one or more neuron of the network. In one or more implementations, the activity evaluation may comprise determination of post-synaptic responses by individual neurons.

[0128] When the neuron responds more frequently (e.g., characterized by higher firing rate), a plasticity mechanism that is configured to reduce neuron excitability may be applied at step 922. In some implementations, the plasticity mechanism may comprise potentiation of inhibitory connections into the neuron in accordance with, for example, rules 304, 324, 344 of FIGS. 3A-3C, respectively.

[0129] When the neuron responds less frequently (e.g., characterized by lower firing rate), a plasticity mechanism that is configured to increase neuron excitability may be applied at step 924. In some implementations, the plasticity mechanism may comprise depression of inhibitory connections into the neuron in accordance with, for example, rules 302, 322, 342 of FIGS. 3A-3C, respectively. In one or more implementations, increase in excitability may be achieved by reduction of inhibition.

[0130] Various exemplary spiking network apparatus implementing one or more of the methods set forth herein (e.g., using the exemplary activity dependent plasticity mechanisms described above) are now described with respect to FIGS. 10-11D.

[0131] One apparatus for processing of sensory information (e.g., visual, audio, somatosensory) using a spiking neural network (including one or more of the activity dependent plasticity mechanisms described herein) is shown in FIG. 10. The illustrated processing apparatus 1000 includes an input interface configured to receive an input sensory signal 1010. In some implementations, this sensory input comprises electromagnetic waves (e.g., visible light, IR, UV, and/or other wavelength) entering an imaging sensor array (comprising RGCs, a charge coupled device (CCD), CMOS device, or an active-pixel sensor (APS)). The input signal in this example is a sequence of images (image frames) received from a CCD or a CMOS camera via a receiver apparatus, or downloaded from a file. Alternatively, the image may be a two-dimensional matrix of RGB values refreshed at a 25 Hz frame rate. It will be appreciated by those skilled in the art that the above image parameters and components are merely exemplary, and many other image representations (e.g., bitmap, CMYK, grayscale, and/or another implementation) and/or frame rates are equally useful with the present disclosure.

[0132] The apparatus 1000 may comprise an encoder 1020 configured to transform (encode) the input signal so as to form an encoded signal 1024. In one variant, the encoded signal comprises a plurality of pulses (also referred to as a group of pulses) configured to model neuron behavior. The encoded signal 1026 may be communicated from the encoder 1024 via multiple connections (also referred to as transmission channels, communication channels, or synaptic connections) 1004 to one or more neuronal nodes (also referred to as the detectors) 1002.

[0133] In the implementation of FIG. 10, different detectors of the same hierarchical layer are denoted by an “_n” designator, such that e.g., the designator 1002_1 denotes the first detector of the layer 1002. Although only two detectors (1002_1, 1002_—n) are shown in FIG. 10 for clarity, it is appreciated that the encoder can be coupled to any number of detector nodes that is compatible with the detection apparatus hardware and software limitations. Furthermore, a single detector node may be coupled to any practical number of encoders.

[0134] In one implementation, each of the detectors 1002_1, 1002_—n contain logic (which may be implemented as a software code, hardware logic, or a combination of thereof) configured to recognize a predetermined pattern of pulses in the encoded signal 1004, using for example any of the mechanisms described in U.S. patent application Ser. No. 12/869,573, filed Aug. 26, 2010 and entitled “SYSTEMS AND METHODS FOR INVARIANT PULSE LATENCY CODING”, U.S. patent application Ser. No. 12/869,583, filed Aug. 26, 2010, entitled “INVARIANT PULSE LATENCY CODING SYSTEMS AND METHODS”, U.S. patent application Ser. No. 13/117,048, filed May 26, 2011 and entitled “APPARATUS AND METHODS FOR POLYCHRONOUS ENCODING AND MULTIPLEXING IN NEURONAL PROSTHETIC DEVICES”, U.S. patent application Ser. No. 13/152,084, filed Jun. 2, 2011, entitled “APPARATUS AND METHODS FOR PULSE-CODE INVARIANT OBJECT RECOGNITION”, each incorporated herein by reference in its entirety, to produce post-synaptic detection signals transmitted over communication channels 1008. In FIG. 10, the designators 1008_1, 1008_—n denote output of the detectors 1002_1, 1002_—n, respectively.

[0135] In one implementation, the detection signals are delivered to a next layer of the detectors 1012 (comprising detectors 1012_1, 1012_—m, 1012_—k) for recognition of complex object features and objects, similar to the exemplary configuration described in commonly owned and co-pending U.S. patent application Ser. No. 13/152,084, filed Jun. 2, 2011, entitled “APPARATUS AND METHODS FOR PULSE-CODE INVARIANT OBJECT RECOGNITION”, incorporated herein by reference in its entirety. In this configuration, each subsequent layer of detectors is configured to receive signals from the previous detector layer, and to detect more complex features and objects (as compared to the features detected by the preceding detector layer). For example, a bank of edge detectors is followed by a bank of bar detectors, followed by a bank of corner detectors and so on, thereby enabling alphabet recognition by the apparatus.

[0136] Each of the detectors 1002 may output detection (post-synaptic) signals on communication channels 1008_1, 1008_—n (with appropriate latency) that may propagate with different conduction delays to the detectors 1012. The detector cascade of the apparatus of FIG. 10 may contain any practical number of detector nodes and detector banks determined, inter alia, by the software/hardware resources of the detection apparatus, and/or complexity of the objects being detected.

[0137] The sensory processing apparatus implementation illustrated in FIG. 10 may further comprise lateral connections 1006. In some variants, the connections 1006 are configured to communicate post-synaptic activity indications between neighboring neurons of the same hierarchy level, as illustrated by the connection 1006_1 in FIG. 10. In some variants, the neighboring neuron may comprise neurons having overlapping inputs (e.g., the inputs 1004_1, 1004_—n in FIG. 10), so that the neurons may compete in order to not learn the same input features. In one or more implementations, the neighboring neurons may comprise spatially proximate neurons such as being disposed within a certain volume/area from one another on a 3-dimensional (3D) and or two-dimensional (2D) space.

[0138] The apparatus 1000 may also comprise feedback connections 1014, configured to communicate context information from detectors within one hierarchy layer to previous layers, as illustrated by the feedback connections 1014_1 in FIG. 10. In some implementations, the feedback connection 1014_2 is configured to provide feedback to the encoder 1024, thereby facilitating sensory input encoding, as described in detail in commonly owned and co-pending U.S. patent application Ser. No. 13/152,084, filed Jun. 2, 2011, entitled “APPARATUS AND METHODS FOR PULSE-CODE INVARIANT OBJECT RECOGNITION”, incorporated supra.

[0139] One particular implementation of the computerized neuromorphic processing system, adapted for operating a computerized spiking network (and implementing the exemplary plasticity methodology described supra), is illustrated in FIG. 11A. The computerized system 1100 of FIG. 11A comprises an input interface 1110, such as for example an image sensor, a computerized spiking retina, an audio array, a touch-sensitive input device, and/or another I0 device. The input interface 1110 is coupled to the processing block (e.g., a single or multi-processor block) via the input communication interface 1114. The system 1100 further comprises a random access memory (RAM) 1108, configured to store neuronal states and connection parameters (e.g., weights 108 in FIG. 1A), and to facilitate synaptic updates. In some exemplary implementations, synaptic updates are performed according to the description provided in, for example, in U.S. patent application Ser. No. 13/239,255 filed Sep. 21, 2011, entitled “APPARATUS AND METHODS FOR SYNAPTIC UPDATE IN A PULSE-CODED NETWORK”, incorporated by reference supra.

[0140] In some implementations, the memory 1108 is coupled to the processor 1102 via a direct connection (memory bus) 1116. The memory 1108 may also be coupled to the processor 1102 via a high-speed processor bus 1112).

[0141] The system 1100 may further comprise a nonvolatile storage device 1106, comprising, inter alia, computer readable instructions configured to implement various aspects of spiking neuronal network operation (e.g., sensory input encoding, connection plasticity, operation model of neurons, and/or other processing functions). The nonvolatile storage 1106 may be used for instance to store state information of the neurons and connections when, for example, saving/loading network state snapshot, or implementing context switching (e.g., saving current network configuration (comprising, inter alia, connection weights and update rules, neuronal states and learning rules, and/or other parameters) for later use, and/or loading of a previously stored network configuration.

[0142] In some implementations, the computerized apparatus 1100 is coupled to one or more external processing/storage/input devices via an I/O interface 1120, such as a computer I/O bus (PCI-E), wired (e.g., Ethernet) or wireless (e.g., Wi-Fi) network connection.

[0143] In another variant, the input/output interface comprises a speech input (e.g., a microphone) and a speech recognition module configured to receive and recognize user commands.

[0144] It will be appreciated by those skilled in the arts that various processing devices may be used with computerized system 1100, including but not limited to, a single core/multicore CPU, DSP, FPGA, GPU, ASIC, combinations thereof, and/or other processors. Various user input/output interfaces are similarly applicable to implementations of the disclosure including, for example, an LCD/LED monitor, touch-screen input and display device, speech input device, stylus, light pen, trackball, end the likes.

[0145] Referring now to FIG. 11B, one implementation of neuromorphic computerized system configured to implement modulated plasticity mechanism in a spiking network is described in detail. The neuromorphic processing system 1130 of FIG. 11B comprises a plurality of processing blocks (micro-blocks) 1140, where each micro-block comprises a computing logic core 1132 and a memory block 1134. The logic core 1132 is configured to implement various aspects of neuronal node operation, such as the node model, and synaptic update rules and/or other tasks relevant to network operation. The memory block is configured to store, inter alia, neuronal state variables and connection parameters (e.g., weights, delays, I/O mapping) of connections 1138.

[0146] The micro-blocks 1140 are interconnected with one another using connections 1138 and routers 1136. As it is appreciated by those skilled in the arts, the connection layout in FIG. 11B is exemplary, and many other connection implementations (e.g., one to all, all to all, and/or another mapping) are compatible with the disclosure.

[0147] The neuromorphic apparatus 1130 is configured to receive input (e.g., visual input) via the interface 1142. In one or more implementations, applicable for example to interfacing with a computerized spiking retina or an image array, the apparatus 1130 may provide feedback information via the interface 1142 to facilitate encoding of the input signal.

[0148] The neuromorphic apparatus 1130 is configured to provide output (e.g., an indication of recognized object or a feature, or a motor command, e.g., to zoom/pan the image array) via the interface 1144.

[0149] The apparatus 1130, in one or more implementations, may interface to external fast response memory (e.g., RAM) via high bandwidth memory interface 1148, thereby enabling storage of intermediate network operational parameters (e.g., spike timing, and/or other parameters). The apparatus 1130 may also interface to external slower memory (e.g., Flash, or magnetic (hard drive)) via lower bandwidth memory interface 1146, in order to facilitate program loading, operational mode changes, and retargeting, where network node and connection information for a current task is saved for future use and flushed, and previously stored network configuration is loaded in its place,

[0150] FIG. 11C illustrates one or more implementations of shared bus neuromorphic computerized system 1145 comprising micro-blocks 1140, described with respect to FIG. 11B, supra. The system 1145 of FIG. 11C may utilize shared bus 1147, 1149 to interconnect micro-blocks 1140 with one another.

[0151] FIG. 11D illustrates one implementation of cell-based neuromorphic computerized system architecture configured to implement activity based plasticity mechanism in a spiking network. The neuromorphic system 1150 of FIG. 11D comprises a hierarchy of processing blocks (cells block). In some implementations, the lowest level L1 cell 1152 of the apparatus 1150 may comprise logic and memory, and may be configured similar to the micro block 1140 of the apparatus shown in FIG. 11B. A number of cell blocks may be arranges in a cluster and communicate with one another a local interconnects 1162, 1164. Each such cluster may form a higher-level cell, e.g., cell L2, denoted as 1154 in FIG. 11D. Similarly, several L2 clusters may communicate with one another via a second-level interconnect 1166 and form a super-cluster L3, denoted as 1156 in FIG. 11D. The super-clusters 1154 may for example communicate via a third level interconnect 1168, and may form a next level cluster, and so on. It will be appreciated by those skilled in the arts that the hierarchical structure of the apparatus 1150, comprising a given number (e.g., four) cells per level, is merely one exemplary implementation, and other implementations may comprise more or fewer cells per level, and/or fewer or more levels, as well as yet other types of architectures.

[0152] Different cell levels (e.g., L1, L2, L3) of the exemplary apparatus 1150 of FIG. 11D may be configured to perform functionality with various levels of complexity. In one implementation, different L1 cells may process in parallel different portions of the visual input (e.g., encode different frame macro-blocks), with the L2, L3 cells performing progressively higher-level functionality (e.g., edge detection, object detection). Different L2, L3 cells may also perform different aspects of operating for example a robot, with one or more L2/L3 cells processing visual data from a camera, and other L2/L3 cells operating a motor control block for implementing lens motion when e.g., tracking an object, or performing lens stabilization functions.

[0153] The neuromorphic apparatus 1150 may receive input (e.g., visual input) via the interface 1160. In one or more implementations, applicable for example to interfacing with a computerized spiking retina or image array, the apparatus 1150 may provide feedback information via the interface 1160 to facilitate encoding of the input signal.

[0154] The neuromorphic apparatus 1150 may provide output (e.g., an indication of recognized object or a feature, or a motor command, e.g., to zoom/pan the image array) via the interface 1170. In some implementations, the apparatus 1150 may perform all of the I/O functionality using single I/O block (not shown).

[0155] The apparatus 1150, in one or more implementations, may also interface to external fast response memory (e.g., RAM) via high bandwidth memory interface (not shown), thereby enabling storage of intermediate network operational parameters (e.g., spike timing, and/or other parameters). The apparatus 1150 may also interface to external slower memory (e.g., flash, or magnetic (hard drive)) via lower bandwidth memory interface (not shown), in order to facilitate program loading, operational mode changes, and retargeting, where network node and connection information for a current task is saved for future use and flushed, and a previously stored network configuration is loaded in its place.

[0156] In one or more implementations, networks of the apparatus 1130, 1145, 1150 may be implemented using the Elementary Network Description (END) language, described for example in U.S. patent application Ser. No. 13/239,123, entitled “ELEMENTARY NETWORK DESCRIPTION FOR NEUROMORPHIC SYSTEMS”, filed Sep. 21, 2011, and/or the High Level Neuromorphic Description (HLND) framework, described for example in U.S. patent application Ser. No. 13/385,938, entitled “TAG-BASED APPARATUS AND METHODS FOR NEURAL NETWORKS”, filed Mar. 15, 2012, each of the foregoing being incorporated herein by reference in its entirety. In one or more implementations, the HLND framework may be augmented to handle event based update methodology described, for example U.S. patent application Ser. No. 13/588,774, entitled “APPARATUS AND METHODS FOR IMPLEMENTING EVENT-BASED UPDATES IN SPIKING NEURON NETWORK”, filed Aug. 17, 2012, the foregoing being incorporated herein by reference in its entirety. In some implementations, the networks may be updated using an efficient network update methodology, described, for example, in U.S. patent application Ser. No. 13/385,938, entitled “APPARATUS AND METHODS FOR EFFICIENT UPDATES SPIKING NEURON NETWORKS”, filed Jul. 27, 2012, the foregoing being incorporated herein by reference in its entirety.

[0157] The plasticity mechanism described herein may advantageously improve the quality of the feature learning (as may be characterized by a number of false positives, and/or a number of identifiable features), and/or to increase neuron receptive selectivity.

[0158] Various aspects of the disclosure may advantageously be applied to, inter alia, the design and operation of large spiking neural networks configured to process streams of input stimuli, in order to aid in detection and functional binding related aspect of the input.

[0159] In some implementations, activity-based and/or plasticity modulation mechanisms described herein may be implemented in a spiking neuron of a network, or in a connection of the network.

[0160] It is appreciated by those skilled in the arts that above implementation are exemplary, and the framework of the disclosure is equally compatible and applicable to processing of other information, such as, for example information classification using a database, where the detection of a particular pattern can be identified as a discrete signal similar to a spike, and where coincident detection of other patterns influences detection of a particular one pattern based on a history of previous detections in a way similar to an operation of exemplary spiking neural network.

[0161] Advantageously, exemplary implementations of the various aspects of the present innovation are useful in a variety of devices including without limitation prosthetic devices, autonomous and robotic apparatus, and other electromechanical devices requiring sensory processing functionality. Examples of such robotic devises are manufacturing robots (e.g., automotive), military, medical (e.g. processing of microscopy, x-ray, ultrasonography, tomography). Examples of autonomous vehicles include rovers, unmanned air vehicles, underwater vehicles, smart appliances (e.g. ROOMBA®), Lego® robotic toys, and/or other devices.

[0162] Implementations of the principles of the disclosure are applicable to video data compression and processing in a wide variety of stationary and portable devices, such as, for example, smart phones, portable communication devices, notebook, netbook and tablet computers, surveillance camera systems, and practically any other computerized device configured to process vision data.

[0163] Implementations of the principles of the disclosure are further applicable to a wide assortment of applications including computer human interaction (e.g., recognition of gestures, voice, posture, face, and/or other aspects), controlling processes (e.g., an industrial robot, autonomous and other vehicles), augmented reality applications, organization of information (e.g., for indexing databases of images and image sequences), access control (e.g., opening a door based on a gesture, opening an access way based on detection of an authorized person), detecting events (e.g., for visual surveillance or people or animal counting, tracking), data input, financial transactions (payment processing based on recognition of a person or a special payment symbol) and many others.

[0164] Advantageously, the disclosure can be used to simplify tasks related to motion estimation, such as where an image sequence is processed to produce an estimate of the object position (and hence velocity) either at each points in the image or in the 3D scene, or even of the camera that produces the images. Examples of such tasks are: ego motion, i.e., determining the three-dimensional rigid motion (rotation and translation) of the camera from an image sequence produced by the camera; following the movements of a set of interest points or objects (e.g., vehicles or humans) in the image sequence and with respect to the image plane.

[0165] In another approach, portions of the object recognition system are embodied in a remote server, comprising a computer readable apparatus storing computer executable instructions configured to perform pattern recognition in data streams for various applications, such as scientific, geophysical exploration, surveillance, navigation, data mining (e.g., content-based image retrieval). A myriad of other applications exist that will be recognized by those of ordinary skill given the present disclosure.