Reduced power consumption analog or hybrid mac neural network

[0001] A neural processor may implement an artificial intelligence (AI) neural network (NN) model using a hardware accelerator with digital hybrid or analog multiply and accumulate (MAC) modules/circuits that consume significant amounts of power.

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Methods, systems, and computer program products are provided for power-efficient performance of a hardware accelerator (e.g., a neural processing unit (NPU)) comprising hybrid or analog multiply and accumulate (MAC) processing elements (PEs). Power efficiency may be implemented based on the type of activation function (AF) implemented by an activation layer and detection of one or more AF predictors that predict AF output for an AF input. For example, power consumption may be reduced in neural networks with a rectified linear unit (ReLU) activation layer. A hybrid or analog MAC circuit may be configured with a look-ahead sign detector to dynamically stop computations prior to completion, for example, based on detection of a negative value, which a ReLU activation layer may (e.g., subsequently) convert to zero. A controller may provide power and/or clock cycles to an analog to digital converter (ADC) to determine a sign of a value being computed. The sign of a value may be indicated by a most significant bit (MSB). The sign may be used to selectively complete computations for positive values and selectively terminate computations for negative values, thereby reducing power consumption of the MAC circuit.

[0004] Selective completion and termination of computations by an array of MAC PEs may reduce power consumption of a neural processor. Power may be conserved by dynamically controlling operation of one or more components in a MAC circuit (e.g., enablement signal(s), a power supply and/or a clock for ADCs). In some examples, power consumption may be reduced by selectively avoiding ADC cycles unnecessary for a known outcome of some computations (e.g., conversion of negative values to zero by a ReLU activation layer). An AF detector may detect a type of AF and one or more AF predictors indicating or predicting an AF output for a given AF input. The known output may be used to control ADC operation. An AF predictor may indicate a predictable AF output. For example, with a ReLU activation layer, an AF predictor may be a sign of a value. A look-ahead sign detector may be used to detect the sign of a digital value being computed before completion of the computation. An ADC controller may use the detected sign of a value being computed to stop computation. Other types of activation layers and circuits to determine one or more AF predictors for the activation layers may be implemented to provide information to one or more MAC circuit controllers (e.g., ADC controllers).

[0005] Further features and advantages of the subject matter (e.g., examples) disclosed herein, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings. It is noted that the present subject matter is not limited to the specific embodiments described herein. Such embodiments are presented herein for illustrative purposes only. Additional embodiments will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein.

[0014] The features and advantages of the examples disclosed will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference number.

[0015] The present specification and accompanying drawings disclose one or more embodiments that incorporate the features of the various examples. The scope of the present subject matter is not limited to the disclosed embodiments. The disclosed embodiments merely exemplify the various examples, and modified versions of the disclosed embodiments are also encompassed by the present subject matter. Embodiments of the present subject matter are defined by the claims appended hereto.

[0016] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an example embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0017] In the discussion, unless otherwise stated, adjectives such as “substantially” and “about” modifying a condition or relationship characteristic of a feature or features of an example embodiment of the disclosure, are understood to mean that the condition or characteristic is defined to within tolerances that are acceptable for operation of the embodiment for an application for which it is intended.

[0018] Numerous exemplary embodiments are described as follows. It is noted that any section/subsection headings provided herein are not intended to be limiting. Embodiments are described throughout this document, and any type of embodiment may be included under any section/subsection. Furthermore, embodiments disclosed in any section/subsection may be combined with any other embodiments described in the same section/subsection and/or a different section/subsection in any manner.

[0019] Methods, systems, and computer program products are provided for power-efficient performance of a hardware accelerator (e.g., a neural processing unit (NPU)) comprising hybrid or analog multiply and accumulate (MAC) processing elements (PEs). Power efficiency may be implemented based on the type of activation function (AF) implemented by an activation layer and detection of one or more AF predictors that predict AF output for an AF input. For example, power consumption may be reduced in neural networks with a rectified linear unit (ReLU) activation layer. A hybrid or analog MAC circuit may be configured with a look-ahead sign detector to dynamically stop computations prior to completion, for example, based on detection of a negative value, which a ReLU activation layer may (e.g., subsequently) convert to zero. The sign of a value may be indicated by a most significant bit (MSB). A controller may provide power and/or clock cycles to an analog to digital converter (ADC) to determine a sign of a value being computed. The sign may be used to selectively complete computations for positive values and selectively terminate computations for negative values, thereby reducing power consumption of the MAC circuit.

[0020] Such embodiments may be implemented in various configurations. For instance, FIG. 1 shows a block diagram of an example neural network (NN) training and inference computing environment (referred to herein as “NN computing environment”) 100 for improving performance (e.g., reducing power consumption) of a hardware accelerator (e.g., neural processor), according to an example embodiment. Example NN computing environment 100 may include, for example, one or more computing devices 104, one or more networks 114, and one or more servers 116. Example NN computing environment 100 presents one of many possible examples of computing environments. Example system 100 may comprise any number of computing devices and/or servers, such as example components illustrated in FIG. 1 and other additional or alternative devices not expressly illustrated.

[0021] Network(s) 114 may include, for example, one or more of any of a local area network (LAN), a wide area network (WAN), a personal area network (PAN), a combination of communication networks, such as the Internet, and/or a virtual network. In example implementations, computing device(s) 104 and server(s) 116 may be communicatively coupled via network(s) 114. In an implementation, any one or more of server(s) 116 and computing device(s) 104 may communicate via one or more application programming interfaces (APIs), and/or according to other interfaces and/or techniques. Server(s) 116 and/or computing device(s) 104 may include one or more network interfaces that enable communications between devices. Examples of such a network interface, wired or wireless, may include an IEEE 802.11 wireless LAN (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth™ interface, a near field communication (NFC) interface, etc. Further examples of network interfaces are described elsewhere herein.

[0022] Computing device(s) 104 may comprise computing devices utilized by one or more users (e.g., individual users, family users, enterprise users, governmental users, administrators, hackers, etc.) generally referenced as user(s) 102. Computing device(s) 104 may comprise one or more applications, operating systems, virtual machines (VMs), storage devices, etc., that may be executed, hosted, and/or stored therein or via one or more other computing devices via network(s) 114. In an example, computing device(s) 104 may access one or more server devices, such as server(s) 116, to provide information, request one or more services (e.g., content, model(s), model training) and/or receive one or more results (e.g., trained model(s)). Computing device(s) 104 may represent any number of computing devices and any number and type of groups (e.g., various users among multiple cloud service tenants). User(s) 102 may represent any number of persons authorized to access one or more computing resources. Computing device(s) 104 may each be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a Microsoft® Surface® device, a personal digital assistant (PDA), a laptop computer, a notebook computer, a tablet computer such as an Apple iPad™, a netbook, etc.), a mobile phone, a wearable computing device, or other type of mobile device, or a stationary computing device such as a desktop computer or PC (personal computer), or a server. Computing device(s) 104 are not limited to physical machines, but may include other types of machines or nodes, such as a virtual machine, that are executed in physical machines. Computing device(s) 104 may each interface with server(s) 116, for example, through APIs and/or by other mechanisms. Any number of program interfaces may coexist on computing device(s) 104. An example computing device with example features is presented in FIG. 7.

[0023] Computing device(s) 104 have respective computing environments. Computing device(s) 104 may execute one or more processes in their respective computing environments. A process is any type of executable (e.g., binary, program, application) that is being executed by a computing device. A computing environment may be any computing environment (e.g., any combination of hardware, software and firmware). For example, computing device(s) 104 may include central processing unit(s) (CPUs) 106 that execute instructions, neural processing unit(s) (NPUs) 108 that implement neural network (NN) models, NN applications 110 that utilize one or more NN models, etc.

[0024] Server(s) 116 may comprise one or more computing devices, servers, services, local processes, remote machines, web services, etc. for providing NN training, models and/or content to computing device(s) 104. In an example, server(s) 116 may comprise a server located on an organization's premises and/or coupled to an organization's local network, a remotely located server, a cloud-based server (e.g., one or more servers organized in a distributed manner), or any other device or service that may host, manage, and/or provide NN training, models and/or content. Server(s) 116 may be implemented as a plurality of programs executed by one or more computing devices. Server programs and content may be distinguished by logic or functionality (e.g., as shown by example in FIG. 1).

[0025] Server(s) 116 may each include one or more model trainers 118, one or more NN models 120, and/or content 122. In some examples, computing device(s) 104 may include model trainer(s) 118, NN model(s) 120, and/or content 122, which may be developed on computing device(s) 104, downloaded from server(s) 116, etc.

[0026] Example NN computing environment 100 may operate at the edge or in an edge domain, referring to the edge or boundary of one or more networks in network(s) 114. An edge domain may include an end user device (e.g., computing device(s) 104), such as a laptop, mobile phone, and/or any IoT device (e.g., security camera).

[0027] Artificial intelligence (AI) neural network (NN) models (e.g., NN model(s) 120) may be used in many applications (e.g., NN application(s) 110), such as image classification and speech recognition applications. An AI NN model, also referred to herein as a model, may represent an algorithm learned by a machine (e.g., machine learning (ML)). An algorithm/model may be trained. Model training may refer to a machine learning one or more parameters of an algorithm/model, for example, to improve performance of the algorithm/model. A trained model (e.g., NN model(s) 120) may be used for inference. For example, NN application(s) 110 may use a trained model (e.g., NN model(s) 120) to infer a classification (e.g., classify an image in content 122 as a person or a vehicle).

[0028] There may be one or more user experience (UX) scenarios on computing device(s) 104 that may rely on AI. Experiences driven by AI may involve creating and/or running algorithms without a human writer (e.g., a machine may train algorithms itself). Humans may (e.g., alternatively and/or in conjunction with AI) write programs or algorithms manually in software (e.g., C code) to perform tasks.

[0029] NN Application(s) 110 may pertain to a wide variety of AI applications, such as audio applications (e.g., noise suppression, spatial audio, speaker separation to distinguish between speakers), video applications (e.g., enhancement compression), speech applications (e.g., dictation, NTTS, voice access, translation), system health applications (e.g., security such as antivirus, battery usage, power usage), etc.

[0030] User(s) 102 may use computing device(s) 104 to run NN application(s) 110, which may, for example, allow user(s) 102 to browse files on server(s) 116 and/or select content 122. User(s) 102 may use computing device(s) 104, for example, to process content 122 (e.g., using NN model(s) 120). NN application(s) 110 may process (e.g., selected) content 122 using a trained model (e.g., among NN model(s) 120). An example of an NN application 110 may be a pattern recognition application to identify objects (e.g., people, animals, plants) in image frames. User(s) 102 may use computing device(s) 104 to run NN application(s) 110, for example, to select, train or implement model(s) 120 (e.g., use models to infer classifications of content 122).

[0031] Model trainer(s) 118 may train and evaluate (e.g., generate) one or more models (e.g., NN model(s) 120) to improve performance of a hardware accelerator (e.g., neural processor) comprising hybrid or analog multiply and accumulate (MAC) processing elements (PEs). Model trainer(s) 118 may receive as input an original or modified form of content 122 generated by one or more computing devices (e.g., computing device(s) 104, server(s) 116). Model trainer(s) 118 may provide (e.g., manual and/or automated) labeling (e.g., pre-classification) of features (e.g., Ifmaps) for training content 122, for example, to produce a featurized training dataset with known labels. A training dataset may be split into a training dataset and a testing dataset. A training process may train a model with a training dataset. A trained model may be retrained, for example, as needed or periodically (e.g., with an expanded training dataset and/or Ifmaps).

[0032] Multiple models with multiple (e.g., different) feature sets may be trained (and evaluated). Various neural network models may be trained and evaluated, such as convolutional neural network (CNN) models and recurrent neural network (RNN) models. A training process may utilize different settings to determine the best hyper parameters values. In an example of a neural network, parameter values may be determined for activation type, number of neurons in each layer, and the number of layers.

[0033] Trained NN model 120 may include, for example, a feature extractor, a feature transformer, and a classifier. A feature extractor may extract features from content 122. A feature transformer may transform extracted features into a format expected by a classifier. A feature transformer may, for example, convert the output of feature extractor into feature vectors expected by a classifier. A classifier may classify the extracted features as one or more classes. A classifier may generate an associated confidence level for a (e.g., each) classification (e.g., prediction).

[0034] Trained NN model 120 may receive as input an original or modified form of content 122 generated by one or more computing devices (e.g., computing device(s) 104 or server(s) 116). NN model(s) 120 may generate classifications based on inputs (e.g., Ifmaps) based on the training received from model trainer(s) 120. Classifications may include, for example, binary or multiclass classifications. Classifications may include or be accompanied by a confidence level, which may be based on a level of similarity to labels for one or more training set Ifmaps.

[0035] Trained NN model 120 may be saved (e.g., by model trainer(s)) 118) in a file. The file may be loaded into one or more devices (e.g., computing device(s) 104) to use the model (e.g., to infer). NN model 120 may interface to network(s) 114 for input (e.g., content 122) to generate results (e.g., by trained NN model 120 processing content 122). In an example, a NN model 120 may be trained to detect four classes based on training frames associated with training labels. For example, a deep neural network (DNN) may be tasked to understand what viewable objects (e.g., cat, dog, person, car) appear in content 122.

[0036] NN model 120 may be a DNN model. A CNN may be a type of DNN. NN model(s) 120 may be implemented (e.g., in part) by hardware (HW). For example, a HW accelerator may accelerate computations for one or more CNN layers. HW used to implement an AI model (e.g., a HW accelerator) may have an (e.g., a significant) impact on the power efficiency of an AI model during inference on an edge device (e.g., a personal computer (PC)). Power efficiency and/or model accuracy may play a (e.g., significant) role in the performance of an AI model.

[0037] An example of a hardware accelerator that may be used to implement one or more models is/(are) NPU(s) 108. For example, NN application(s) 110 (e.g., and/or operating system(s)) executed by CPU(s) 106 may utilize NPU(s) 108 to implement NN model(s) 120. Computing device(s) 104 may be a battery-operated device, such as a mobile phone. It may be important for NPU(s) 108 to implement NN model(s) 120 with less power to conserve energy stored in the device battery and/or in general to conserve energy. Energy conservation may depend on the implementation of NPU(s) 108. For example, NPU(s) 108 may include digital and/or hybrid or analog multiply and accumulate (MAC) processing elements (PEs).

[0038] Computational power efficiency may be important, given that AI compute demand may continue to increase exponentially year over year. Energy cost and/or memory access may be reduced, for example, by reducing data movement.

[0039] In some examples, a DNN (e.g., a CNN) may be implemented with a (e.g., highly) parallel computation architecture, such as single instruction, multiple data (SIMD), to provide high-throughput convolutions. Convolutions may dominate CNN runtime (e.g., convolutions may account for over 90% of CNN operations). Memory bandwidth may impact power efficiency and/or may cause a memory access bottleneck. For example, a (e.g., each) MAC operation may involve four memory accesses, which may lead to high energy consumption.

[0040] FIG. 2 shows a block diagram of an example NPU 200 with a systolic array architecture and weight stationary data movement, according to an embodiment. Example NPU 200 presents one of many possible example configurations of NPU(s) 108 shown in FIG. 1. Example NPU 200 may be a DNN accelerator. Example NPU 200 shows a HW (hardware) accelerator (e.g., NPU(s) 108) implemented, with a MAC/AMAC circuit 218. NPU 200 may include, for example, MAC/AMAC circuit 218 (which includes a plurality of unit elements (UEs) 202a-n), one or more SRAM buffer(s) 206, a processor 210, and/or an activation function (AF) 212.

[0041] As shown in FIG. 2, MAC/AMAC circuit 218 may include a systolic array of MAC PEs comprising UEs 202a-n. In some examples, a systolic array of MAC PEs may perform trillions of operations per second (TOPS). In one of many example implementations, a systolic array of PEs is organized in clusters in a DNN hardware accelerator, e.g., with a weight-stationary data flow. A DNN HW accelerator may be configured and/or implemented to reduce energy cost and/or memory access. Data movements intimate convolutions performed by PEs in the systolic array. Data flow may be weight stationary (WS). A WS dataflow may improve (e.g., reduce) data-movements, for example, by (e.g., maximizing) usage of a storage hierarchy, which may include, for example, a low-cost register file (RF) and global buffer. A (e.g., each) PE may include or may be associated with (e.g., coupled to) an RF.

[0042] Example NPU 200 may implement a model with a digital, hybrid or analog MAC (AMAC) module such as a digital, analog or hybrid MAC circuit. A hybrid MAC circuit may include, for example, digital multiplication and analog accumulation. An AMAC (e.g., referring to analog and hybrid MACs) may be more power efficient than a digital MAC (DMAC) circuit.

[0043] One or more enhancements may be implemented to improve AMAC performance (e.g., accuracy, energy efficiency or power consumption). An AMAC may include an analog to digital converter, for example, to convert an analog accumulated signal to a digital signal. In some examples, an AMAC may be implemented with a successive approximation register (SAR) ADC. In some examples, an AMAC may be implemented with charge-sharing. One or more enhancements described herein may be implemented to improve AMAC performance, such as performance of an AMAC implemented with charge-sharing and/or a SAR-based ADC.

[0044] UEs 202a-n may include MAC/AMAC units 214a-n, local buffers 208a-n, and multiplexer (MUX) logic 209a-n. UEs 202a-n may perform (e.g., a large number of) multiply-accumulate (MAC) operations in parallel. Arithmetic operations may be fed new operands in each operational cycle using, for example, a memory hierarchy that exploits data-reuse opportunities. In some examples, there may be three levels of memory: external SDRAM 204, global SRAM buffer(s) 206 (e.g., SRAM global buffer(s) or double buffer(s)), and local buffers 208a-n (e.g., local buffers within each UE).

[0045] For example, during operations, as the filters slide across the input feature maps, the filters may be temporarily “cached” in SRAM buffer(s) 206 or within UE local buffers 208a-n (e.g., in a register file or flip-flops), which avoids reloading from external memory (e.g., SDRAM) 204 for each operation. Data reuse may support highly efficient power consumption, since accessing memory utilizes significantly more energy than arithmetic operations.

[0046] As shown in FIG. 2, the first UE local buffers (e.g., local buffer 208a) may be fed with input X and weight W from SRAM (e.g., global) buffer(s) 206, while inputs X and/or weights W for other UEs 202a-n may be shifted from neighboring UE local buffer(s) 208a-n.

[0047] In some examples, the output of UEs 202a-n may be fed through an activation function and then stored in SRAM buffer(s) 206 (e.g., for subsequent use as the input to the next processing cycle). In some examples, the output of UEs 202a-n may be stored in the global buffer (e.g., without passing through an output activation function), for example, for processing by processor 210. Processor 210 may include, for example, a microcontroller (uC), a central processing unit (CPU), a digital signal processor (DSP), and/or other type of processing circuit.

[0048] In some examples, data may flow between SDRAM 204, SRAM (e.g., global) buffer(s) 206 and UE local buffers 208a-n. In some examples, data may flow between UE local buffers 208a-n. For example, the data flow may be dynamically configured or statically pre-routed within MUX logic (e.g., SRAM buffer(s) MUX logic 209a, 209b and/or local buffer logic 208a-n) to reduce power consumption where data re-use occurs.

[0049] In some examples (e.g., with weights stationary), only the input (X) may move between UE local buffer(s) 208a-n. For example, weights may remain static (e.g., until the completion of tensor processing) after weights are loaded to each UE 202a-n.

[0050] A DNN may perform operations on an input activation (IA) vector (e.g., IA_x,y,z) (e.g., from a previous layer) with a filter represented by a weights vector (e.g., W_{i j,k}), where i may range from 1 to N, producing an output activation (OA) vector (e.g., OA_x,y,n). A (e.g., each) PE may generate an output, for example, in accordance with Eq. (1):

[0051] $\begin{array}{cc}\mathrm{PE}\left(i\right)=\mathrm{Xi}*\mathrm{Wi}& \mathrm{Eq}.\left(1\right)\end{array}$
An OA vector may be the total generated by multiple (e.g., all) PEs (e.g., PE[0] to PE[N−1]). The OA vector may be provided to the next layer (e.g., for additional operations). The OA vector may be stated, for example, in accordance with Eq. (2):

[0052] $\begin{array}{cc}OA={\sum }_{i=1}^N\mathrm{Xt}\star \mathrm{Wt}& \mathrm{Eq}.\left(2\right)\end{array}$

[0053] A digital MAC (DMAC) may perform operations to generate inner products (e.g., mid-terms) and to accumulate the multiple multiplication results utilizing OR gates or half adders, which may consume significant power. Performance of MAC operations with less power may significantly reduce total NPU power consumption based on the number of MACs and MAC operations. For example, a hybrid MAC with digital multiplication and analog accumulation may reduce MAC power consumption. In some examples, mid-terms of IA vectors and weight vectors may be calculated digitally while mid-terms may be accumulated (and summed) using analog charge sharing.

[0054] A (e.g., battery-powered) device (e.g., a smart phone) may run one or more models (e.g., DNNs). Analog computations may support low energy consumption for model inference computations. In some examples, a hybrid or analog MAC may perform (e.g., binary) multiplication using AND gates (e.g., to generate multiple intermediate products). Intermediate products at the output of the AND gates may be referred to as mid-terms. Summation of the products may be performed by charge sharing. Multiple mid-terms with the same weight may charge a capacitor. Capacitors may be coupled to respective wires in a charge collection bus according to bit position in the binary multiplication represented by the output of the AND gates. The wires (e.g., and the charges accumulated on the wires) in the charge collection bus may be associated with weights relative to other wires based on bit position (e.g., LSB to MSB). The accumulated charges on the wires (e.g., from one or more charge capacitors in each of one or more PEs) may be combined (e.g., based on their respective weights) into a (e.g., single) charge representing the analog value (e.g., signal) of the multiplication. A summation of multiplication products from multiple PEs (e.g., sharing the weighted charge lines) may be referred to as a dot product. Convolution may be simplified into a dot product.

[0055] ADC power may consume a significant portion of the power consumed by a MAC/AMAC circuit. ADC power (e.g., in a SAR ADC) may be proportional to the number of ADC output bits. Each clock cycle of a SAR ADC may generate a bit. A first cycle may output the MSB and successive cycles may generate less significant bits. Training and/or observation may determine that not all bits are needed (e.g., not all bits are effective bits) for one or more inputs or operations. A model (e.g., DNN) may have many (e.g., millions) of parameters, coefficients or weights, which may be trained to specific values. The computation graph or DNN may have many (e.g., millions of) nodes or multipliers with many (e.g., tens of thousands of) dot products. In some examples, a level of precision may be fixed and in other examples precision may be variable. The dynamic range (e.g., maximum number of effective bits) for a specific node of a computation graph in a DNN may be determined. For example, some nodes may have a smaller dynamic range than other nodes. The precision needed at various nodes to maintain the overall accuracy of the model (e.g., algorithm) may vary. For example, an observation may be made that one or more particular bits in particular nodes are always zero.

[0056] As shown in FIG. 2, precision 224 may represent accuracy information developed during model training. Accuracy information, such as expected intermediate quantization (e.g., the number of useful bits at ADC output) may be saved, for example, in a model file, which may be loaded into an NPU for AI model operation.

[0057] Precision 224 may be used to control NPU operation to conserve power. NPU power efficiency may be improved by controlling operation of analog circuitry (e.g., by controlling ADC operation). MAC power consumption (e.g., by ADC components) may be improved, for example, by dynamically controlling (e.g., reducing) ADC clock cycles to avoid power consumption for cycling the ADC beyond accuracy of ADC output. Accuracy information may be developed during training or inference (e.g., by observation). Precision 208 may be an input to control logic (e.g., a controller) to dynamically adjust output. For example, the controller may dynamically adjust the number of clock cycles of an ADC to dynamically control the precision output by the ADC (e.g., SAR ADC).

[0058] FIG. 3 shows a block diagram of a processing array with power-efficient hybrid multiply and accumulate (MAC) processing elements (PEs), according to an example embodiment. Example MAC circuit 300 presents one of many possible example configurations of MAC circuit 218 shown in FIG. 2.

[0059] Example MAC circuit 300 includes N processing elements (e.g., PE[0] to PE[N−1]) coupled to (e.g., weighted) charge collection bus 302. PEs may include a plurality of UEs 320 (e.g., UEs 202a-n shown in FIG. 2). The charge-sharing hybrid (digital-analog) MAC architecture shown in FIG. 3 may (e.g., significantly) reduce MAC power consumption by splitting multiply and accumulate operations between digital and analog domains. For example (e.g., for a 12 nm Fin-Fet process), an approximately 10 TMAC NPU 4-bit analog MAC architecture compared to an approximately 10 TMAC NPU 4-bit digital MAC architecture may increase area by approximately two (2) square millimeters while decreasing power consumption by nearly one (1) Watt.

[0060] As shown by example in FIG. 3, mid-terms may be calculated by digital circuitry (e.g., AND gates) configured to multiply input Xi[2:0] by weights Wi[2:0]. Mid-terms may be accumulated by analog circuitry. For example, mid-term outputs of the digital circuitry may charge (e.g., relatively small) charge accumulation capacitors C coupled to charge collection lines in charge collection bus 302. Charge accumulation capacitors C may have (e.g., significantly) reduced Goad. In an example implementation, a value of charge accumulation capacitors C may be 0.5 femto Farads (fF) (e.g., for a 12 nm Fin-Fet process).

[0061] Mid-term summation may be calculated based on a charge-sharing concept. A charge for each mid-term result may be transferred from the digital multiplication to a (e.g., global) charge collection line (e.g., metal bit line). Mid-terms with the same “weight” may be coupled to the same charge collection line. The accumulation of multiple mid-terms with the same “weight” may be performed by (e.g., passively) accumulating their charges on the same metal bit line. Passive accumulation may conserve energy because passive accumulation does not consume energy. Mid-terms on a charge collection line representing a smaller weight (e.g., least significant (LS) charge collection line) may have less value than mid-terms on charge collection lines representing higher weights (e.g., most significant (MS) charge collection line).

[0062] Combiner 304 may be coupled to charge collection bus 302 and ADC 306. Charges on charge collection bus 302 may be inputs to combiner 304. Combiner 304 may generate an analog output (e.g., analog Psum), which may be provided as input to ADC 306 for conversion to a digital value. Combiner 304 may be controlled or calibrated (e.g., at least in part) by a bias input. The bias may be fixed or variable. Inputs may be normalized, for example, to maintain values within the dynamic range of ADC 306.

[0063] Combiner 304 may combine charges on the charge collection lines in charge collection bus 302. Charges on the bit lines may be weighted (e.g., by circuitry in combiner 304). Weights may be implemented, for example, in charge lines, in capacitor values, and/or in combiner 304. Combiner 304 may include passive and/or active circuitry. In some examples, combiner 304 may perform a weighted charge summation. Charges on each bit line may be accumulated with charges on other bit lines through one or more voltage dividers (e.g., resistive or capacitive dividers). For example, the charge on each bit line may be accumulated with charges on other bit lines through a capacitance value corresponding to the weight of the bit line (e.g., each charge collection line may be coupled to a capacitor with a different value). For example, an MSB line on charge collection bus 302 may not have a resistor while other lines may have increasing values of resistors to reduce their relative weights by a resistive or capacitive divider.

[0064] Charge collection bus 302 may be shared by UEs 320 in multiple PEs (e.g., PE[0] through PE [N−1]). As shown in FIG. 3, charge collection bus 304 may be coupled to (e.g., and may accumulate charges from) each of multiple UEs 320. Each of multiple UEs 320 (e.g., in PE[0] through PE [N−1]) may couple multiplication product charges to shared charge collection bus 302. In some examples, charge summation by combiner 304 may be performed on a (e.g., single) multiplication result from a (e.g., single) PE. In some examples, charge summation by combiner 304 may be performed on multiple multiplication results from each of multiple MAC PEs coupled to the same bit lines. The combined analog value may be an analog Psum (e.g., as shown by example in FIG. 3).

[0065] ADC 306 may be, for example, a successive approximation register (SAR) ADC. ADC 306 may receive the combined analog value (e.g., analog Psum) generated by combiner 304. ADC 306 may be coupled to (e.g., and may provide a digital output value to) an activation function (AF) (e.g., AF 212). ADC 306 may (e.g., be configured to) convert the total combined or summed charge generated by combiner 304 (e.g., an analog Psum) into a digital representation (e.g., Y[4:0]). In some examples, digital conversion by ADC 306 may be performed on a (e.g., single) multiplication result from a (e.g., single) PE. In some examples, digital conversion by ADC 306 may be performed on multiple multiplication results from each of multiple MAC PEs coupled to the same bit lines. The combined analog value may be an analog Psum. Digital representation (e.g., Y[4:0]) may represent summation of one or multiple PE products. Digital representation (e.g., Y[4:0]) may be referred to as a dot product. In some examples, Y[4:0] may represent a digital Psum. ADC 306 may or may not be used to convert partial summation (Psum) results. In some examples, ADC 306 may convert the (e.g., entire) dot product operation (e.g., using the relevant inputs, such as pixels, and channels of the input data and filters, such as weights, that may be used to calculate an output pixel).

[0066] In an example (e.g., as shown in FIG. 3), there may be two three-bit vectors IA[2:0] and W[2:0], which may be multiplied (e.g., by AND gates) and accumulated (e.g., using capacitors C coupled to charge collection bus 302). Multiplication results may be indicated by mid-terms. Mid-terms may be accumulated on weighted charge lines (e.g., five charge lines of charge collection bus 302). The accumulated mid-term charges may be accumulated into a single charge by combiner 304. The single charge (e.g., analog Psum) may be converted (e.g., partially or completely) into a digital value (e.g., of one or more bits) by ADC 306. A least significant (LS) charge line may have a weight of 1 while a second charge line may have a weight of 2, a third weighted at 4, a fourth weighted at 8, a fifth (e.g., most significant (MS) line) weighted at 16, etc. An example of digital multiplication and weighted analog accumulation is shown below:

[0067] $1*\left(\mathrm{IA}\left[0\right]*W\left[0\right]\right)+2*\left(\mathrm{IA}\left[1\right]*W\left[0\right]+\mathrm{IA}\left[0\right]*W\left[1\right]\right)+4*\left(\mathrm{IA}\left[2\right]*W\left[0\right]+\mathrm{IA}\left[1\right]*W\left[1\right]+\mathrm{IA}\left[0\right]*W\left[2\right]\right)+8*\left(\mathrm{IA}\left[2\right]*W\left[1\right]+\mathrm{IA}\left[1\right]*W\left[2\right]\right)+16*\left(\mathrm{IA}\left[2\right]*W\left[2\right]\right)$
Combiner 304 may combine charges on the charge lines according to these weights. Many other weights may be implemented, e.g., LS line at 1/32, second line at 1/16, third at ⅛, fourth at ¼, fifth (e.g., MS line) at ½, etc. ADC 306 may (e.g., be configured to) convert the combined charge into output Y[4:0].

[0068] As shown by example in FIG. 3, NPU power efficiency may be improved by controlling operation of MAC circuitry (e.g., by controlling operation of ADC 306). Performance improvements described herein may apply to MAC architectures and/or other analog computation circuitry. Power consumption of ADC 306 may be proportional to the number of cycles used by ADC 306 (e.g., a SAR ADC) to convert an analog value to a digital value. MAC control logic (e.g., ADC control logic) may reduce MAC power consumption, and thereby reduce NPU power consumption. One or more operating characteristics of an NPU (e.g., type of AF) may be leveraged to improve performance (e.g., reduce power consumption) of a MAC circuit. An NPU may have one or more activation layers with one or more activation functions (AFs). AFs may have one or more outputs that may be predicted (e.g., known) for one or more inputs. Determination of one or more AF predictors may permit one or more performance improvements, such as a reduction in energy consumption by an NPU.

[0069] For example, a ReLU AF may convert negative values generated by ADC 306 to zero. The sign (e.g., the negative sign) of a value may be referred to as an AF predictor. The sign of a value of digital output Y[4:0] may be indicated, for example, by an MSB (e.g., Y[4]). An MSB indicating a sign may be referred to as an AF predictor (e.g., for a ReLU AF). Determination of an MSB before conversion of other bits may support early termination of an analog to digital conversion (e.g., after conversion of the MSB), for example, based on a determination that the sign of the value is negative, thereby conserving energy. This conservation of energy may occur for one or more operations of one or more (e.g., many) MAC PEs (e.g., hundreds, thousands, millions) for one or more layers (e.g., 50 layers), thereby (e.g., substantially) reducing power consumption of an NPU to implement model operation.

[0070] As shown by example in FIG. 3, ADC controller 308 may (e.g., be configured to) control ADC 306 to conserve power, e.g., during inference operation. ADC controller 308 may conserve power by dynamically controlling operation of ADC 306 (e.g., by controlling enablement signal(s), clock and/or power supply provided to ADC 306). ADC controller 308 may control ADC 306 to selectively terminate conversion of an analog value (e.g., analog Psum) prior to completing conversion of all bits in digital output Y. ADC controller 308 may (e.g., be configured to) selectively stop (e.g., suspend or terminate operation of) ADC 306 before completely converting the analog output (e.g., analog Psum) to the digital output Y based on one or more AF predictors for the type of AF deployed (e.g., in use). The AF type may be known or may be detected. In some examples, ADC controller 308 may control ADC 306, for example, based on determinations about the AF type (e.g., AF 212) applied to the digital output Y (e.g., ReLU AF) and one or more AF predictors (e.g., sign of digital output Y).

[0071] ADC controller 308 may be coupled to activation function detector 310 and look-ahead sign detector 312. ADC controller 308 may receive as inputs, for example an AF type indication (e.g., signal) generated by activation function detector 310 and an AF predictor (e.g., Y value sign) indication (e.g., signal) generated by look-ahead sign detector 312. In some examples, the AF type may be known, in which case the AF type may not be detected or signaled to ADC controller 308 by activation function detector 310. In some examples, the AF type and the one or more AF predictors may differ from those shown by example in FIG. 3.

[0072] Activation function (AF) detector 310 may be coupled to the input of ADC controller 308 and to storage 314, which may store DNN topology (e.g., including the type of AF). AF detector 310 may access storage 314 to determine AF type. For example, DNN topology in storage 314 may indicate AF 212 is a ReLU AF or another type of AF. AF detector 310 may provide the determined AF type to ADC controller 308.

[0073] Look-ahead sign detector 312 may be coupled to the output of ADC 306 and the input of ADC controller 308. During analog to digital conversion of the analog value (e.g., analog Psum) by ADC 306, look-ahead sign detector 312 may read the MSB of digital output Y (e.g., Y[4], depending on the fixed or variable precision of digital output Y). Look-ahead sign detector 312 may determine the sign of digital output Y as positive or negative based on the MSB of digital output Y. Look-ahead sign detector 312 may provide the indication of the sign of digital output Y to ADC controller 308.

[0074] ADC controller 308 may (e.g., be configured to) selectively stop operation of ADC 306 based on determination of an AF predictor for a determined AF type. ADC controller 308 may be configured with logic for one or more AF types. In an example, ADC controller 308 may (e.g., be configured to) selectively terminate the operation of ADC 306 during a conversion based on at least a determination of a negative sign of digital output Y (e.g., for a known or a determined ReLU AF type), thereby reducing power consumption of the MAC circuit.

[0075] ADC controller 308 may selectively terminate operation of ADC 306, for example, by interrupting a power supply to ADC 306, interrupting a clock provided to ADC 306, negating an enablement signal to ADC 306, etc. ADC controller 308 may (e.g., be configured to) selectively stop ADC 306 (e.g., a SAR ADC) to reduce a number of cycles of ADC 306 for a particular conversion operation, thereby conserving power. For example, ADC controller 308 may terminate conversion operation of ADC 306 after one cycle based on determination of a negative sign of digital output Y, avoiding four cycles (e.g., for unnecessary bits Y[3:0]). ADC controller 308 may reduce the number of clock cycles of ADC 306 to one clock cycle, thereby avoiding consumption of energy for four additional clock cycles of ADC 306. This conservation of energy may occur for one or more operations of one or more (e.g., many) MAC PEs (e.g., hundreds, thousands, millions) for one or more layers (e.g., 50 layers), thereby (e.g., substantially) reducing power consumption of an NPU to implement model operation.

[0076] FIG. 4 shows a flowchart of an example of a method 400 for model training, detection and storage of DNN topology and other information, according to an example embodiment. Example training method 400 presents one of many possible example methods of model trainer(s) 118 training NN model(s) 120. Embodiments disclosed herein and other embodiments may operate in accordance with example training method 400. Example training method 400 comprises steps 402-420. However, other embodiments may operate according to other methods. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the foregoing discussion of embodiments. No order of steps is required unless expressly indicated or inherently required. There is no requirement that a method embodiment implement all of the steps illustrated in FIG. 4. Embodiments may implement fewer, more or different steps.

[0077] As shown in FIG. 4, in step 402, a model architecture and hyper parameters may be selected and/or updated for a (e.g., DNN) model. For example, as shown in FIG. 1, user(s) 102 and/or model trainer(s) 118 may select or update a model architecture and hyper parameters for a (e.g., DNN) model (e.g., NN model(s) 120). A model architecture may be selected by selecting a topology of a neural network. A topology may be a computational graph of multiplication, summation and connectivity.

[0078] In step 404, weights (e.g., X, W input bit width) and MAC precision may be initialized. For example, as shown in FIGS. 1-3, user(s) 102 and/or model trainer(s) 118 may initialize weights 220 and MAC precision 224 for a (e.g., DNN) model (e.g., NN model(s) 120). For example, AMAC node precision may be initialized with maximum precision values, mixed precision values, random precision values, etc.

[0079] In step 406, the model (e.g., with initialize values) may be run in a feed forward path to (e.g., iteratively) develop the model weights, precision, etc. based on training data, including determining precision information. For example, as shown in FIGS. 1-3, model trainer(s) 118 may train NN model 120 with training data input as Ifmaps 222 and revise weights 220 and precision 224 accordingly to improve the accuracy of NN model 120 based on labels in Ifmaps 222.

[0080] In step 408, error, loss and/or cost functions may be evaluated. For example, as shown in FIGS. 1-3, model trainer(s) 118 may evaluate error, loss and/or cost functions with respect to the performance of NN model 120 as it may be configured at a certain stage of training.

[0081] In step 410, a determination may be made whether the model in training has reached convergence. For example, as shown in FIGS. 1-3, model trainer(s) 118 may determine (e.g., based on the error, loss and/or cost function evaluation) whether additional training may or may not improve NN model 120. A model (e.g., DNN) may be trained, for example, when weights are converged with accurate target labels. Convergence may be reached by model training, for example, if error, loss and/or cost functions settle to within a (e.g., predetermined) range of acceptable final values.

[0082] In step 412, a determination may be made whether the model in training has reached a target if the trained model has converged. For example, as shown in FIGS. 1-3, model trainer(s) 118 may determine (e.g., if the model has converged) whether the trained model has reached or achieved one or more targets (e.g., precision).

[0083] In step 414, the model may be deemed trained if the model achieves one or more targets. For example, as shown in FIGS. 1-3, model trainer(s) 118 may determine NN model 120 to be trained completely based on determinations that NN model 120 converged and achieved one or more (e.g., specified) targets.

[0084] In step 416, DNN topology, weight values and accuracy information (e.g., Y precision) may be stored. For example, as shown in FIGS. 1-3, model trainer(s) 118 may store the topology of NN model 120, store weights 220 and store precision 224 in one or more files (e.g., for subsequent loading and use when NN application(s) 110 utilizes NN model 120 for inference).

[0085] Precision may be fixed or monitored, determined and/or saved throughout training on multiple training input frames (e.g., training Ifmaps) to capture the dynamic range for the multiple frame inputs (e.g., over the whole training dataset). The precision information may be applied for numerous frames processed during inference by the trained model to provide dynamic control of ADC cycles, for example, by monitoring and storing the precision (e.g., effective number of bits) for many frames.

[0086] In step 418, back propagation (backprop) may be performed if the trained model is determined at step 410 to not be converged and return to step 406 following backprop. For example, as shown in FIGS. 1-3, model trainer(s) 118 may perform backprop on NN model 120 if the trained model is determined at step 410 to not be converged and return to step 406. Backprop may reduce error, loss and/or cost functions in an attempt to converge (e.g., while maintaining model precision). For example, an image from a dataset may be run through the computational graph (e.g., DNN). The results may be compared to a ground truth (e.g., labels). Error may be measured or calculated based on the difference between the results and labels. If the error is unsatisfactory, back propagation may be performed to propagate the error backwards (e.g., in reverse direction on the computational graph) to determine (e.g., learn) how the error may be created or affected by the weight values. One or more weight values in the DNN may be adjusted to reduce (e.g., minimize) the error.

[0087] In step 420, a network architecture search (NAS) improvement loop may be engaged if the model was determined to be converged at step 410 but determined to have failed to achieve one or more targets at step 412. For example, as shown in FIGS. 1-3, model trainer(s) 118 may perform the NAS improvement loop on NN model 120 if the model was determined to be converged at step 410 but determined to have failed to achieve one or more targets at step 412. Model trainer(s) 118 may return to step 402. For example, if NN model 120 is converged but a target is not achieved an attempt may be made to adjust the topology and/or hyper parameters, such as the number of layers, number of neurons, activation function(s), enhancer (e.g., improver, optimizer), learning rate, batch size, epochs, number of processing elements N summed together per MAC circuit (e.g., 64, 128, 512 based on vector length), and so on.

[0088] FIG. 5 shows a flowchart of a method 500 for using a trained model during inference to improve performance (e.g., reduce power consumption) of a hardware accelerator (e.g., neural processor) comprising a power-efficient hybrid or analog MAC circuit, according to an example embodiment. Embodiments disclosed herein and other embodiments may operate in accordance with example method 500. Method 500 comprises steps 502-512. However, other embodiments may operate according to other methods. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the foregoing discussion of embodiments. No order of steps is required unless expressly indicated or inherently required. There is no requirement that a method embodiment implement all of the steps illustrated in FIG. 5. FIG. 5 is simply one of many possible embodiments. Embodiments may implement fewer, more or different steps.

[0089] As shown in FIG. 5, in step 502, DNN topology, weights (W), Y precision, features or intermediate layer outputs (X) may be received and configured. For example, as shown in FIGS. 1-3, computing device(s) 104 may receive from server(s) 116 (e.g., based on a request by NN application(s) 110), NN model(s) 120 in the form of DNN topology, weights (W), Y precision, features or intermediate layer outputs (X). NN application(s) 110 may configure the DNN model according to received DNN topology, weights (W), Y precision, features or intermediate layer outputs (X). For example, a topology file with parameters (e.g., converged weights) in a file may be loaded and executed to infer object labels for new image frames with one or more objects.

[0090] In step 504, the type of activation function may be determined for MAC circuit operation. For example, as shown in FIGS. 1-3, MAC/AMAC circuit 218 (e.g., activation function detector 310) may determine the type of activation function (AF) 212 for use by ADC controller 308.

[0091] In step 506, vector/matrix multiplication and accumulation operations may be performed by a MAC circuit. For example, as shown in FIGS. 1-3, MAC circuit 300 may perform vector/matrix multiplication and accumulation operations. Each of UEs 320 in each of PEs PE[0]-PE [N−1] may multiply inputs Xi[2:0] and weights Wi[2:0] to generate mid-term values coupled to charge collection bus 302 by capacitors C. Combiner 304 may generate an analog Psum value from charge collection bus 302 for each column of UEs 320 in PEs PE[0]-PE [N−1].

[0092] In step 508, an AF predictor (e.g., one or more AF predictors) may be determined. For example, as shown in FIGS. 1-3, ADC 306 may begin digitizing the analog Psum accumulated by combiner 304. The first bit determined may be the MSB (e.g., Y[4]). Look-ahead sign detector 312 may determine the sign of the digitized Psum value based on the MSB. The MSB or the sign determined from the MSB may be the AF predictor (e.g., a parameter that indicates or predicts the output of the AF function). For example, an MSB or the sign determined from the MSB may indicate the output that would be generated by a ReLU AF layer. In examples, a ReLU AF layer may convert negative sign Psum values to zero.

[0093] In step 510, partial sums (Psums) may be digitized based on the AF type determination and the AF predictor determination, such as based on a determination of ReLU type AF and determination of an MSB or sign of a Psum value. For example, as shown in FIGS. 1-3, ADC controller 308 may control the operation of ADC 306 based on the AF type determined by activation function detector 310 and the sign of a Psum value determined by look-ahead sign detector 312. In an example, ADC controller 308 may allow ADC 306 to convert an analog Psum value generated by combiner 304 based on a determination that the AF type is not ReLU and/or based on a determination that the sign of the Psum value is positive. ADC controller 308 may stop ADC 306 conversion of an analog Psum value generated by combiner 304 based on determinations that the AF type is ReLU and that the sign of the Psum value is negative. ADC controller 308 may stop the operation of ADC 306, for example, by interrupting power to ADC 306, by negating enablement signal(s) to ADC 306, by interrupting a clock to ADC 306, and/or the like.

[0094] In step 512, Psums may be collected and an output activation function (e.g., ReLU) may be applied to the Psums. For example, as shown in FIGS. 1-3, MAC circuit 218 may generate Psums, a Psum handler may collect and store Psums in output Fmap and Psum memory (not shown), and NN model 120 may apply output activation function 212 to the (e.g., stored) Psums. Positive Psums may be indicated by a full digital value Y[4:0] while negative Psums may be indicated by the MSB Y[4]. A ReLU AF may convert negative Psums to zero.

[0095] FIG. 6 shows a flowchart of a method 600 for using a trained model during inference to improve performance (e.g., reduce power consumption) of a hardware accelerator (e.g., neural processor) comprising a power-efficient hybrid or analog MAC circuit, according to an example embodiment. Embodiments disclosed herein and other embodiments may operate in accordance with example method 600. FIG. 6 shows an example implementation pertaining to a ReLU AF layer and an associated ReLU AF predictor (e.g., sign of MAC circuit output values). Method 600 comprises steps 602-612. However, other embodiments may operate according to other methods. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the foregoing discussion of embodiments. No order of steps is required unless expressly indicated or inherently required. There is no requirement that a method embodiment implement all of the steps illustrated in FIG. 6. FIG. 6 is simply one of many possible embodiments. Embodiments may implement fewer, more or different steps.

[0096] As shown in FIG. 6, in step 602, MAC operations may be performed on inputs and weights provided to a MAC circuit. For example, as shown in FIGS. 1-3, MAC circuit 218 or 300 may perform multiply and accumulate operations on respective input Ifmaps (X) and weights (W). Each of UEs 320 in each of PEs PE[0]-PE [N−1] may multiply inputs Xi[2:0] and weights Wi[2:0] to generate mid-term values coupled to charge collection bus 302 by capacitors C. Combiner 304 may generate an analog Psum value from charge collection bus 302 for each column of UEs 320 in PEs PE[0]-PE [N−1].

[0097] In step 604, an analog output may be generated from the MAC operations.

[0098] For example, as shown in FIGS. 1-3, charge collection bus 302 may accumulate charges from multiplication products while combiner 304 may combine the charges weighted based on their significance into an analog output (e.g., analog Psum value) provided to ADC 306.

[0099] In step 606, an ADC may partially convert the analog output to a digital output. For example, as shown in FIGS. 1-3, ADC 306 may convert the analog output received from combiner 304 into an MSB Y[4]. ADC 306 may begin digitizing the analog Psum accumulated by combiner 304. The first bit determined may be the MSB (e.g., Y[4]).

[0100] In step 608, a determination may be made whether the AF layer is a ReLU AF layer and whether MSB Y[4] indicates a negative value. If both are true then step 610 may be implemented. If either is false then step 612 may be implemented. For example, as shown in FIGS. 1-3, activation function detector 310 may determine the type of activation function (AF) 212. Look-ahead sign detector 312 may determine the sign of the digitized Psum value based on MSB Y[4].

[0101] In step 610, the power consumption of a MAC circuit may be reduced by stopping the operation of an ADC. For example, as shown in FIGS. 1-3, ADC controller 308 may determine the AF type to be ReLU and the sign to be negative. ADC controller 308 may stop operation of ADC 306 (e.g., after generated MSB Y[4]), given that a negative sign indicates or predicts that the ReLU AF layer will convert the negative digital output to zero.

[0102] In step 612, a MAC circuit may be allowed to complete conversion of the analog output to a digital output. For example, as shown in FIGS. 1-3, ADC controller 308 may determine the AF type to be other-than ReLU and/or the sign to be positive. ADC controller 308 may allow operation of ADC 306 to continue to complete conversion of digital output Y[4:0].

[0103] As noted herein, the embodiments described, along with any circuits, components and/or subcomponents thereof, as well as the flowcharts/flow diagrams described herein, including portions thereof, and/or other embodiments, may be implemented in hardware, or hardware with any combination of software and/or firmware, including being implemented as computer program code configured to be executed in one or more processors and stored in a computer readable storage medium, or being implemented as hardware logic/electrical circuitry, such as being implemented together in a system-on-chip (SoC), a field programmable gate array (FPGA), and/or an application specific integrated circuit (ASIC). A SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and/or further circuits and/or embedded firmware to perform its functions.

[0104] FIG. 7 shows an exemplary implementation of a computing device 700 in which example embodiments may be implemented. Consistent with all other descriptions provided herein, the description of computing device 700 is a non-limiting example for purposes of illustration. Example embodiments may be implemented in other types of computer systems, as would be known to persons skilled in the relevant art(s).

[0105] As shown in FIG. 7, computing device 700 includes one or more processors, referred to as processor circuit 702, a system memory 704, and a bus 706 that couples various system components including system memory 704 to processor circuit 702. Processor circuit 702 is an electrical and/or optical circuit implemented in one or more physical hardware electrical circuit device elements and/or integrated circuit devices (semiconductor material chips or dies) as a central processing unit (CPU), a microcontroller, a microprocessor, and/or other physical hardware processor circuit. Processor circuit 702 may execute program code stored in a computer readable medium, such as program code of operating system 730, application programs 732, other programs 734, etc. Bus 706 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. System memory 704 includes read only memory (ROM) 708 and random-access memory (RAM) 710. A basic input/output system 712 (BIOS) is stored in ROM 708.

[0106] Computing device 700 also has one or more of the following drives: a hard disk drive 714 for reading from and writing to a hard disk, a magnetic disk drive 716 for reading from or writing to a removable magnetic disk 718, and an optical disk drive 720 for reading from or writing to a removable optical disk 722 such as a CD ROM, DVD ROM, or other optical media. Hard disk drive 714, magnetic disk drive 716, and optical disk drive 720 are connected to bus 706 by a hard disk drive interface 724, a magnetic disk drive interface 726, and an optical drive interface 728, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the computer. Although a hard disk, a removable magnetic disk and a removable optical disk are described, other types of hardware-based computer-readable storage media can be used to store data, such as flash memory cards, digital video disks, RAMs, ROMs, and other hardware storage media.

[0107] A number of program modules may be stored on the hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include operating system 730, one or more application programs 732, other programs 734, and program data 736. Application programs 732 or other programs 734 may include, for example, computer program logic (e.g., computer program code or instructions) for implementing example embodiments described herein.

[0108] A user may enter commands and information into the computing device 700 through input devices such as keyboard 738 and pointing device 740. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, a touch screen and/or touch pad, a voice recognition system to receive voice input, a gesture recognition system to receive gesture input, or the like. These and other input devices are often connected to processor circuit 702 through a serial port interface 742 that is coupled to bus 706, but may be connected by other interfaces, such as a parallel port, game port, or a universal serial bus (USB).

[0109] A display screen 744 is also connected to bus 706 via an interface, such as a video adapter 746. Display screen 744 may be external to, or incorporated in computing device 700. Display screen 744 may display information, as well as being a user interface for receiving user commands and/or other information (e.g., by touch, finger gestures, virtual keyboard, etc.). In addition to display screen 744, computing device 700 may include other peripheral output devices (not shown) such as speakers and printers.

[0110] Computing device 700 is connected to a network 748 (e.g., the Internet) through an adaptor or network interface 750, a modem 752, or other means for establishing communications over the network. Modem 752, which may be internal or external, may be connected to bus 706 via serial port interface 742, as shown in FIG. 7, or may be connected to bus 706 using another interface type, including a parallel interface.

[0111] As used herein, the terms “computer program medium,” “computer-readable medium,” and “computer-readable storage medium” are used to refer to physical hardware media such as the hard disk associated with hard disk drive 714, removable magnetic disk 718, removable optical disk 722, other physical hardware media such as RAMs, ROMs, flash memory cards, digital video disks, zip disks, MEMs, nanotechnology-based storage devices, and further types of physical/tangible hardware storage media. Such computer-readable storage media are distinguished from and non-overlapping with communication media (do not include communication media). Communication media embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wireless media such as acoustic, RF, infrared and other wireless media, as well as wired media. Example embodiments are also directed to such communication media that are separate and non-overlapping with embodiments directed to computer-readable storage media.

[0112] As noted above, computer programs and modules (including application programs 732 and other programs 734) may be stored on the hard disk, magnetic disk, optical disk, ROM, RAM, or other hardware storage medium. Such computer programs may also be received via network interface 750, serial port interface 742, or any other interface type. Such computer programs, when executed or loaded by an application, enable computing device 700 to implement features of example embodiments described herein. Accordingly, such computer programs represent controllers of the computing device 700.

[0113] Example embodiments are also directed to computer program products comprising computer code or instructions stored on any computer-readable medium. Such computer program products include hard disk drives, optical disk drives, memory device packages, portable memory sticks, memory cards, and other types of physical storage hardware.

[0114] Power efficient performance may be implemented in a hardware accelerator (e.g., a neural processor) comprising hybrid or analog multiply and accumulate (MAC) processing elements (PEs). For example, power consumption may be reduced in neural networks with a rectified linear unit (ReLU) activation layer. A hybrid or analog MAC circuit may be configured with a look-ahead sign detector to dynamically stop computations prior to completion, for example, based on detection of a negative value, which a ReLU activation layer may (e.g., subsequently) convert to zero. The sign of a value may be indicated by a most significant bit (MSB). A controller may provide power and/or clock cycles to an analog to digital converter (ADC) to determine a sign of a value being computed. The sign may be used to selectively complete computations for positive values and selectively terminate computations for negative values, thereby reducing power consumption of the MAC circuit.

[0115] In examples, an NPU may be configured to implement an AI NN model. The NPU may comprise at least one MAC circuit. A MAC circuit may include a plurality of MAC PEs. MAC PEs may include a multiplier (e.g., configured to perform multiplication operations on inputs and weights to generate mid-term values) and an accumulator configured to accumulate the mid-term values as an analog output. A MAC circuit may include an ADC configured to convert the analog output to a digital output. A MAC circuit may include an ADC (e.g., a power) controller configured to selectively stop (e.g., suspend operation of) the ADC before completely converting the analog output to the digital output based at least on a sign of the digital output, thereby reducing power consumption of the MAC circuit.

[0116] In examples, the sign of the digital output may be indicated by a most significant bit (MSB) of the digital output.

[0117] In examples, the ADC controller may be configured to selectively stop the ADC by selectively interrupting a power supply to the ADC.

[0118] In examples, the ADC may comprise a successive approximation register (SAR) ADC. The ADC controller may be configured to selectively stop the SAR ADC to reduce a number of cycles of the SAR ADC.

[0119] In examples, the ADC controller may be configured to selectively stop the ADC based on an indication of a type of activation function applied to the digital output and based on the sign of the digital output.

[0120] In examples, the ADC controller may be configured to selectively stop the ADC based on an indication that the activation function applied to the digital output comprises a rectified linear unit (ReLU) activation function and the sign of the digital output is negative.

[0121] In examples, a MAC circuit may (e.g., further) comprise an activation function detector configured to determine the type of activation function.

[0122] In examples, a MAC circuit may (e.g., further) comprise a look-ahead sign detector configured to determine the sign of the digital output.

[0123] In examples, a method may improve implementation of an artificial intelligence (AI) neural network (NN) model. The method may comprise, for example, performing multiply and accumulate (MAC) operations on inputs and weights provided to a MAC circuit; generating an analog output from the MAC operations; partially converting, by an analog to digital converter (ADC), the analog output to a digital output; determining a sign of the digital output based on the partial conversion; completing conversion of the analog output to the digital output by the ADC responsive to at least determining that the sign is a first sign; and reducing power consumption of the MAC circuit by stopping conversion of the analog output to the digital output by the ADC responsive to at least determining that the sign is a second sign, wherein the second sign different from the first sign.

[0124] In examples, the partially converting may comprise converting the analog output to a most significant bit (MSB) of the digital output. The sign of the digital output may be indicated by the MSB.

[0125] In examples, the stopping the conversion of the analog output to the digital output by the ADC may comprise interrupting a power supply to the ADC.

[0126] In examples, the ADC may comprise a successive approximation register (SAR) ADC. The stopping the conversion of the analog output to the digital output by the ADC may reduce the power consumption of the MAC circuit by reducing a number of cycles of the SAR ADC.

[0127] In examples, stopping the conversion of the analog output to the digital output by the ADC responsive to at least determining that the sign is the second sign may comprise stopping the conversion of the analog output to the digital output by the ADC responsive to an indication of a first type of activation function applied to the digital output and responsive to determining that the sign is the second sign.

[0128] In examples, the first type of activation function may be a rectified linear unit (ReLU) activation function, the second sign of the digital output may be negative and the first sign of the digital output may be positive.

[0129] In examples, a method may (e.g., further) comprise, for example, determining the type of activation function to be one of the first type of activation function or a second type of activation function.

[0130] In examples, a system may comprise one or more memory devices that store a deep neural network (DNN) topology. The system may comprise a neural processing unit (NPU). The NPU may comprise a multiply and accumulate (MAC) circuit. A MAC circuit may comprise a plurality of MAC processing elements (PEs) configured to generate an analog output from multiply and accumulation operations on inputs and weights; an analog to digital converter (ADC) configured to convert the analog output to a digital output; and an ADC (e.g., power) controller configured to selectively stop (e.g., suspend operation of) the ADC before completely converting the analog output to the digital output based at least on a sign of the digital output, thereby reducing power consumption of the MAC circuit.

[0131] In examples, the ADC controller may be configured to selectively stop the ADC based on an indication of a type of activation function applied to the digital output and based on the sign of the digital output.

[0132] In examples, the ADC controller may be configured to selectively stop the ADC based on an indication that the activation function comprises a rectified linear unit (ReLU) activation function and the sign of the digital output is negative.

[0133] In examples, the MAC circuit may (e.g., further) comprise an activation function detector configured to determine the type of activation function.

[0134] In examples, the MAC circuit may (e.g., further) comprise a look-ahead sign detector configured to determine the sign of the digital output.

[0135] While various examples have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be understood by those skilled in the relevant art(s) that various changes in form and details may be made therein without departing from the spirit and scope of the present subject matter as defined in the appended claims. Accordingly, the breadth and scope of the present subject matter should not be limited by any of the above-described examples, but should be defined only in accordance with the following claims and their equivalents.