Data processing method for recurrent neural network using neural network accelerator based on systolic array and neural network accelerator

[0001] This application claims priority to Korean Patent Application No. 10-2021-0174873, filed Dec. 8, 2021 and Korean Patent Application No. 10-2022-0154210, filed Nov. 17, 2022, the entire contents of which are incorporated herein for all purposes by this reference.

[0002] The following description relates generally to a data parallel processing method for a recurrent neural network in a neural network accelerator based on a systolic array. In particular, the present disclosure relates to a data distribution method in a memory of a neural network accelerator.

[0003] Artificial intelligence (AI) based operation (or computation) generally processes data according to an algorithm having a predetermined pattern. In this regard, AD-dedicated hardware has been developed. As a representative example, a neural network accelerator having a systolic array architecture optimized for matrix operation of input data reuse is used.

[0004] A representative neural network model processing time series data is a recurrent neural network (RNN) based model. The RNN-based model has limited ability to perform parallel processing, due to recurrent characteristics thereof using an operation result of a previous step. Thus, the neural network accelerator based on a systolic array of the related art is suitable for image processing having high inter-operation independency but not suitable for parallel processing of time series dependent data, such as voice data.

[0005] The foregoing is intended merely to aid in the understanding of the background of the present disclosure, and is not intended to mean that the present disclosure falls within the purview of the related art that is already known to those skilled in the art.

[0006] The following description is intended to propose a data processing method for processing time series data in parallel in a neural network accelerator based on a systolic array. The following description is also intended to propose a multiple placement processing method for voice data parallel processing in a neural network accelerator based on a systolic array.

[0007] In one general aspect, there is provided a voice data processing method including: receiving, by a data processing device, voice data of a user in a predetermined time section; generating, by the data processing device, a plurality of voice data units by dividing the voice data by sentence; generating, by the data processing device, a plurality of input vectors by vectorizing the plurality of voice data units; and inputting, by the data processing device, the plurality of input vectors to a neural network accelerator based on a systolic array so as to process the data. The neural network accelerator may place the plurality of input vectors in a plurality of lines of a memory, distribute a single weight matrix to the plurality of lines, apply weight tiles in a single line of the weight matrix to the plurality of input vectors, respectively, and place intermediate vectors generated by the application of the weight tiles to the plurality of input vectors in same lines as the input vectors used in operation, respectively.

[0008] In another aspect, there is a data processing method including: receiving, by a data processing device, data in a predetermined time section; separating, by the data processing device, the data into data units having an equal length; generating, by the data processing device, a plurality of input vectors by vectorizing the data units; and inputting, by the data processing device, the plurality of input vectors to a neural network accelerator based on a systolic array so as to process the data. The neural network accelerator may place the plurality of input vectors in a plurality of lines of a memory, distribute a single weight matrix to the plurality of lines, apply weight tiles in a single line of the weight matrix to the plurality of input vectors, respectively, and place intermediate vectors generated by the application of the weight tiles to the plurality of input vectors in same lines as the input vectors used in operation, respectively.

[0009] According to another aspect of the present disclosure, provided is a neural network accelerator based on a systolic array, the accelerator including: a processing element array including a plurality of processing elements; a memory; and a controller configured to generate intermediate vectors from each of a plurality of input vectors stored in the memory and a weight matrix using the processing element array and control an internal operation of a neural network to be performed for the intermediate vector. The controller may place the plurality of input vectors in a plurality of lines of a memory, distribute a single weight matrix to the plurality of lines, apply weight tiles in a single line of the weight matrix to the plurality of input vectors, respectively, and place intermediate vectors generated by the application of the weight tiles to the plurality of input vectors in same lines as the input vectors used in operation, respectively. The plurality of input vectors may be obtained by parallel-processing time series data which is vectorized information of the plurality of data units generated by separating source data of a predetermined time section to be analyzed into equal sizes.

[0016] Throughout the drawings and the detailed description, the same reference numerals refer to the same elements. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience.

[0017] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, with the exception of operations necessarily occurring in a certain order. Also, descriptions of features that are known in the art may be omitted for increased clarity and conciseness.

[0018] The features described herein may be embodied in different forms, and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure of this application.

[0019] As used herein, the term “and/or” includes any one and any combination of any two or more of the associated listed items.

[0020] The terminology used herein is for describing various examples only, and is not to be used to limit the disclosure. The articles “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises,” “includes,” and “has” specify the presence of stated features, numbers, operations, members, elements, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, members, elements, and/or combinations thereof.

[0021] The technologies described below relate to a data processing or operating method in a neural network accelerator based on a systolic array. Hereinafter, the neural network accelerator refers to a parallel processing accelerator having a systolic array-based architecture.

[0022] The technologies described below relate to a data processing or operating method in a neural network accelerator in a neural network accelerator suitable for time series data.

[0023] Time series data refers to data having characteristics in which data in a step is influenced by data in a previous process on the time axis. A representative example of time series data includes voice data. In the following description, voice data will be mainly described. However, it should be understood that the following description is not only applied to voice data but is applicable to various types of data having time series characteristics.

[0024] In addition, the technologies described below relate to a data processing or operating method in a neural network model, such as a recurrent neural network (RNN), having recurrent characteristics.

[0025] Furthermore, the technologies described below relate to a neural network accelerator capable of parallel processing of time series data.

[0026] In the following description, it will be assumed that a neural network for services, such as voice recognition, has been constructed (or trained) in advance. Thus, in the following description, descriptions of a learning process and an inference process of the neural network will be omitted.

[0027] Hereinafter, a unit that processes data using the neural network accelerator will be referred to as a data processing device. The data processing device may perform the inference process using the neural network accelerator. The data processing device may be implemented as a computing device, a personal computer (PC), a smart device, a server on a network, a chipset in which a program is embedded, or the like. In addition, the data processing device may be a service device that provides a service using the trained neural network.

[0028] FIG. 1 is a schematic block diagram illustrating an example of a data service providing process 100 using a neural network accelerator.

[0029] The data processing device receives time series data to be analyzed in 110. For example, the time series data may include voice data.

[0030] The data processing device may uniformly perform pre-processing for the input time series data in 120. The pre-processing may include a process of uniformly separating the time series data. In the pre-processing, the data processing device may separate the time series data into a plurality of data units. Here, the plurality of data units may be comprised of data having the same length. The plurality of data units may be comprised of data having different lengths. The data processing device may convert the plurality of data units into predetermined vectors.

[0031] The data processing device uniformly processes data by inputting the plurality of vectors to the neural network accelerator in 130. Data processing in the neural network accelerator will be described later.

[0032] The data processing device may produce predetermined service information on the basis of an output value of the neural network accelerator. The service information may be one of various pieces of information according to the neural network constructed in advance and services to be provided. For example, when the service is a voice, the output service information may be text corresponding to or extracted from the voice.

[0033] FIG. 1 illustrates a process of processing time series data in a predetermined section. The data processing device may repeat a process of processing time series data in a predetermined section and a process of not processing time series data in a next section.

[0034] FIG. 2 is a diagram illustrating an example of a voice recognition process 200 using the neural network accelerator. FIG. 2 illustrates an example of a process of performing a voice recognition application using the neural network accelerator.

[0035] The data processing device receives voice data of a user.

[0036] The data processing device separates voice in a predetermined section into a plurality of units in 210. For example, the data processing device may separate the input voice by the sentence. Variety of commonly known technologies may be used to separate a series of voices into predetermined units. The data processing device generates a plurality of voice data units.

[0037] The data processing device may the correct the plurality of separated voice data units to have the same voice length in 220. This is intended to make vectors input to the neural network have the same size.

[0038] The process of separating voice according to a predetermined standard and correcting separated voice units to have the same length may enable pieces of time series data to have operation independency and parallel processing to be performed.

[0039] The data processing device may convert a voice signal into a frequency range signal in 230.

[0040] The 210 to 230 steps may be pre-processing for time series data.

[0041] The data processing device may process pre-processed data in parallel using the neural network accelerator in 240. The data processing device may generate input vectors by uniformly vectorizing signals separated by the sentence. The data processing device loads a plurality of input vectors of the series of voices to a memory of the neural network accelerator in 241. The data processing device loads a weight matrix to the memory in 242. The weight matrix may be information set through a learning process. The weight matrix may be a matrix to be commonly used to input vectors. The data processing device performs operation to generate intermediate vectors by applying the weight matrix to the input vectors in 243. The data processing device may perform activation function operation for intermediate vectors in 244. Types of activation function are determined according to the constructed model. Tan h and sigmoid activation functions may be used in a long short-term memory model for natural language processing. A variety of activation functions (e.g., softmax, ReLU, etc.) may be used according to the type of neural network and the class of neural network. The data processing device may perform Hadamard product operation for the result of the intermediate vector operation and the result of the activation function operation in 245. The intermediate vector operation, the activation function operation, and the Hadamard product operation may be a series of operations performed in a long short-term memory (LSTM).

[0042] Furthermore, when data processing using the neural network accelerator for the series of voices is completed, the data processing device may perform post processing to convert the data into text in 250.

[0043] FIG. 2 illustrates an example of voice recognition. For example, a natural language processing model to which text is input may process data using a similar process. The data processing device may separate a predetermined section of text by the sentence, and generate plurality of tokens. The data processing device may perform parallel processing by inputting the plurality of tokens to the neural network accelerator.

[0044] FIG. 3 is a diagram illustrating an example of data placement in the memory of the neural network accelerator. FIG. 3 is an example of the data placement for parallel processing of data, such as voice, in a systolic array.

[0045] The left part in FIG. 3 illustrates data placed in an internal memory of the neural network accelerator. Referring to FIG. 3, the data placed in the internal memory includes N number of input vectors a placed in N number of lines (e.g., columns or rows) and a single weight matrix b distributed to N number of lines.

[0046] The neural network accelerator performs an operation, by a processor, using the weight matrix common to input vectors of respective lines, thereby generating intermediate vectors. The internal memory stores intermediate vectors c for the input vectors of respective lines in the same lines as the input vectors.

[0047] FIG. 4 is a diagram illustrating an example of placement of input vectors and a weight matrix in the memory of the neural network accelerator. In FIG. 4, accelerator processing for voice data will be described as an example. In FIG. 4, D and H refer to sizes (i.e., lengths) of corresponding memory fields.

[0048] Each of the input vector has a form in which x 1 as a result of pre-processing for input voice and H_t 2 initialized to a zero vector are connected.

[0049] The weight matrix is a matrix comprised of a plurality of rows and lines common to input vectors. The weight matrix is separated into 4H number of tiles, which are distributed to lines of the internal memory. In the weight matrix, weight tile₁ of the weight matrix participates in operation of single input vector₁. The weight matrix is comprised of W 3, R 4, and B 5.

[0050] In addition, the neural network accelerator may use double buffering in consideration of a delay time taken for data movement between an external memory and the internal memory.

[0051] FIGS. 5A and 5B are diagrams illustrating an example of a data operating process of a recurrent neural network in the neural network accelerator. FIGS. 5A and 5B illustrate an example of a process of performing LSTM operation using data stored in the memory of the neural network accelerator.

[0052] FIG. 5A illustrates an example of a process of intermediate vector operation and activation function operation. The neural network accelerator generates a single component of an intermediate vector (i, f, c, o) by multiplying a W component of a weight matrix by an x vector of input vectors, multiplying an R component of the weight matrix by H_t of input vectors, and adding a bias B to a resultant value ({circle around (1)} operation of intermediate vector). The neural network accelerator performs operation of an activation function, such as sigmoid and tan h, to the intermediate vector ({circle around (2)} operation of activation function). Shadowed components in FIG. 5A indicate result data of the operation, including the operation of the activation function.

[0053] The neural network accelerator performs intermediate vector operation and activation function operation for the input data by the number of lines of the internal memory. Afterwards, the neural network accelerator performs Hadamard product operation. FIG. 5B illustrates an example process of Hadamard product operation. The neural network accelerator generates final results C_t and H_t by performing Hadamard product operation using operated intermediate vector values and stores C_t and H_t in the memory ({circle around (3)} Hadamard product operation). The operation results C_t and H_t are used in operation of input data of a next t. C_t indicates a cell state also referred to as a long term state in the LSTM.

[0054] By placing the input vectors, the weight matrix, and the intermediate vectors in the memory as described above, the neural network accelerator may exclude inter-operation dependency by continuously performing operations while sharing weights by performing operations.

[0055] FIG. 6 is a diagram illustrating an example of a neural network accelerator 300 based on a systolic array. FIG. 6 corresponds to a neural network accelerator using the method of distributing data to the internal memory and the data processing method as described above. The neural network accelerator 300 includes an interface device 310, a controller 320, an internal memory 330, and a processing element array 340.

[0056] In FIG. 6, an external memory may store a plurality of input vectors generated by the data processing device having uniformly processed input data. Here, the input vectors may be vectors generated after source data is separated into a plurality of data units having a predetermined size. The source data may be time series data such as voice data. The above-described data processing device may have a configuration including an external memory, a processor performing data pre-processing, and the neural network accelerator 300.

[0057] The interface device 310 is a device that receives instructions and data from an external source. The interface device 310 may receive input vectors from a physically connected input device or an external memory. The interface device 310 may transfer the input vectors to the internal memory 330. Although not shown, the interface device 310 may store a result processed by the processing element array 340 in the internal memory or transfer the result to an external memory.

[0058] The controller 320 controls data processing and operations in the neural network accelerator 300.

[0059] The internal memory 330 stores data for processing data according to the architecture of the neural network.

[0060] The controller 320 may perform control so that the input vectors, the weight matrix, and the intermediate vectors placed as illustrated in FIGS. 3 and 4 are stored in the internal memory 330. The controller 320 may place the plurality of input vectors in a plurality of lines of the internal memory 330, distribute the single weight matrix to the plurality of input vectors, and place the intermediate vectors generated by application of weight tiles to the plurality of input vectors, respectively, in the same lines as the input vectors used in the operation.

[0061] As illustrated in FIGS. 5A and 5B, the controller 320 may produce an intermediate vector for an input vector for each of the plurality of input vectors by controlling the internal memory 310 and the processing element array 340. That is, the controller 320 may calculate the intermediate vector for the input vector using the input vector and a weight component (i.e., a weight tile) used in operation with the input vector among weight components of the weight matrix. The controller 320 may perform control so that the input vector for each of the plurality of input vectors, the weight component (i.e. a weight tile) used in operation with the input vector, and the intermediate vector for the input vector to be stored in the same line in the internal memory 330.

[0062] In addition, as illustrated in FIGS. 5A and 5B, the controller 320 may perform an internal operation of an additional neural network for the produced intermediate vector. For example, the controller 320 may control the processing element array 340 to perform a series of LSTM operations by applying an activation function to each of the intermediate vectors and performing the Hadamard product operation to the result obtained by the application of the activation function.

[0063] The processing element array 340 may be a systolic array-based parallel processor.

[0064] In addition, the data processing method of the neural network accelerator, the data placement method in the memory of the neural network accelerator, and the operating method using data placed in the memory of the neural network accelerator as described above may be implemented as a program (or an application) including an executable algorithm that may be executed in a computer. The program may be stored in a transitory computer readable medium or a non-transitory computer readable medium.

[0065] The non-transitory computer readable medium refers to a medium that stores data semi-permanently and is readable by devices, rather than a medium, such as a register, cash, and a memory, which stores data temporarily. More specifically, the above-described various applications or programs may be stored in the non-transitory computer readable medium, such as a compact disc (CD), a digital video disc (DVD), a hard disk, a Blu-ray disk, a universal serial bus (USB) memory, a memory card, read-only memory (ROM), programmable read only memory (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), or flash memory.

[0066] The transitory computer readable medium refers to various types of RAM, such as static RAM (SRAM), dynamic RAM (RAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous-link DRAM (SLDRAM), and direct Rambus RAM (DRRAM).

[0067] While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.