Multidimensional data processing device, method, and computer-readable recording medium

[0001] This application is a National Stage Entry of PCT/JP2020/020567 filed on May 25, 2020, the contents of all of which are incorporated herein by reference, in their entirety.

[0002] The present invention relates to a neural network processing device and a neural network processing method that perform a process of a neural network, and a computer-readable recording medium recording a neural network processing program.

[0003] In a neural network, a block is a batch of multiple layers which are basic components.

[0004] NPL 1 describes a SE (Squeeze-and-Excitation) block, as a block that improves the accuracy of CNN (Convolutional Neural Network). FIG. 8 is a schematic diagram showing the SE block described in NPL 1. NPL 1 shows the case where 3 dimensional data U is input to the SE block. FIG. 9 is a schematic diagram showing the 3 dimensional data U input to the SE block.

[0005] The individual dimensions in the 3 dimensional data are referred to as the H dimension, the W dimension, and the C dimension. The H dimension is, for example, the dimension related to the height of an image. The W dimension is, for example, the dimension related to the width of the image. The C dimension is the dimension related to the channel. It is assumed that the number of elements of the H dimension in the 3 dimensional data U is H. It is assumed that the number of elements of the W dimension in the 3 dimensional data U is W. It is assumed that the number of elements of the C dimension in the 3 dimensional data U is C. The size of the 3 dimensional data U can be expressed as H×W×C.

[0006] In the Global Pooling layer (step S101), the number of elements of the H dimension and the W dimension are respectively 1. The number of elements of the C dimension remains unchanged at C. In other words, based on the 3 dimensional data U whose size is H×W×C, 1 dimensional data whose size is 1×1×C is generated. FIG. 10 is a schematic diagram showing the 1 dimensional data obtained in the Global Pooling layer.

[0007] In the first FC (Fully Connected) layer (step S102), the number of elements in the 1 dimensional data obtained in the Global Pooling layer is reduced. FIG. 11 is a schematic diagram showing the 1 dimensional data obtained in the FC layer. Here, the number of elements after the reduction is A. A<C.

[0008] FIG. 12 is a schematic diagram showing the process in the first FC layer (step S102). In the first FC layer (step S102), the number of elements that are outputs is less than the number of elements that are inputs. Then, the elements that are inputs and the elements that are outputs are fully connected as shown in FIG. 12, and weights are determined for respective individual connections. When the number of elements that are inputs is C and the number of elements that are outputs is A, the number of weights is C×A. Each weight is determined in advance by learning. The value of an element that is an output is calculated based on the values of the individual elements that are inputs connected with the element and the weights determined for each pair of the element that is an output and the individual element that is an input. By finding the values of the A elements that are outputs, 1 dimensional data (see FIG. 11) whose number of elements is A is obtained.

[0009] In the ReLU (Rectified Linear Unit) layer (Step S103), among the elements in the 1 dimensional data obtained in the FC layer (Step S102), the values of elements with negative values are changed to 0. The values of elements with values equal to or greater than 0 are not changed. In the ReLU layer, the number of elements in 1 dimensional data remains unchanged at A.

[0010] In the second FC layer (step S104), the number of elements in the 1 dimensional data obtained in the ReLU layer is increased back to the original number of elements (C elements).

[0011] FIG. 13 is a schematic diagram showing the process in the second FC layer (step S104). In the second FC layer (step S104), the number of elements that are outputs is greater than the number of elements that are inputs. Then, the elements that are inputs and the elements that are outputs are fully connected as shown in FIG. 13, and weights are determined for respective individual connections. When the number of elements that are inputs is A and the number of elements that are outputs is C, the number of weights is A×C. Each weight is determined in advance by learning. The value of an element that is an output is calculated based on the values of the individual elements that are inputs connected with the element and the weights determined for each pair of the element that is an output and the individual element that is an input. By finding the values of the C elements that are outputs, 1 dimensional data whose number of elements is C is obtained.

[0012] The first FC layer and the second FC layer differ only in whether the number of elements that are outputs decreases or increases with respect to the number of elements that are inputs; the essential process is the same.

[0013] In the Sigmoid layer (step S105), the sigmoid function is applied to each element in the 1 dimensional data obtained in the second FC layer. In the Sigmoid layer, the number of elements in the 1 dimensional data remains unchanged at C.

[0014] Individual elements in the 1 dimensional data obtained by the Sigmoid layer are used as coefficients representing the degree of importance of the channel corresponding to the individual element. For example, the 0th element in the 1 dimensional data is the coefficient representing the degree of importance of the 0th channel.

[0015] In the Scale layer (step S106), the elements of each channel of the first input 3 dimensional data U (see FIG. 9) are multiplied by a coefficient indicating the degree of importance of that channel. At this time, by copying the 1 dimensional data obtained in the Sigmoid layer H×W times, 3 dimensional data whose size is H×W×C is generated. This 3 dimensional data is denoted by a symbol X′. Since the size of the 1 dimensional data obtained in the Sigmoid layer is 1×1×C, by copying this 1 dimensional data H×W times, 3 dimensional data X′ with size H×W×C is obtained. FIG. 14 is a schematic diagram showing the 3 dimensional data X′ obtained by copying the 1 dimensional data of size 1×1×C, H×W times. FIG. 15 is a schematic diagram showing calculation of element-wise product of the 3 dimensional data U and the 3 dimensional data X′. The sizes of both the 3 dimensional data U and the 3 dimensional data X′ are H×W×C and are common. Furthermore, the elements in the 3 dimensional data U and the elements in the 3 dimensional data X′ can both be specified by 3 dimension coordinates. Therefore, it is possible to associate elements in the 3 dimensional data U and elements in the 3 dimensional data X′ that share the same 3 dimension coordinates. As a result, the elements in the 3 dimensional data U and the elements in the 3 dimensional data X′ are associated one-to-one. By calculating the product of the values of elements for each pair of elements to be associated, new 3 dimensional data whose size is H×W×C is obtained. This 3 dimensional data is the result of the element-wise product of the 3 dimensional data U and the 3 dimensional data X′, and is the output of the Scale layer. The 3 dimensional data obtained by this element-wise product operation can be said to be the data obtained by multiplying the multiple elements for each individual channel of the 3 dimensional data U by the coefficient corresponding to the channel (coefficient representing the degree of importance).

[0016] The output of the Scale layer (element-wise product of the 3 dimensional data U and the 3 dimensional data X′) is also the output of the SE block.

[0017] NPL 1: Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Enhua Wu, “Squeeze-and-Excitation Networks”, [online], [retrieved Apr. 3, 2020], Internet<URL: https://arxiv.org/pdf/1709.01507.pdf>

[0018] The SE block can improve the accuracy of CNN. However, the SE block can significantly reduce processing speed.

[0019] Therefore, it is the object of the present invention to provide a neural network processing device, a neural network processing method, and a computer-readable recording medium recording a neural network processing program that can obtain the same CNN accuracy as the SE block and perform operations faster than the SE block.

[0020] A neural network processing device according to the present invention includes: 1 dimensional data generation means for generating, by setting the number of elements of each dimension other than predetermined one dimension to 1 based on multidimensional data corresponding to one input data, 1 dimensional data whose number of elements is the same as the number of elements of the predetermined one dimension in the multidimensional data; number of elements reducing means for reducing the number of elements included in the 1 dimensional data; copy means for copying the 1 dimensional data whose number of elements has been reduced multiple times, thereby to generate multidimensional data in which the number of elements of each dimension other than the predetermined one dimension is restored to original number of elements; convolution layer processing means for performing a convolution layer process with a filter size of 1×1 on the multidimensional data generated by the copy means, thereby to generate multidimensional data in which the number of element of the predetermined one dimension in multidimensional data generated by the copy means is restored to original number of elements; and element-wise product operation means for performing an element-wise product operation, based on the multidimensional data corresponding to one input data and the multidimensional data generated by the convolution layer processing means.

[0021] A neural network processing method according to the present invention includes: generating, by setting the number of elements of each dimension other than predetermined one dimension to 1 based on multidimensional data corresponding to one input data, 1 dimensional data whose number of elements is the same as the number of elements of the predetermined one dimension in the multidimensional data; reducing the number of elements included in the 1 dimensional data; executing a copy process of copying the 1 dimensional data whose number of elements has been reduced multiple times, thereby to generate multidimensional data in which the number of elements of each dimension other than the predetermined one dimension is restored to original number of elements; performing a convolution layer process with a filter size of 1×1 on the multidimensional data generated by the copy process, thereby to generate multidimensional data in which the number of element of the predetermined one dimension in multidimensional data generated by the copy process is restored to original number of elements;

[0022] and performing an element-wise product operation, based on the multidimensional data corresponding to one input data and the multidimensional data whose number of element of the predetermined one dimension is restored to original number of elements.

[0023] A computer-readable recording medium according to the present invention is a computer-readable recording medium in which a neural network processing program is recorded, wherein the neural network processing program causes a computer to execute: a 1 dimensional data generation process of generating, by setting the number of elements of each dimension other than predetermined one dimension to 1 based on multidimensional data corresponding to one input data, 1 dimensional data whose number of elements is the same as the number of elements of the predetermined one dimension in the multidimensional data; a number of elements reducing process of reducing the number of elements included in the 1 dimensional data; a copy process of copying the 1 dimensional data whose number of elements has been reduced multiple times, thereby to generate multidimensional data in which the number of elements of each dimension other than the predetermined one dimension is restored to original number of elements; a multidimensional data generation process of performing a convolution layer process with a filter size of 1×1 on the multidimensional data generated by the copy process, thereby to generate multidimensional data in which the number of element of the predetermined one dimension in multidimensional data generated by the copy process is restored to original number of elements; and an element-wise product operation process of performing an element-wise product operation, based on the multidimensional data corresponding to one input data and the multidimensional data generated by the multidimensional data generation process.

[0024] According to this invention, it is possible to obtain the same CNN accuracy as the SE block and perform operations faster than the SE block.

[0040] As mentioned above, the SE block can improve the accuracy of CNN. However, the SE block can significantly reduce processing speed.

[0041] The inventor of the present invention considered the following reasons for the reduced processing speed when SE block is used.

[0042] As mentioned above, in the SE block, in the Scale layer, the 1 dimensional data whose size is 1×1×C is copied H×W times to obtain the 3 dimensional data X′ (see FIG. 14) whose size is H×W×C. This H×W times copy process causes a large overhead.

[0043] In particular, when the number of elements in the 1 dimensional data is large (in other words, when the value of C is large), the number of times an element is read from a memory and written to memory becomes enormous, and therefore the overhead of H×W times copy process is also enormous. For example, it is assumed that the size of the 1 dimensional data obtained in the Sigmoid layer is 1×1×1024 (i.e. C=1024). It is assumed that the size of the 3 dimensional data U is 7×7×1024. In other words, H=7 and W=7. In this case, for each of the 1024 elements in the 1 dimensional data, the read and write processes must be performed 7×7=49 times, resulting in a very large overhead due to the copy process.

[0044] The inventor of the present invention considered that the large overhead caused by this copy process was the cause of the slow processing speed in the SE block. Based on this consideration, the inventor made the following invention.

[0045] Example embodiment of the present invention is described below with reference to the drawings.

[0046] Multidimensional data is input to the neural network processing device of the example embodiment of the present invention. One multidimensional data corresponds to one input data. In order to make the invention easier to understand, the case in which the multidimensional data corresponding to one input data is 3 dimensional data will be used as an example in the present example embodiment. Even if the multidimensional data is other than 3 dimensional data, the same processing as in the following example embodiment can be applied.

[0047] The multidimensional data (3 dimensional data in the present example embodiment) input to the neural network processing device in the present example embodiment is denoted by the symbol U. As in the previous case, the individual dimensions in the 3 dimensional data U will be referred to as the H dimension, the W dimension, and the C dimension. The H dimension is, for example, the dimension related to the height of an image. The W dimension is, for example, the dimension related to the width of the image. The C dimension is the dimension related to the channel. It is assumed that the number of elements of the H dimension in the 3 dimensional data U is H. It is assumed that the number of elements of the W dimension in the 3 dimensional data U is W. It is assumed that the number of elements of the C dimension in the 3 dimensional data U is C. The size of the 3 dimensional data U can be expressed as H×W×C. The 3 dimensional data U can be represented schematically as shown in FIG. 9.

[0048] FIG. 1 is a block diagram showing an example configuration of a neural network processing device of the example embodiment of the present invention. A neural network processing device 1 of the present example embodiment includes a 1 dimensional data generation unit 2, a FC layer processing unit 3, a ReLU layer processing unit 4, a copy unit 5, a convolution layer processing unit 6, a Sigmoid layer processing unit 7, and a Scale layer processing unit 8.

[0049] The 3 dimensional data U corresponding to one input data is input to the 1 dimensional data generation unit 2. Then, the 1 dimensional data generation unit 2, by setting the number of elements of each dimension other than the predetermined one dimension among the three dimensions (H dimension, W dimension, and C dimension) to 1, generates 1 dimensional data whose number of elements is the same as the number of elements of the predetermined one dimension in the 3 dimensional data U.

[0050] In the present example embodiment, it is assumed that the above predetermined one dimension is the “C dimension”. In this case, the 1 dimensional data generation unit 2 sets the number of elements of each dimension other than the C dimension (H dimension and W dimension) to 1, thereby generates the 1 dimensional data whose number of elements is the same as the number of elements of the C dimension (i.e., C elements) in the 3 dimensional data U. The size of this 1 dimensional data is 1×1×C. The 1 dimensional data generated by the 1 dimensional data generation unit 2 can be represented schematically as shown in FIG. 10.

[0051] The 1 dimensional data generation unit 2 may, for example, generate 1 dimensional data by performing the same process as the Global Pooling layer. For example, the 1 dimensional data generation unit 2 may obtain the average value (which may be the maximum value) of the H×w elements in the 0th channel of the 3 dimensional data U, and determine the value as the value of the 0th element in the 1 dimensional data. The 1 dimensional data generation unit 2 performs the same process for each channel after the first, and determines the value of the element corresponding to each channel. As a result, C elements from the 0th to the C-1st are obtained, and 1 dimensional data with those elements is obtained. In this process, the 1 dimensional data generation unit 2 sets the number of elements of the H dimension and the number of elements of the W dimension to 1. The size of the 1 dimensional data obtained in this process is 1×1×C. The number of elements of the H dimension and the number of elements of the W dimension are both 1. Here, a case in which the 1 dimensional data generation unit 2 generates 1 dimensional data by performing the same process as it of the Global Pooling layer is described, but the 1 dimensional data generation unit 2 may generate 1 dimensional data whose size is 1×1×C by other methods.

[0052] The FC layer processing unit 3 reduces the number of elements in 1 dimensional data by performing FC layer processing on the 1 dimensional data obtained by 1 dimensional data generation unit 2. As in the previous case, the number of elements after the reduction is A. A<C.

[0053] The processing of FC layer processing unit 3 is the same as the processing of the first FC layer of the SE block (step S102 in FIG. 8). That is, the FC layer processing unit 3 takes the C elements in the 1 dimensional data obtained by 1 dimensional data generation unit 2 as the elements that are inputs. In addition, the FC layer processing unit 3 takes the A elements after the number of elements is reduced as the elements that are outputs. The elements that are inputs and the elements that are outputs are fully connected as shown in FIG. 12, and the weights of each connection are determined in advance by learning. In this case, C×A weights have been determined by learning in advance. The FC layer processing unit 3 calculates one element in A elements based on the values of each element of C and the weights determined for each pair of the one element and each of C elements. The FC layer processing unit 3 calculates values of each of A elements, to derive the 1 dimensional data whose number of elements is A. The size of this 1 dimensional data is 1×1×A. The 1 dimensional data derived by the FC layer processing unit 3 can be schematically represented as shown in FIG. 11.

[0054] It could be said that the FC layer processing unit 3 performs a process to reduce the number of elements in the 1 dimensional data.

[0055] The ReLU layer processing unit 4 changes the values of the elements in the 1 dimensional data derived by the FC layer processing unit 3 that have negative values to 0. The ReLU layer processing unit 4 does not change the values of the elements whose values are equal to or greater than 0. The number of elements in the 1 dimensional data remains unchanged at A by the process of the ReLU layer processing unit 4. The ReLU layer processing unit 4 may not be included in the neural network processing device 1, and the above processing by the ReLU layer processing unit 4 may be omitted. Also, instead of the ReLU layer processing unit 4, a component that applies an arbitrary activation function to 1 dimensional data may be included in the neural network processing device 1.

[0056] The copy unit 5 copies the 1 dimensional data after processing by ReLU layer processing unit 4 multiple times (more specifically, H×W times) to generate multidimensional data in which the number of elements of each dimension other than the predetermined one dimension (C dimension) is restored to the original number of elements. Here, the “original number of elements” is the number of elements in the input multidimensional data (3 dimensional data U in the present example embodiment). In other words, by copying 1 dimensional data whose size is 1×1×A, H×W times, the copy unit 5 generates 3 dimensional data in which the number of elements of the H dimension is H, the number of elements of the W dimension is W, and the number of elements of the C dimension is A. The size of this 3 dimensional data is H×W×A.

[0057] Hereafter, this 3 dimensional data is referred to as pre-convolution data. FIG. 2 is a schematic diagram showing the pre-convolution data.

[0058] The convolution layer processing unit 6 performs a convolution layer process with a filter size of 1×1 on the pre-convolution data, thereby to generate 3 dimensional data in which the number of elements (A) in predetermined one dimension (C dimension) is restored to the original number of elements (C). The size of this 3 dimensional data is H×W×C. FIG. 3 is a schematic diagram showing the pre-convolution data and 3 dimensional data (hereinafter referred to as 3 dimensional data Y) obtained after the execution of the convolution layer process with the filter size of 1×1. In FIG. 3, the elements of C dimension are shown vertically aligned for convenience.

[0059] The elements in the 3 dimensional data can be specified by 3 dimension coordinates. In the pre-convolution data, the values of the elements specified by the H dimension coordinate h, the W dimension coordinate w, and the C dimension coordinate c are expressed as (h, w, c)_before. Similarly, in the 3 dimensional data Y, the values of the elements specified by the H dimension coordinate h, the W dimension coordinate w, and the C dimension coordinate c are expressed as (h, W, C)_after.

[0060] The size 1×1 filter values used by the convolution layer processing unit 6 are defined as C sets of A filter values as 1 set. That is, A×C filter values are determined. The A×C filter values are determined in advance by learning.

[0061] The 0th set of A filter values is used to calculate the value of each element of the 0th channel in the 3 dimensional data Y. Similarly, the i-th set of A filter values (i is an integer such that 0≤i≤C−1) is used to calculate the value of each element of the i-th channel in the 3 dimensional data Y.

[0062] For example, it is assumed that the 0th set of A filter values is a_{(0, 0)}, a_{(0, 1)}, . . . , a_{(0, A-1)}, in order from the 0th. In this case, the convolution layer processing unit 6 calculates the value of (0, 0, 0)_alter by the following formula (1).

[0063] $\begin{array}{cc}\begin{array}{c}{\left(0,0,0\right)}_{\mathrm{after}}={\left(0,0,0\right)}_{\mathrm{before}}\times a_{\left(0,0\right)}\\ +{\left(0,0,1\right)}_{\mathrm{before}}\times a_{\left(0,1\right)}\\ +{\left(0,0,2\right)}_{\mathrm{before}}\times a_{\left(0,2\right)}\\ \cdots \\ +{\left(0,0,A-1\right)}_{\mathrm{before}}\times a_{\left(0,A-1\right)}\end{array}& \left(1\right)\end{array}$

[0064] The convolution layer processing unit 6 also obtains the values of the other elements of the 0th channel in the 3 dimensional data Y using the 0th set of A filter values by the similar calculation.

[0065] It is assumed that the i-th set of A filter values is a_{(i, 0)}, a_{i, 1)}, . . . , a_{(i, A−1)}, in order from the 0th. In this case, the convolution layer processing unit 6 calculates the value of (0, 0, i)_after by the following formula (2).

[0066] $\begin{array}{cc}\begin{array}{c}{\left(0,0,i\right)}_{\mathrm{after}}={\left(0,0,0\right)}_{\mathrm{before}}\times a_{\left(i,0\right)}\\ +{\left(0,0,1\right)}_{\mathrm{before}}\times a_{\left(i,1\right)}\\ +{\left(0,0,2\right)}_{\mathrm{before}}\times a_{\left(i,2\right)}\\ \cdots \\ +{\left(0,0,A-1\right)}_{\mathrm{before}}\times a_{\left(i,A-1\right)}\end{array}& \left(2\right)\end{array}$

[0067] The convolution layer processing unit 6 also obtains the values of the other elements of the i-th channel in the 3 dimensional data Y using the i-th set of A filter values by the similar calculation.

[0068] The convolution layer processing unit 6 uses the above calculation to calculate the value of each element of the 0th channel, the value of each element of the 1st channel, . . . , the value of each element of the C-1st channel in the 3 dimensional data Y. Then, the convolution layer processing unit 6 performs the same process for all positions in the plane consisting of the H dimension and W dimension in the pre-convolution data. In other words, the convolution layer processing unit 6 calculates the values of all elements in the 3 dimensional data Y. Then, the convolution layer processing unit 6 derives the 3 dimensional data Y. As a result, 3 dimensional data Y whose size is H×W×C is obtained.

[0069] The Sigmoid layer processing unit 7 applies a sigmoid function to the individual elements in the 3 dimensional data Y derived by the convolution layer processing unit 6. As a result, the value of each element in the 3 dimensional data changes to a value in the range of 0 to 1. The size of the 3 dimensional data is not changed by the processing by the Sigmoid layer processing unit 7. The 3 dimensional data after the processing by the Sigmoid layer processing unit 7 is denoted by the symbol Y′. The neural network processing device 1 may not include the Sigmoid layer processing unit 7, and the above processing by the Sigmoid layer processing unit 7 may be omitted. Also, instead of the Sigmoid layer processing unit 7, a component that applies a function other than the sigmoid function to individual elements in the 3 dimensional data Y may be included in the neural network processing device 1.

[0070] The A×C filter values (value of filter of filter size 1×1) used by the convolution layer processing unit 6 can also be referred to as weights. Then, it can be said that the calculation of C values from (0, 0, 0)_after to (0, 0, C−1)_after (see right side of FIG. 3) is the same as the process to obtain 1 dimensional data with C elements by obtaining the values of C elements in the second FC layer in the SE block (see step S104 in FIG. 8).

[0071] Therefore, the 3 dimensional data Y′ obtained by the Sigmoid layer processing unit 7 is the same as the 3 dimensional data X′ obtained by the copy process performed in the Scale layer of the SE block (see step S106 in FIG. 8).

[0072] Therefore, each element of each channel in the 3 dimensional data Y′ is a coefficient that represents the degree of importance of the channel corresponding to that element (the channel of the 3 dimensional data U). For example, the values of the H×W elements in the 0th channel of the 3 dimensional data Y′ are common, and are the coefficients representing the degree of importance of the 0th channel of the 3 dimensional data U.

[0073] The Scale layer processing unit 8 generates 3 dimensional data by performing an element-wise product operation of the first input 3 dimensional data U and the 3 dimensional data Y′ obtained by the Sigmoid layer processing unit 7. The Scale layer processing unit 8 outputs the 3 dimensional data.

[0074] FIG. 4 is a schematic diagram showing calculation of the element-wise product of the 3 dimensional data U and the 3 dimensional data Y′ obtained by the Sigmoid layer processing unit 7. The sizes of both the 3 dimensional data U and 3 dimensional data Y′ are H×W×C, and are common. Furthermore, the elements in the 3 dimensional data U and the elements in the 3 dimensional data Y′ can both be specified by 3 dimension coordinates. Therefore, it is possible to associate elements in the 3 dimensional data U and elements in the 3 dimensional data Y′ that share the same 3 dimension coordinates. As a result, the elements in the 3 dimensional data U and the elements in the 3 dimensional data Y′ are associated one-to-one. The Scale layer processing unit 8 the generates 3 dimensional data by calculating the product of the values of elements for each pair of elements to be associated (in other words, by performing the element-wise product operation). The 3 dimensional data generated by the Scale layer processing unit 8 can be said to be the data obtained by multiplying the multiple elements for each individual channel of the 3 dimensional data U by the coefficient corresponding to the channel (coefficient representing the degree of importance).

[0075] The 1 dimensional data generation unit 2, the FC layer processing unit 3, the ReLU layer processing unit 4, the copy unit 5, the convolution layer processing unit 6, The Sigmoid layer processing unit 7 and the Scale layer processing unit 8 are realized, for example, by a CPU (Central Processing Unit) of a computer operating according to a neural network processing program. For example, the CPU may read the neural network processing program from a program storage medium such as a program storage device of the computer, and operate as the 1 dimensional data generation unit 2, the FC layer processing unit 3, the ReLU layer processing unit 4, the copy unit 5, the convolution layer processing unit 6, The Sigmoid layer processing unit 7 and the Scale layer processing unit 8 according to the neural network processing program.

[0076] Alternatively, the 1 dimensional data generation unit 2, the FC layer processing unit 3, the ReLU layer processing unit 4, the copy unit 5, the convolution layer processing unit 6, the Sigmoid layer processing unit 7, and the Scale layer processing unit 8 may each be realized by separate hardware.

[0077] Next, the processing flow will be described. FIG. 5 is a flowchart showing an example of the processing flow of the example embodiment of the present invention. In the following explanation, the case in which the input multidimensional data is the aforementioned 3 dimensional data U (see FIG. 9) will be used as an example. In addition, detailed explanations of matters already explained will be omitted.

[0078] When the 1 dimensional data generation unit 2 inputs of the 3 dimensional data U, the 1 dimensional data generation unit 2 generates the 1 dimensional data whose number of elements is the number of elements of the C dimension (C) in the 3 dimensional data U, by setting the number of elements of each dimension other than the C dimension (i.e., the H dimension and the W dimension) to 1 (step S1).

[0079] Next, the FC layer processing unit 3 reduces the number of elements in the 1 dimensional data obtained in step S1 by performing FC layer processing (step S2). It is assumed that the number of elements after the reduction is A. A<C.

[0080] Next, the ReLU layer processing unit 4 changes the values of the elements in the 1 dimensional data obtained in step S2 that have negative values to 0 (step S3). At this time, the ReLU layer processing unit 4 does not change the value of elements whose value is equal to or greater than 0.

[0081] Next, the copy unit 5 generates the pre-convolution data (see FIG. 2) whose size is H×W×A by copying the 1 dimensional data obtained in step S3 H×W times (Step S4).

[0082] Next, the convolution layer processing unit 6 performs convolution layer processing with a filter size of 1×1 on the pre-convolution data, thereby to generate 3 dimensional data Y in which the number of elements of the C dimension is restored to the original number of elements (C) (Step S5).

[0083] Next, the Sigmoid layer processing unit 7 derives the 3 dimensional data Y′ by applying a sigmoid function to the individual elements in the 3 dimensional data Y (step S6). The value of each element included in the 3 dimensional data Y′ is in the range of 0 to 1.

[0084] Next, the Scale layer processing unit 8 generates 3 dimensional data by performing the element-wise product operation of the first input 3 dimensional data U and the 3 dimensional data Y′, and outputs the 3 dimensional data (Step S7).

[0085] Next, the effect of the present example embodiment is explained. As mentioned above, the 3 dimensional data Y′ obtained by the Sigmoid layer processing unit 7 is the same as the 3 dimensional data X′ obtained by the copy process performed in the Scale layer of the SE block (see step S106 in FIG. 8). Therefore, in the present example embodiment, the same result as in the SE block is obtained by the element-wise product operation in the Scale layer processing unit 8. Therefore, according to the neural network processing device of the present example embodiment, the same CNN accuracy as the SE block can be obtained.

[0086] In the present example embodiment, the copy unit 5 copies the 1 dimensional data derived by the ReLU layer processing unit 4 multiple times. The size of this 1 dimensional data is 1×1×A. Also, A<C. Therefore, the overhead of the copy process in the present example embodiment is smaller than the overhead of the copy process in the SE block, which copies 1 dimensional data whose size is 1×1×C multiple times. Therefore, according to the present example embodiment, the operation can be performed at a faster speed than in the SE block.

[0087] In other words, according to the present example embodiment, it is possible to obtain the same CNN accuracy as the SE block and perform operations faster than the SE block. As a result, processing time can be reduced.

[0088] Note that the convolution layer processing unit 6 performs convolution layer process with a filter size of 1×1 on the pre-convolution data, therefore the amount of operations required to obtain the 3 dimensional data Y is larger than the amount of operations of the second FC layer in the SE block (see step S104 in FIG. 8). However, the speed of the convolution layer process is fast. Therefore, even if the amount of operations performed by the convolution layer processing unit 6 in obtaining the 3 dimensional data Y is large, the effect on the processing speed (processing time) is small, and as a result, the processing speed can be faster than in the SE block.

[0089] In a neural network, a block is a batch of multiple layers which are basic components, and the block is applied multiple times. Then, steps S1-S7 (see FIG. 5) of the present example embodiment can be considered as one block. This block is denoted as block P. The block P can be applied in multiple location in the neural network process. Here, the degree of the effect of block P (the effect of the present example embodiment) varies depending on the size of the input multidimensional data. Therefore, in the neural network process, block P may be applied to the location where the effect of block P is large. In the neural network process, the location where the effect is large may be specified in advance by experimentation or other means.

[0090] FIG. 6 is a schematic block diagram showing an example of computer configuration of the neural network processing device of the example embodiment of the present invention. The computer 1000 includes a CPU 1001, a main memory 1002, an auxiliary memory 1003, and an interface 1004.

[0091] The neural network processing device 1 of the example embodiment of the present invention is realized by a computer 1000. The operation of the neural network processing device 1 is stored in the auxiliary memory 1003 in the form of a neural network processing program. The CPU 1001 reads the neural network processing program from auxiliary memory 1003 and expands it to the main memory 1002, and executes the process described in the above example embodiment.

[0092] The auxiliary memory 1003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include magnetic disks connected via interface 1004, magneto-optical disks, CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), semiconductor memory, etc. When the program is delivered to the computer 1000 through a communication line, the computer 1000 may expand the program in the main memory 1002 and execute the process described in the above example embodiment according to the program.

[0093] Some or all of the components may be realized by general-purpose or dedicated circuitry, processor, or a combination of these. These may comprise a single chip or multiple chips connected via a bus. Some or all of the components may be realized by a combination of the above-mentioned circuitry, etc. and a program.

[0094] When some or all of components is realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally located or distributed. For example, the information processing devices and circuits may be realized as a client-and-server system, a cloud computing system, etc., each of which is connected via a communication network.

[0095] The following is an overview of the invention. FIG. 7 is a block diagram showing an overview of the neural network processing device of the present invention. The neural network processing device of the present invention includes 1 dimensional data generation means 92, number of elements reducing means 93, copy means 95, convolution layer processing means 96, and element-wise product operation means 98.

[0096] The 1 dimensional data generation means 92 (e.g., the 1 dimensional data generation unit 2) generates, by setting the number of elements of each dimension (e.g., the H dimension and the W dimension) other than predetermined one dimension (e.g., the C dimension (dimension of channel)) to 1 based on multidimensional data corresponding to one input data (e.g., the 3 dimensional data U), 1 dimensional data whose number of elements is the same as the number of elements of the predetermined one dimension in the multidimensional data (e.g., C).

[0097] The number of elements reducing means 93 (e.g., the FC layer processing unit 3) reduces the number of elements included in the 1 dimensional data.

[0098] The copy means 95 (e.g., the copy unit 5) copies the 1 dimensional data whose number of elements has been reduced multiple times, thereby to generate multidimensional data (e.g., pre-convolution data) in which the number of elements of each dimension (e.g., the H dimension and the W dimension) other than the predetermined one dimension is restored to original number of elements (e.g., H and W).

[0099] The convolution layer processing means 96 (e.g., convolution layer processing unit 6) performs a convolution layer process with a filter size of 1×1 on the multidimensional data generated by the copy means 95, thereby to generate multidimensional data in which the number of element of the predetermined one dimension (e.g., the C dimension) in multidimensional data generated by the copy means 95 is restored to original number of elements (e.g., C).

[0100] The element-wise product operation means 98 performs an element-wise product operation, based on the multidimensional data corresponding to one input data (e.g., the 3 dimensional data U) and the multidimensional data generated by the convolution layer processing means.

[0101] According to such a configuration, it is possible to obtain the same CNN accuracy as the SE block and perform operations faster than the SE block.

[0102] The neural network processing device may include sigmoid function processing means (e.g., the Sigmoid layer processing unit 7) for applying a sigmoid function to each element in the multidimensional data generated by the convolution layer processing means 96, and the element-wise product operation means 98 may perform the element-wise product operation of the multidimensional data corresponding to one input data and the multidimensional data after the sigmoid function is applied to each element.

[0103] The neural network processing device may include change means (e.g., the ReLU layer processing unit 4) for changing values of the elements in the 1 dimensional data whose number of elements has been reduced that have negative values to 0, and the copy means 95 may copy the 1 dimensional data after processing by the change means multiple times.

[0104] The multidimensional data corresponding to one input data may be 3 dimensional data, and the predetermined one dimension may be a dimension of channel.

[0105] Although the present invention has been described above with reference to example embodiment, the present invention is not limited to the above example embodiment. Various changes may be made to the structure and details of the present invention, that may be understood by those skilled in the art within the scope of the present invention.

[0106] The invention is suitable for a neural network processing device that performs a process of a neural network.

• • [0107] 1 Neural network processing device
  • [0108] 2 1 dimensional data generation unit
  • [0109] 3 FC layer processing unit
  • [0110] 4 ReLU layer processing unit
  • [0111] 5 Copy unit
  • [0112] 6 Convolution layer processing unit
  • [0113] 7 Sigmoid layer processing unit
  • [0114] 8 Scale Layer processing unit