Recognition apparatus, recognition method, and storage medium

[0001] The present invention relates to a recognition apparatus, a recognition method, a storage medium, and the like.

[0002] At present, level 3 autonomous driving of automobiles, which are one type of movable apparatuses, on highways has been achieved. Level 3 autonomous driving refers to the system that is mounted on an automobile executing all of the operations necessary for driving such as steering, deceleration, acceleration, and the like when under predetermined conditions.

[0003] In addition, it is expected that level 4 autonomous driving of automobiles on freeways will be achieved in the near future. Level 4 autonomous driving refers to the continuous execution of all operations necessary for driving by a system that is mounted on an automobile.

[0004] In order to achieve the above-described autonomous driving, it is necessary for an automobile to be equipped with an image capturing apparatus that fulfills the role of the eyes of the operator. Such an image capturing apparatus must be able to not only recognize objects that are present on the road such as other automobiles and the like, but also be able to recognize the contents that are indicated on road signs, road information boards, road guidance boards, and the like with high precision.

[0005] Published Japanese Translation No. 2016-538665 of the PCT International Publication discloses a recognition method for traffic signs, and discloses that unclearness due to movements in a particular image region is calculated, this image region is made clear, and the traffic signs are recognized by referring to the image region that has been made clear.

[0006] However, in a case in which the resolution of an image that has been captured by an image capturing apparatus has been increased with the goal of recognizing road signs and the like with a high precision, the data amount becomes high, and smooth transmission becomes difficult.

[0007] In order to solve the problem that has been described above, the recognition apparatus according to one aspect of the present invention comprises at least one processor or circuit configured to function as: a detection unit configured to detect a subject to be recognized from a first image that has been captured by an image capturing apparatus; a determining unit configured to, in a case in which the subject has been detected from the first image, determine a region including at least a portion of the subject in the first image; a control unit configured to control the image capturing apparatus such that a second image that corresponds to the region is generated; an image processing unit configured to generate a third image by processing to increase a resolution of the second image; and a recognition unit configured to recognize contents that are indicated by the subject from the third image.

[0008] Further features of the present invention will become apparent from the following description of embodiments with reference to the attached drawings.

[0016] Hereinafter, with reference to the accompanying drawings, favorable modes of the present invention will be described using Embodiments. In each diagram, the same reference signs are applied to the same members or elements, and duplicate descriptions will be omitted or simplified.

[0017] Below, an Embodiment of the present invention will be explained with reference to FIG. 1 to FIG. 7. FIG. 1 is a diagram showing an example of a recognition system according to the Embodiment. As is shown in FIG. 1, a recognition system 1 is provided with an image capturing apparatus 11, and a recognition apparatus 12.

[0018] The image capturing apparatus 11 is, for example, a camera that is mounted in an arbitrary position on a movable apparatus such as an automobile or the like. The image capturing apparatus 11 is provided with, for example, as is shown in FIG. 1, an optical unit 111, and a sensor unit 112.

[0019] The optical unit 111 is an optical system that is mounted on the image capturing apparatus 11, is provided with lenses, mirrors, and the like, and optically forms a subject image on the sensor unit 112. The sensor unit 112 is provided with a micro-lens, a color filter, a light receiving element, and the like. In addition, the light receiving element is, for example, a photodiode.

[0020] The sensor unit 112 converts the optical subject image that has been formed by the optical unit 111 into an electric signal, and transmits this to the recognition apparatus 12. This electric signal includes a plurality of first images that forms a first movie image. Note that in the present embodiment generating an electrical image signal by converting the optical subject image formed by the optical unit 111 is called capturing an image.

[0021] FIG. 2 is a diagram showing an example of a first image that forms a first movie image according to the Embodiment. FIG. 2 shows a first image P1 that configures a first movie image. As is shown in FIG. 2, the first image P1 shows the road on which the vehicle on which the image capturing apparatus 11 is mounted is travelling, road signs that are placed by this road, and automobiles that are travelling in front of this vehicle.

[0022] In addition, the first image P1 is an image corresponding to the largest range that can be captured by the image capturing apparatus 11 has been captured. However, the first image may also be an image that was captured within a range that is smaller than the largest range that can be captured by the image capturing apparatus 11. The sensor unit 112, for example, transmits an electric signal showing the first image that is shown in FIG. 2 to the recognition apparatus 12.

[0023] In addition, the sensor unit 112 should have sufficient number of pixels such that processing to recognize regions in which there is an object to be detected such as a movable apparatus such as an automobile or the like, a road sign, a road information board, a road guidance board, or the like can be executed without any problems.

[0024] Such number is, for example, 2 million. However, the sensor unit 112 may also have a pixel number of an extent such that the processing by the recognition apparatus 12 to recognize the contents that are shown on an automobile registration number plate attached to an automobile, a road sign, a road information board, a road guidance board, or the like can be executed without any problems. Such a pixel number may be, for example, 12 million pixels. Note that in Japan, an automobile registration number plate is generally called as a “number plate”.

[0025] As is shown in FIG. 1, the recognition apparatus 12 is provided with, for example, an image processing unit 121, an object recognition unit 122, a detection unit 123, a determining unit 124, a control unit 125, a control unit 126, an image processing unit 127, an image synthesizing unit 128, and a recognition unit 129.

[0026] In addition, the recognition apparatus 12 is also provided with a ROM (Read Only Memory), a CPU (Central Processing Unit), and a RAM (Random Access Memory). The ROM stores programs that realize each of the image processing unit 121, the object recognition unit 122, the detection unit 123, the determining unit 124, the control unit 125, the control unit 126, the image processing unit 127, the image synthesizing unit 128, and the recognition unit 129. The CPU reads out and opens these programs on the RAM, and executes each program.

[0027] The image processing unit 121 receives the electrical signal showing the first image from the sensor unit 112, and executes image processing on the first image. This image processing is, for example, de-mosaic processing, wide dynamic range (WDR) correction processing, gamma correction processing, Look Up Table (LUT) correction processing, distortion correction processing, or dodging processing.

[0028] After this processing has been executed, the first image is transmitted to the object recognition unit 122, the image processing unit 127, and the image synthesizing unit 128. In addition, the first image is stored on a storage medium after this processing has been executed. The storage medium referred to in this context is, for example, a semi-conductor storage element such as a DRAM (Dynamic Random Access Memory), or the like, and is able to read and write data.

[0029] The object recognition unit 122 executes processing that recognizes a region in which there is an object to be detected from the first image after image processing has been executed. In addition, the object recognition unit 122 executes this processing using, for example, Artificial Intelligence (AI) including machine learning.

[0030] Machine learning that is referenced in this context is, for example, deep learning or YOLO (You Only Look Once). In particular, YOLO is able to rapidly recognize a region in the image in which there is an object to be recognized by using a neural network, and is therefore useful. The object recognition unit 122 transmits data showing the first image on which image processing has been executed, and data showing the results of this processing to the detection unit 123. In addition, the object recognition unit 122 stores these data on the above-described storage medium.

[0031] The detection unit 123 detects subjects to be recognized from the first image that has been captured by the image capturing apparatus 11. For example, the detection unit 123 identifies subjects from the plurality of first images that configure the first movie image. In addition, for example, the detection unit 123 executes this detection by using artificial intelligence, and stores data showing the detection results on the above-described storage medium.

[0032] Furthermore, this data may also include not only the detection results, but also at least one of the type of subject, or the position or the orientation of the subject in the first image, and the like. In addition, the subject to be recognized that is referred to in this context, is for example, an object that indicates information necessary in order to autonomously move a movable apparatus such as an automobile or the like using at least one of written characters, numerals, symbols, or graphics, and can be arbitrarily set.

[0033] FIG. 3 is a diagram showing an example of a subject to be recognized by the recognition apparatus according to the Embodiment. For example, the detection unit 123 determines that a road sign T1 and an automobile T2 that are shown in FIG. 3 are subjects to be recognized, and that these subjects are detected from the first image P1.

[0034] The determining unit 124 determines a region that includes at least a portion of the subject in the first image in a case in which a subject has been detected from the first image. In addition, in a case in which it has been identified that the subject is shown in the plurality of first images, the determining unit 124 determines the region such that this is the same range in the plurality of first images.

[0035] Furthermore, the determining unit 124 preferably determines this region such that all of the subjects to be recognized are included, and the area of this region becomes as small as possible. This is because it is necessary to avoid a situation in which the contents that are indicated by the subject to be recognized are failed to be recognized, while reducing data amount for a third image and a third movie image that will be described below.

[0036] FIG. 4 is a diagram showing an example of a region that has been determined by the recognition apparatus according to the embodiment. For example, the determining unit 124 determines a rectangular region A that includes the road sign T1 and the automobile T2 that are the subjects to be recognized in the first image P1. As is shown in FIG. 4, in the region A, the coordinates for the upper left vertex in the first image P1 are (X1, Y1), and the coordinates for the lower right vertex are (X2, Y2).

[0037] The control unit 125 controls the image capturing apparatus 11 so as to generate a second image corresponding to the region that has determined by the determining unit 124. For example, the control unit 125 controls the image capturing apparatus 11 so as to capture the second movie image consisting of a plurality of second images corresponding to the region that has been determined by the determining unit 124.

[0038] FIG. 5 is a diagram showing an example of a second image that corresponding to the second movie image according to the embodiment. In a case in which the first image P1 has been captured, the control unit 125 controls the image capturing apparatus 11 so as to capture the second image P2 such that corresponds to only the region A.

[0039] In other words, in a case in which the first image P1 has been captured, the control unit 125 controls the image capturing apparatus 11 so as to capture a second movie image consisting of a plurality of second images including the second image P2.

[0040] In addition, the second image P2 has the same position, orientation, and shape as the region A that was shown in FIG. 4, and the coordinates for the upper left vertex thereof are (X1, Y1), and the coordinates for the lower right vertex are (X2, Y2). In addition, the pixel number for the second image P2 is, for example, 2 million pixels. Note that the rectangle B shown in FIG. 5 is a rectangle that has the same position, orientation, and shape as the first image P1 that was shown in FIG. 2 to FIG. 4.

[0041] The determining unit 124 and the control unit 125 make it possible to suppress the data amounts for the third image and the third movie image, which will be described below, by executing the above-described processing.

[0042] The control unit 126 controls the image capturing apparatus 11 so as to increase the number per unit of time of the plurality of second images that forms the second movie image. FIG. 6 is a diagram showing an example of a frame rate for the first movie image and a frame rate for the second movie image according to the embodiment.

[0043] FIG. 6 shows a time period C1, a time period C2, and a time period C3, during which the second movie image is captured by the image capturing apparatus 11. The control unit 126, for example, controls the image capturing apparatus 11 so as to switch from a frame rate for the second movie image of 30 fps (frames per second) to 300 fps at the point in time when the time period C1 has elapsed and the time period C2 begins, as is shown in FIG. 6.

[0044] In addition, after this, for example, the control unit 126 controls the image capturing apparatus 11 such that it switches from a framerate for the second movie image of 300 fps to 30 fps at the point in time when the time period C2 has elapsed and the time period C3 begins, as is shown in FIG. 6.

[0045] In addition, the control unit 126 may also control the image capturing apparatus 11 so as to increase the number per unit of time of second images that forms the second movie image according to a resolution that has been set by a processing to increase the resolution of each of the second images that forms the second movie image.

[0046] The image processing unit 127 generates a third image by processing to increase the resolution of a second image. Conversely, the image processing unit 127 generates a third movie image consisting of a plurality of third images by generating a plurality of third images by applying processing to increase the resolution of each of the plurality of second images that forms the second movie image.

[0047] In addition, the image processing unit 127 can also execute this processing on either of a second image in which the frame rate is controlled by the control unit 126, or a second image in which the frame rate is not controlled by the control unit 126. Note that at least one of the third image or the third movie image may also be displayed on a suitable monitor display to be used for any arbitrary purpose.

[0048] In addition, the image processing unit 127 may also generate a third image by applying super resolution processing to the plurality of second images. Super resolution processing is a technology that performs feedback to super resolution images that have been estimated by a super resolution image until the difference between each of a plurality of real images and a plurality of estimated images is converged, and increases the resolution of an image in which a subject has been captured.

[0049] The plurality of real images is generated by image capturing of the same subject performed by an image capturing apparatus, and these are images in which the positions of the subject in each of the images are deviated from one another. The estimated images are images that have been generated by applying the results of a simulation of a decrease in quality that occurs during the process of the image capturing of the subject by the image capturing apparatus to a super resolution image.

[0050] In a case in which a second movie image is captured that is configured by a plurality of second images, these second images show the same subject to be recognized, and the positions of this subject in the second images are deviated from one another, and therefore, these become the plurality of real images that was described above.

[0051] In addition, in a case in which the number per unit of time of second images that configure the second movie image has been increased by the control unit 126, these second images often become a further optimal plurality of real images. This is because the deviations between the positions of the same subject to be recognized within the second images often become smaller.

[0052] Therefore, super resolution technology is the preferable processing for when the image processing unit 127 generates the third image. Note that the image processing unit 127 may also use a technology other than a super resolution technology in order to generate the third image by increasing the resolution of the second image.

[0053] The image processing unit 127 generates the third image or the third movie image by executing the above-described processing, and it thereby becomes possible for the recognition unit 129 to recognize the contents that are indicated by the subject with a high degree of precision.

[0054] The image synthesizing unit 128 synthesizes the above described third image and the first image that is stored on the above-described storage medium, and transmits a movie image with a normal frame rate in which only the resolution for the region that was determined by the determining unit 124 has been increased to a movable apparatus control system 13.

[0055] The movable apparatus control unit 13 is, for example, an electronic control unit (ECU). The movable apparatus control unit 13 is able to integrally control the movable apparatus, to control the movable apparatus by using data and the like that have been received from the recognition apparatus 12, and to perform appropriate output to an external monitor.

[0056] The recognition unit 129 recognizes the contents that are indicated by the subject from the third image. In other words, the recognition unit 129 recognizes the contents that are indicated on a subject that is shown in the third movie image. In addition, the recognition unit 129 recognizes at least one of written characters, numerals, symbols, or graphics that are indicated by the subject.

[0057] Furthermore, the recognition unit 129 may also recognize the coordinates of at least one of the written characters, numerals, symbols, or graphics that are indicated by the subject. In addition, the recognition unit 129 transmits data showing the recognition results to the movable apparatus control unit 13.

[0058] Next, the processing that is executed by the recognition system 1 will be explained while referencing FIG. 7. FIG. 7 is a flowchart showing processing that is executed by the recognition system according to the Embodiment.

[0059] In step S1, the image capturing apparatus 11 captures the first movie image including the plurality of first images.

[0060] In step S2, the object recognition unit 122 executes processing to recognize regions in which there is an object to be detected from the first image.

[0061] In step S3, the detection unit 123 identifies whether or not a subject to be recognized has been detected from the first image. In a case in which the detection unit has identified that a subject to be recognized has been detected in the first image (step S3: YES), the processing proceeds to step S4. In contrast, in a case in which the detection unit 123 has identified that a subject to be recognized has not been detected in the first image (step S3: NO), the processing shown in FIG. 7 is ended.

[0062] In step S4, the determining unit 124 determines a region that includes at least a portion of the subject in the first image, and which has the same region in the plurality of first images.

[0063] In step S5, the control unit 125 controls the image capturing apparatus 11 so as to generate a second movie image including the plurality of second images corresponding to the region that has been determined in step S4.

[0064] In step S6, the image processing unit 127 generates a third movie image including the plurality of third images which are generating by performing processing to increase the resolution of each of the plurality of second images forming the second movie image.

[0065] In step S7, the recognition unit 129 recognizes the contents that are indicated by the subject from the third movie image, then the processing shown in FIG. 7 is ended.

[0066] As explained in the above, the recognition apparatus 12 according to the embodiment is able to recognize the contents that are indicated by the subject at a high precision while suppressing the amount of data for the images by executing the above-described processing. In addition, it is possible for the recognition apparatus 12 to suppress the data amount for the images and the pixel number for the sensor unit 112, and therefore, it is also possible to reduce the cost of the hardware that is necessary to transmit the images, and the cost of the sensor unit 112.

[0067] Note that in the embodiment that has been described above, an example has been explained of a case in which the image capturing apparatus 11 is mounted on an automobile, which is one example of a movable apparatus. However, the present invention is not limited thereto. For example, the image capturing apparatus 11 may also be a camera that is mounted in an arbitrary position near a path such as a road or the like on which movable apparatuses move.

[0068] In addition, in the above-described embodiment, an example has been explained in which each function of the recognition apparatus 12 is realized by reading out and expanding a program that is stored on the ROM onto the RAM, an executing each of these programs. However, the present invention is not limited thereto.

[0069] Each function of the recognition apparatus 12 may also be realized by using, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) instead of the ROM, CPU, and RAM.

[0070] In addition, at least one of the processing that is executed by the determining unit 124, the processing that is executed by the image processing unit 127, and the processing that is executed by the control unit 126 may also be executed such that the data amount for the first movie image and the data amount for the third movie image match.

[0071] In addition, at least one of the processing executed by the determining unit 124 or the processing executed by the image processing unit 127 may also be executed such that the data amount for the first image P1 matches the data amount for a third image.

[0072] In addition, the definition of matching in this context includes not only cases in which both are perfect matches, but also cases in which both match within a range that has been set in advance. In addition, the range that has been set in advance may be an arbitrarily set range, or it may also be set based on measurement errors for the data amounts, or the like.

[0073] In addition, although an example has been explained in the above-described embodiment for a case in which the image processing unit 121 and the image processing unit 127 are separate, the present invention is not limited thereto. That is, the image processing unit 121 and the image processing unit 127 may also be included in one unit.

[0074] In addition, although an example has been explained in the above-described embodiment for a case in which the control unit 125 and the control unit 126 are separate, the present invention is not limited thereto. That is, the control unit 125 and the control unit 126 may also be included in one unit.

[0075] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation to encompass all such modifications and equivalent structures and functions.

[0076] In addition, as a part or the whole of the control according to the embodiments, a computer program realizing the function of the embodiments described above may be supplied to the recognition apparatus through a network or various storage media. Then, a computer (or a CPU, an MPU, or the like) of the recognition apparatus may be configured to read out and execute the program. In such a case, the program and the storage medium storing the program configure the present invention.

[0077] In addition, the present invention may include, for example, at least one processor or circuit configured to function as the embodiment explained above.

[0078] This application claims the benefit of Japanese Patent Application No. 2022-166444, filed on Oct. 17, 2022, which is hereby incorporated by reference herein in its entirety.