Information processing apparatus, information processing method, program, and imaging system

[0001] This application is a National Stage Patent Application of PCT International Patent Application No. PCT/JP2022/000043 (filed on Jan. 4, 2022) under 35 U.S.C. § 371, which claims priority to Japanese Patent Application No. 2021-037179 (filed on Mar. 9, 2021), which are all hereby incorporated by reference in their entirety.

[0002] This technology relates to an information processing apparatus, an information processing method, a program, and an imaging system, and expands a region where event information can be acquired.

[0003] Conventionally, Non-Patent Document 1 discloses an asynchronous imaging device called a dynamic vision sensor (DVS) as one of event-driven imaging devices. This asynchronous imaging device detects, as an event, that a luminance change of a pixel that photoelectrically converts incident light exceeds a predetermined threshold value, and functions as an event detection device. Such an asynchronous imaging device (event detection device) has an advantage that low power consumption, data saving amount, low latency, high frame rate imaging, and high dynamic range imaging can be performed.

[0004] Furthermore, as described in Non-Patent Document 2, it is possible to estimate a luminance image by temporally integrating event information. Furthermore, as described in Patent Document 1, there is also proposed a technique in which an event detection device and a normal imaging device (for example, an imaging device that performs imaging at a predetermined frame rate and generates a captured image for each frame) are provided, and when an event is detected by the event detection device, imaging is performed by the normal imaging device, thereby taking advantage of low power consumption and a data saving amount.

• [0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2020-161992

• [0006] Non-Patent Document 1: Gallego, Guillermo, Tobi Delbruck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, et al. 2019. “Event-Based Vision: A Survey,” April. http://arxiv.org/abs/1904.08405.
• [0007] Non-Patent Document 2: Rebecq, Henri, Rene' Ren', Rene' Ranftl, Vladlen Koltun, and Davide Scaramuzza. 2019. “High Speed and High Dynamic Range Video with an Event Camera MULTIMEDIA MATERIAL.”. http://rpg.ifi.uzh.ch/e2vid.

[0008] By the way, since an event detection device detects a positive event that a pixel value has changed to be larger than a reference value and a negative event that the pixel value has changed to be smaller than the reference value, event information cannot be acquired for a region where a change in the pixel value is small.

[0009] Furthermore, in a case where luminance information is estimated by temporally integrating event information, the luminance information cannot accurately be estimated from the event information without background information. For example, in a case where the event detection device is fixedly installed like a monitoring camera, event information can be acquired for a region of a moving subject, but event information cannot be acquired for a stationary background region. Furthermore, since luminance estimated using the event information of the region indicating the moving subject indicates a relative difference with respect to color of the background region (background color), there is a case where the luminance of the subject is different from actual luminance under the influence of the background color.

[0010] Moreover, in a case where the event detection device and the normal imaging device are used, downsizing and cost reduction are difficult.

[0011] Therefore, an object of this technology is to provide an information processing apparatus, an information processing method, a program, and an imaging system capable of expanding a region where event information can be acquired.

[0012] A first aspect of this technology is

• • [0013] an information processing apparatus including:
  • [0014] a determination unit that determines whether no event is detected on the basis of event detection information indicating a detection result of an event generated by an event detection unit, the event detection unit detecting, as the event, that a luminance change of a pixel in an imaging unit that photoelectrically converts an optical image indicating a subject exceeds a preset threshold value; and
  • [0015] a movement control unit that moves a position of the optical image indicating the subject in the imaging unit in a case where the determination unit determines that no event is detected.

[0016] In this technology, the determination unit determines whether no event is detected on the basis of event detection information indicating a detection result of an event generated by an event detection unit, the event detection unit detecting, as the event, that a luminance change of a pixel in an imaging unit that performs photoelectric conversion of an optical image indicating a subject exceeds a preset luminance threshold value. For example, in a case where there is an event non-detection region in which the number of events indicated by the event detection information is smaller than the preset threshold value, in a case where there is a direction in which an edge is not detected, in a case where a motion of the event is determined and there is a direction in which there is no motion, in a case where capacity shortage in a communication path occurs in transmission of the event detection information, and the like, the determination unit determines that no event is detected. Furthermore, the determination unit may divide a region of the optical image, determine whether each divided region is the event non-detection region, and determine that the region is the event non-detection region in a case where the region is the event non-detection region.

[0017] In a case where it is determined that no event is detected, the movement control unit moves the position of the optical image indicating the subject in the imaging unit so that an event can be detected in a region where no event is detected. The event detection unit includes the imaging unit and an imaging optical system that forms the optical image indicating the subject on an imaging surface of the imaging unit, and the movement control unit moves the position of the optical image indicating the subject by moving a position of an optical axis of the imaging optical system in the imaging unit. The movement control unit may move the imaging unit in a direction orthogonal to the optical axis of the imaging optical system, or may move the optical axis of the imaging optical system with respect to the imaging unit. Moreover, the movement control unit may control movement of a mobile body including the event detection unit. Furthermore, in a case where it is determined that no event is detected in a case where the capacity shortage in the communication path occurs in the transmission of the event detection information, the movement control unit moves the imaging unit in a direction in which the detected events decrease.

[0018] Furthermore, in this technology, an event information generation unit that generates event information including the event detection information generated by the event detection unit and movement information regarding movement of the position of the optical image indicating the subject is further included. The event information generation unit uses information indicating the movement of the imaging unit as the movement information in a case where the imaging unit is moved in the direction orthogonal to the optical axis of the imaging optical system, and uses information indicating the movement of the optical axis as the movement information in a case where the optical axis of the imaging optical system is moved with respect to the imaging unit. Furthermore, the event information generation unit generates event information using motion information generated by a motion sensor that detects a motion of the mobile body including the event detection unit.

[0019] A second aspect of this technology is

• • [0020] an information processing method including:
  • [0021] determining, by a determination unit, whether no event is detected on the basis of event detection information indicating a detection result of an event generated by an event detection unit, the event detection unit detecting, as the event, that a luminance change of a pixel in an imaging unit that photoelectrically converts an optical image indicating a subject exceeds a preset threshold value; and
  • [0022] performing, by a movement control unit, control to move a position of the optical image indicating the subject in the imaging unit in a case where the determination unit determines that no event is detected.

[0023] A third aspect of this technology is

• • [0024] a program for causing a computer to execute control according to a detection result of an event generated by an event detection unit, the event detection unit detecting, as the event, that a luminance change of a pixel in an imaging unit that photoelectrically converts an optical image indicating a subject exceeds a preset threshold value,
  • [0025] the program for causing the computer to execute:
  • [0026] a procedure of determining whether no event is detected on the basis of the event detection information; and
  • [0027] a procedure of moving a position of the optical image indicating the subject in the imaging unit.

[0028] Note that the program of the present technology is, for example, a program that can be provided for a general-purpose computer capable of executing various program codes by a storage medium provided in a computer-readable form, or a communication medium, for example, a storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, or a communication medium such as a network. By providing such program in the computer-readable form, processing according to the program is implemented on the computer.

[0029] A fourth aspect of this technology is

• • [0030] an imaging system including:
  • [0031] a determination unit that determines whether no event is detected on the basis of event detection information indicating a detection result of an event generated by an event detection unit, the event detection unit detecting, as the event, that a luminance change of a pixel in an imaging unit that photoelectrically converts an optical image indicating a subject exceeds a preset threshold value;
  • [0032] a movement control unit that moves a position of the optical image indicating the subject in the imaging unit in a case where the determination unit determines that no event is detected;
  • [0033] an event information generation unit that generates event information including the event detection information generated by the event detection unit and movement information regarding movement of the position of the optical image indicating the subject; and
  • [0034] an image generation unit that generates an image indicating the subject on the basis of the event information generated by the event information generation unit.

[0048] Hereinafter, modes for carrying out the present technology will be described. Note that the description is given in the following order.

• • [0049] 1. First Embodiment
    • [0050] 1-1. Configuration of First Embodiment
    • [0051] 1-2. Operation of First Embodiment
  • [0052] 2. Second Embodiment
    • [0053] 2-1. Configuration of Second Embodiment
    • [0054] 2-2. Operation of Second Embodiment
  • [0055] 3. Third Embodiment
    • [0056] 3-1. Configuration of Third Embodiment
    • [0057] 3-2. Operation of Third Embodiment
  • [0058] 4. About Generation of Image
  • [0059] 5. Other Embodiments

[0060] FIG. 1 illustrates a configuration of a first embodiment of an imaging system using an information processing apparatus of the present technology. An imaging system 10-1 includes an event detection unit 20 and an information processing unit 40-1. The event detection unit 20 and the information processing unit 40-1 may be provided independently or integrally.

[0061] The event detection unit 20 includes an imaging optical system 21, an imaging unit 22, an event detection processing unit 23, and a movement processing unit 24.

[0062] The imaging optical system 21 is configured with use of a focus lens, a zoom lens, and the like. The imaging optical system 21 drives the focus lens, the zoom lens, and the like to form an optical image indicating a subject (a subject optical image) on an imaging surface of the imaging unit 22. Note that the imaging optical system 21 may be integrally fixed to the imaging unit 22, or may be detachably fixed to the imaging unit 22.

[0063] The imaging unit 22 performs photoelectric conversion of the subject optical image for each pixel and generates a pixel signal corresponding to luminance. For example, the imaging unit 22 is configured with use of an imaging sensor such as a complementary metal oxide semiconductor (CMOS) and the like. Although not illustrated, the imaging unit 22 is configured with use of a pixel array unit and a pixel array drive unit. A plurality of pixels is two-dimensionally arranged in a matrix on the pixel array unit. Each pixel performs photoelectric conversion to generate a pixel signal of a captured image indicating the subject optical image. The pixel array drive unit drives the pixel array unit to output the pixel signal generated for each pixel to the event detection processing unit 23.

[0064] The event detection processing unit 23 is provided integrally with the imaging unit 22, and detects presence or absence of an event on the basis of whether or not a change (luminance change) in which the pixel signal generated for each pixel by the imaging unit 22 exceeds a preset threshold value (luminance threshold value) has occurred. The event includes, for example, an on-event indicating that a change amount of the pixel signal exceeds an upper limit luminance threshold value and an off-event indicating that the change amount falls below a lower limit luminance threshold value. Event detection information includes an event detection result including a polarity of a light amount change, for example, one bit indicating a detection result of the on-event and one bit indicating a detection result of the off-event. Furthermore, the event detection information includes at least the detection result of the event and position information such as coordinates representing a position of a pixel in which the event has been detected and the like.

[0065] Furthermore, the event detection information implicitly includes time information indicating relative time at which the event has occurred as long as an output interval of the event detection information is maintained as it is at the time of occurrence of the event. However, when the time interval of the event detection information cannot be maintained in a state at the time of occurrence of the event due to the event detection information being recorded in a memory and the like, the time information implicitly included is lost. Therefore, the event detection processing unit 222 may include time information such as a time stamp and the like in the event detection information so as to be able to determine timing at which the event has occurred.

[0066] As a data format of the event detection information including the position information of the pixel in which the event has been detected, the time information indicating the time when the event has occurred, and the polarity information of the light amount change as the event, for example, a data format called an address event representation (AER) can be adopted. The event detection processing unit 23 outputs the generated event detection information to the information processing unit 40-1.

[0067] The movement processing unit 24 moves a position of the subject optical image on the imaging surface of the imaging unit 22 on the basis of a control signal supplied from a movement control unit 42a of the information processing unit 40-1 as described later. For example, as in a case where a shake correction lens is provided in the imaging optical system 21 to perform shake correction, the event detection unit 20 drives the shake correction lens to move the position of the subject optical image on the imaging surface of the imaging unit 22. Furthermore, as in a case where the imaging unit 22 is moved in a direction orthogonal to an optical axis of the imaging optical system 21 to perform shake correction, the event detection unit 20 may move the imaging unit 22 to move the position of the subject optical image.

[0068] The information processing unit 40-1 includes a determination unit 41a, the movement control unit 42a, and an event information generation unit 43.

[0069] The determination unit 41a determines whether no event is detected on the basis of the event detection information indicating the detection result of the event generated by the event detection unit 20, and outputs a determination result to the movement control unit 42a.

[0070] In a case where the determination result of the determination unit 41a indicates that no event is detected, the movement control unit 42a performs movement control to move the position of the subject optical image in the imaging unit 22, and outputs a control signal to the movement processing unit 24 of the event detection unit 20. Furthermore, the movement control unit 42a generates movement information regarding the movement of the position of the subject optical image and outputs the movement information to the event information generation unit 43.

[0071] The event information generation unit 43 generates event information including the event detection information generated by the event detection unit 20 and the movement information generated by the movement control unit 42a, and outputs the event information to an image generation unit 50, an external device, and the like.

[0072] The image generation unit 50 generates a subject image captured by the event detection unit 20 on the basis of the event information.

[0073] Next, an operation of the first embodiment will be described. In the imaging system 10-1, presence or absence of a region where no event is detected is determined on the basis of the event detection information generated by the event detection unit 20, and in a case where it is determined that there is a region where no event is detected, a position of a subject optical image in the imaging unit is moved, and event detection information for the region where no event is detected is acquired.

[0074] FIG. 2 is a flowchart showing an operation of the first embodiment. In step ST1, the imaging system 10-1 performs imaging by the event detection unit. The event detection unit 20 of the imaging system 10-1 performs photoelectric conversion of the subject optical image for each pixel to generate a pixel signal. Furthermore, the event detection unit 20 detects, as an event, that a luminance change of a pixel exceeds a preset luminance threshold value on the basis of the pixel signal. The event detection unit 20 generates event detection information indicating a detection result of the event and proceeds to step ST2.

[0075] FIG. 3 illustrates an operation of the event detection unit. (a) of FIG. 3 illustrates a captured image acquired by an image sensor 221, and (b) of FIG. 3 illustrates an event detection result. In (a) of FIG. 3, subjects OBa and OBb are moving, and backgrounds BG1 to BG4 are stationary. Furthermore, luminance of the background is higher in the order of the background BG1, a background BG2, a background BG3, and the background BG4. In this case, as illustrated in (b) of FIG. 3, in an edge portion including a component in a direction orthogonal to a motion direction of the subjects OBa and OBb, an on-event (a white portion) is detected in a pixel in which a luminance change amount exceeds an upper limit luminance threshold value, and an off-event (a black portion) is detected in a pixel in which the luminance change amount falls below a lower limit luminance threshold value. Furthermore, a region where the luminance change amount is small is a region where no event is detected (gray portion). Note that (c) of FIG. 3 will be described later.

[0076] Returning to FIG. 2, in step ST2, the imaging system 10-1 records the event detection information. The information processing unit 40-1 of the imaging system 10-1 records the event detection information generated in step ST1 in the event information generation unit 43, and proceeds to step ST3.

[0077] In step ST3, the imaging system 10-1 performs determination of an event non-detection state. In the information processing unit 40-1 of the imaging system 10-1, the determination unit 41a determines whether or not there is a region in which no event is detected on the basis of the event detection information generated in step ST1. In a case where there is a region where no event is detected, the determination unit 41a determines the event non-detection state and proceeds to step ST4, and in a case where the event non-detection state is not determined, the determination unit proceeds to step ST8. The determination unit 41a determines that no event is detected in a case where there is an event non-detection region in which the number of events indicated by the event detection information is smaller than a preset threshold value (non-detection region determination threshold value). For example, in a case where the captured image is (a) of FIG. 3, since the backgrounds BG1 to BG4 are stationary, no event is detected in the backgrounds BG1 to BG4 as illustrated in (b) of FIG. 3. Thus, the number of events is smaller than the non-detection region determination threshold value. Therefore, the determination unit 41a determines the event non-detection state.

[0078] Furthermore, when the movement control unit 42a moves the event detection unit 20 in a horizontal direction, the event detection unit 20 detects an event in a vertical edge portion, and when the event detection unit 20 is moved in a vertical direction, the event detection unit 20 detects an event in a horizontal edge portion. Therefore, the determination unit 41a may determine a direction of an edge in the optical image (captured image) of the subject, determine that no event is detected in a case where there is a direction in which the edge is not detected, and detect an event in an edge portion in the direction in which the edge is not detected.

[0079] In step ST4, the imaging system 10-1 performs movement control of the event detection unit. The information processing unit 40-1 of the imaging system 10-1 performs movement control on the event detection unit 20 so as to be able to acquire event detection information of a region where no event is detected, outputs a control signal to the event detection unit 20, moves a position of the subject optical image in the imaging unit 22 of the event detection unit 20, and proceeds to step ST5.

[0080] In step ST5, the imaging system 10-1 records movement information. The movement control unit 42a generates movement information (a movement amount, a movement direction, and the like) regarding the movement of the position of the optical image performed by the event detection unit 20 on the basis of the control signal. For example, in a case where the imaging unit 22 is moved in a direction orthogonal to the optical axis of the imaging optical system 21, the movement control unit 42a generates information indicating the movement amount and the movement direction of the imaging unit 22 as the movement information. Furthermore, in a case where the optical axis of the imaging optical system 21 is moved with respect to the imaging unit 22, the movement control unit 42a generates information indicating a drive amount and a drive direction of the shake correction lens of the imaging optical system 21 as the movement information. The movement control unit 42a causes the event information generation unit 43 to record the generated movement information and proceeds to step ST6.

[0081] In step ST6, the imaging system performs imaging by the event detection unit. The event detection unit 20 of the imaging system 10-1 performs photoelectric conversion of the subject optical image for each pixel to generate a pixel signal. Furthermore, the event detection unit 20 detects, as an event, that a luminance change of a pixel exceeds a preset luminance threshold value on the basis of the pixel signal. That is, the event detection unit 20 detects an event related to a stationary subject, generates event detection information indicating a detection result, and proceeds to step ST7.

[0082] FIG. 4 illustrates an operation in a case where movement control is performed in the event detection unit. (a) of FIG. 4 illustrates a captured image acquired by the imaging unit 22, and (b) of FIG. 4 illustrates an event detection result. When the movement control of the event detection unit is performed, a position of the background moves. Thus, it is possible to detect an event related to a stationary region, for example, an event at a boundary of the backgrounds BG1 to BG4. Note that (c) of FIG. 4 will be described later.

[0083] In step ST7, the imaging system 10-1 records the event detection information. The information processing unit 40-1 of the imaging system 10-1 records the event detection information generated in step ST6 in the event information generation unit 43, and proceeds to step ST8.

[0084] In step ST8, the imaging system 10-1 outputs event information. The information processing unit 40-1 of the imaging system 10-1 generates event information including the event detection information recorded in the event information generation unit 43 and the movement information generated by the movement control unit 42a, and outputs the event information to the image generation unit 50, the external device, and the like.

[0085] The event detection unit 20 may be an asynchronous type or a synchronous type. Furthermore, the event detection unit 20 may generate, for example, event detection information with a matrix index. The event information generation unit 43 of the information processing unit 40-1 may hold event data for each matrix index, or may hold the event detection information by dividing the matrix index for each region of a certain size. Moreover, in a case where the event detection unit 20 is the synchronous type, the event information generation unit 43 may hold the event detection information in a state of being added in units of frames, for example.

[0086] Determination of the region where no event is detected by the determination unit 41a may be performed by dividing a detection region into small regions and determining whether or not the number of events added for each small region is smaller than the threshold value (non-detection region determination threshold value). FIG. 5 is a diagram illustrating a case where a detection region is divided into small regions. In this case, the event detection unit 20 determines whether each divided region is a region where no event is detected. Note that a black frame portion is a region determined as an event detection region, and a white frame portion is a region determined as an event non-detection region.

[0087] Furthermore, the movement control unit 42a may perform movement control so that more boundaries can be detected. For example, as described above, when the event detection unit is driven in the horizontal direction, the vertical edge can be detected, and when the event detection unit is moved in the vertical direction, the horizontal edge can be detected. Therefore, the event detection unit is only required to be moved first in the horizontal direction, and then moved in the vertical direction. Furthermore, the boundary may be detected by moving in an oblique direction. Moreover, in the event detection unit 20, in a case where the movement processing unit 24 can move the position of the subject optical image in a unit smaller than one pixel unit (subpixel unit) on the basis of the control signal supplied from the movement control unit 42a, setting may be made such that the subject optical image is moved stepwise in subpixel units.

[0088] Furthermore, the event information generation unit 43 may generate event information indicating only an event non-detection region. For example, the event information generation unit 43 may generate event information indicating only an event non-detection region indicated by a white frame in FIG. 5. In this case, the information processing unit 40-1 can detect only an edge of the background, for example.

[0089] As described above, in the first embodiment, in a case where the event non-detection state is determined, the position of the subject optical image in the imaging unit 22 of the event detection unit 20 is moved, so that the event can also be detected for a stationary region where a motion has not occurred, for example, the background illustrated in FIG. 4. Therefore, according to the first embodiment, the information processing unit 40-1 can expand a region where the event information can be acquired.

[0090] Next, in a second embodiment, a case where an event detection unit is provided in a mobile body will be described.

[0091] FIG. 6 illustrates a configuration of the second embodiment. An imaging system 10-2 includes an event detection unit 20, a motion sensor 30, and an information processing unit 40-2. The event detection unit 20 and the information processing unit 40-2 may be provided independently or integrally. Furthermore, the motion sensor 30 is provided integrally with the event detection unit 20 so as to be able to detect a motion caused in the event detection unit 20 provided in the mobile body.

[0092] Similarly to the first embodiment, the event detection unit 20 includes an imaging optical system 21, an imaging unit 22, an event detection processing unit 23, and a movement processing unit 24.

[0093] The imaging optical system 21 includes a focus lens, a zoom lens, and the like, and forms a subject optical image on an imaging surface of the imaging unit 22.

[0094] The imaging unit 22 performs photoelectric conversion of the subject optical image for each pixel, generates a pixel signal according to luminance, and outputs the pixel signal to the event detection processing unit 23. The event detection processing unit 23 detects presence or absence of an event on the basis of the pixel signal output from the imaging unit 22 and outputs event detection information indicating a detection result to the information processing unit 40-2.

[0095] The movement processing unit 24 moves a position of the subject optical image on the imaging surface of the imaging unit 22 on the basis of a control signal supplied from a movement control unit 42b of the information processing unit 40-2 as described later.

[0096] The motion sensor 30 detects a motion of the motion event detection unit 20 caused in the event detection unit 20. The motion sensor 30 uses, for example, an inertial measurement unit (IMU) and the like, detects a motion (a position movement, an attitude change, and the like) of the event detection unit 20, and outputs motion information indicating a detection result to the information processing unit 40-2.

[0097] The information processing unit 40-2 includes a determination unit 41b, the movement control unit 42b, and an event information generation unit 43.

[0098] The determination unit 41b determines whether no event is detected on the basis of the event detection information generated by the event detection unit 20 and the motion information generated by the motion sensor 30. For example, the determination unit 41b determines a motion of an event on the basis of the event detection information and the motion information, and determines that no event is detected in a case where there is a direction in which there is no motion. The determination unit 41b outputs a determination result as to whether or not no event is detected to the movement control unit 42b.

[0099] In a case where the determination result of the determination unit 41b indicates that no event is detected, the movement control unit 42b performs movement control to move the position of the subject optical image in the imaging unit 22, and outputs a control signal to the movement processing unit 24 of the event detection unit 20. Furthermore, the movement control unit 42b performs movement control on the basis of the motion information supplied from the motion sensor 30. For example, in a case where the event detection unit 20 is provided in a mobile body (for example, a car) and moves only in a first direction (for example, a forward direction of the car), the movement control unit 42b determines that information in another direction different from the first direction has not been obtained, generates a control signal so that the information in the other direction can be acquired, and outputs the control signal to the movement processing unit 24 of the event detection unit 20. Furthermore, the movement control unit 42b may perform movement control of the mobile body provided with the event detection unit 20 so that the information in the other direction can be acquired. Furthermore, the movement control unit 42b generates movement information regarding the movement of the position of the subject optical image and outputs the movement information to the event information generation unit 43.

[0100] The event information generation unit 43 generates event information including the event detection information generated by the event detection unit 20, the motion information generated by the motion sensor 30, and the movement information generated by the movement control unit 42b, and outputs the event information to an image generation unit 50, an external device, and the like.

[0101] The image generation unit 50 generates a subject image captured by the event detection unit 20 on the basis of the event information.

[0102] Next, an operation of the second embodiment is described. The imaging system 10-2 determines presence or absence of a region in which no event is detected on the basis of the event detection information generated by the event detection unit 20 and the motion information generated by the motion sensor 30. In a case where it is determined that there is a region where no event is detected, the imaging system 10-2 moves a position of a subject optical image in the imaging unit on the basis of the motion of the event detection unit 20, and acquires event detection information regarding the region where no event is detected.

[0103] FIG. 7 is a flowchart illustrating an operation of the second embodiment. In step ST11, the imaging system 10-2 performs imaging by the event detection unit. The event detection unit 20 of the imaging system 10-2 performs photoelectric conversion of the subject optical image for each pixel to generate a pixel signal. Furthermore, the event detection unit 20 detects, as an event, that a luminance change of a pixel exceeds a preset luminance threshold value on the basis of the pixel signal. The event detection unit 20 generates event detection information indicating a detection result of the event and proceeds to step ST12.

[0104] In step ST12, the imaging system 10-2 records motion information and the event detection information. The information processing unit 40-2 of the imaging system 10-2 records the motion information generated by the motion sensor 30 and the event detection information generated in step ST11 in the event information generation unit 43, and proceeds to step ST13.

[0105] In step ST13, the imaging system 10-2 performs determination of an event non-detection state. In the information processing unit 40-2 of the imaging system 10-2, the determination unit 41b determines whether or not there is a region in which no event is detected on the basis of the motion information supplied from the motion sensor 30 and the event detection information generated in step ST11. For example, at the time of moving in a motion direction indicated by the motion information generated by the motion sensor 30, since no event is detected in an edge portion extending in the motion direction, the determination unit 41b determines that there is a region where no event is detected. In a case where there is a region where no event is detected, the determination unit 41b determines the event non-detection state and proceeds to step ST14, and in a case where the event non-detection state is not determined, the determination unit proceeds to step ST18.

[0106] In step ST14, the imaging system 10-2 performs movement control of the event detection unit 20. In the information processing unit 40-2 of the imaging system 10-2, the movement control unit 42b performs movement control of the event detection unit 20 so that the event detection information of the region where no event is detected can be acquired. For example, on the basis of the motion information supplied from the motion sensor 30, the movement control unit 42b generates a control signal so that the event detection unit 20 can acquire information in another direction different from a movement direction of the event detection unit 20. The movement control unit 42b outputs the generated control signal to the movement processing unit 24 of the event detection unit 20, moves a position of the subject optical image in the imaging unit 22, and proceeds to step ST15.

[0107] In step ST15, the imaging system 10-2 records movement information. The movement control unit 42b generates movement information regarding the movement of the position of the optical image performed by the event detection unit 20 on the basis of the control signal, records the movement information in the event information generation unit 43, and proceeds to step ST16.

[0108] In step ST16, the imaging system performs imaging by the event detection unit. The event detection unit 20 of the imaging system 10-1 performs photoelectric conversion of the subject optical image for each pixel to generate a pixel signal. Furthermore, the event detection unit 20 detects, as an event, that a luminance change of a pixel exceeds a preset luminance threshold value on the basis of the pixel signal. The event detection unit 20 generates event detection information indicating a detection result of the event and proceeds to step ST17.

[0109] In step ST17, the imaging system 10-2 records the motion information and the event detection information. The information processing unit 40-2 of the imaging system 10-2 records the motion information supplied from the motion sensor 30 and the event detection information generated in step ST16 in the event information generation unit 43, and proceeds to step ST18.

[0110] In step ST18, the imaging system 10-2 outputs event information. The information processing unit 40-2 of the imaging system 10-2 generates event information including the event detection information, the movement information generated by the movement control unit 42b, and the motion information generated by the motion sensor 30, which are recorded in the event information generation unit 43, and outputs the event information to the image generation unit 50, the external device, and the like.

[0111] FIG. 8 is a diagram illustrating a case where the event detection unit is provided in a mobile body (for example, a car). (a) of FIG. 8 illustrates a captured image acquired by an image sensor 221 of the event detection unit 20 provided in the mobile body, and (b) of FIG. 8 illustrates an event detection result of a conventional method. When the car moves, a boundary extending radially from the center of the front has less movement accompanying the movement of the car, and boundaries in other directions have more movement than the boundary extending radially from the center of the front. Therefore, street trees RT are detected by the event detection, but it is difficult to detect boundaries RB and a white line WL of a road extending in a traveling direction.

[0112] However, according to the second embodiment, a region of the boundaries RB and the white line WL of the road extending in the traveling direction is determined as an event non-detection region. In the event detection unit 20, the position of the subject optical image is moved by the movement processing unit 24 on the basis of the control signal supplied from the movement control unit 42b. Therefore, as illustrated in (c) of FIG. 8, the boundaries RB and the white line WL of the road extending in the traveling direction can be detected as an event. Note that (d) and (e) of FIG. 8 will be described later.

[0113] According to such a second embodiment, not only effects similar to those of the first embodiment can be obtained, but also the event detection information can be acquired for the region where the event detection information cannot be obtained when the event detection unit 20 moves. For example, even in a case where event detection information indicating a vertical edge cannot be obtained because the event detection unit 20 moves with little movement in a horizontal direction, the position of the optical image is moved in the horizontal direction, so that the event detection information indicating the vertical edge can be acquired.

[0114] Moreover, by including the motion information generated by the motion sensor 30 in the event detection information, not only an occurrence status of the event but also accurate motion of the event detection unit 20 can be stored.

[0115] Meanwhile, as the number of events detected by the event detection unit increases, an information amount of event detection information increases. Therefore, in a case where event information including the event detection information is transmitted to an external device and the like, if the number of events increases and a capacity of a communication path becomes insufficient, the event information is lost.

[0116] Therefore, in a third embodiment, when the events detected by the event detection unit are in an excess state, the imaging unit is moved so that the number of events detected by the event detection unit is reduced, and an information amount of the event information does not exceed a communication capacity.

[0117] FIG. 9 illustrates a configuration of the third embodiment. An imaging system 10-3 includes an event detection unit 20, a motion sensor 30, and an information processing unit 40-3. The event detection unit 20 and the information processing unit 40-3 may be provided independently or integrally. Furthermore, the motion sensor 30 is provided integrally with the event detection unit 20 so as to be able to detect a motion the event detection unit 20.

[0118] Similarly to the first embodiment, the event detection unit 20 includes an imaging optical system 21, an imaging unit 22, an event detection processing unit 23, and a movement processing unit 24.

[0119] The imaging optical system 21 includes a focus lens, a zoom lens, and the like, and forms a subject optical image on an imaging surface of the imaging unit 22.

[0120] The imaging unit 22 performs photoelectric conversion of the subject optical image for each pixel, generates a pixel signal according to luminance, and outputs the pixel signal to the event detection processing unit 23. The event detection processing unit 23 detects presence or absence of an event on the basis of the pixel signal output from the imaging unit 22 and outputs event detection information indicating a detection result to the information processing unit 40-3.

[0121] The movement processing unit 24 moves a position of the subject optical image on the imaging surface of the imaging unit 22 on the basis of a control signal supplied from a movement control unit 42c of the information processing unit 40-3 as described later.

[0122] The motion sensor 30 detects a motion of the motion event detection unit 20 caused in the event detection unit 20. The motion sensor 30 uses, for example, an inertial measurement unit (IMU) and the like, detects motion (a position movement, an attitude change, and the like) of the event detection unit 20, and outputs motion information indicating a detection result to the information processing unit 40-3.

[0123] The information processing unit 40-3 includes a determination unit 41c, the movement control unit 42c, and an event information generation unit 43.

[0124] The determination unit 41c determines whether there is an event excess state. For example, on the basis of the motion information generated by the motion sensor 30 and the event detection information generated by the event detection unit 20, the determination unit 41c determines that there is an event excess state in a case where the number of events detected by the moving event detection unit 20 is larger than a preset threshold value (an excess state determination threshold value). The determination unit 41c outputs a determination result of the event excess state to the movement control unit 42c.

[0125] In a case where the determination result of the determination unit 41c indicates the event excess state, the movement control unit 42c performs movement control to move the position of the subject optical image in the imaging unit 22 in a direction in which the number of detected events decreases on the basis of the motion information supplied from the motion sensor 30 and the event detection information generated by the event detection unit 20, and outputs a control signal to the movement processing unit 24 of the event detection unit 20. Furthermore, the movement control unit 42c generates movement information regarding the movement of the position of the subject optical image and outputs the movement information to the event information generation unit 43.

[0126] The event information generation unit 43 generates event information including the event detection information generated by the event detection unit 20, the motion information generated by the motion sensor 30, and the movement information generated by the movement control unit 42c, and outputs the event information to an image generation unit 50, an external device, and the like.

[0127] The image generation unit 50 generates a subject image captured by the event detection unit 20 on the basis of the event information.

[0128] FIG. 10 is a flowchart illustrating an operation of the third embodiment. In step ST21, the imaging system 10-3 performs imaging by the event detection unit. The event detection unit 20 of the imaging system 10-3 performs photoelectric conversion of the subject optical image for each pixel to generate a pixel signal. Furthermore, the event detection unit 20 detects, as an event, that a luminance change of a pixel exceeds a preset luminance threshold value on the basis of the pixel signal. The event detection unit 20 generates event detection information indicating a detection result of the event and proceeds to step ST22.

[0129] In step ST22, the imaging system 10-3 records the motion information and the event detection information. The information processing unit 40-3 of the imaging system 10-3 records the motion information generated by the motion sensor 30 and the event detection information generated in step ST21 in the event information generation unit 43, and proceeds to step ST23.

[0130] In step ST23, the imaging system 10-3 performs determination of an event excess state. In the information processing unit 40-3 of the imaging system 10-3, on the basis of the motion information supplied from the motion sensor 30 and the event detection information generated in step ST21, the determination unit 41c determines whether there is an event excess state in which the number of events detected by the moving event detection unit 20 is larger than a preset event number threshold value. The determination unit 41c returns to step ST21 in a case where it is determined that there is no event excess state, and proceeds to step ST24 in a case where it is determined that there is an event excess state.

[0131] In step ST24, the imaging system 10-3 performs movement control of the event detection unit 20. In the information processing unit 40-3 of the imaging system 10-3, the movement control unit 42c performs movement control on the event detection unit 20 in order to reduce the number of detected events. For example, the movement control unit 42c generates a control signal to move a position of the subject optical image in the imaging unit 22 in a direction in which movement of a subject on the imaging surface of the imaging unit 22 caused by the movement of the event detection unit 20 is reduced. The movement control unit 42c outputs the generated control signal to the movement processing unit 24 of the event detection unit 20, causes the event detection unit 20 to move the position of the subject optical image, and proceeds to step ST25.

[0132] In step ST25, the imaging system 10-3 records movement information. The movement control unit 42c generates movement information regarding the movement of the position of the subject optical image in the imaging unit 22 on the basis of the control signal, records the movement information in the event information generation unit 43, and proceeds to step ST26.

[0133] In step ST26, the imaging system performs imaging by the event detection unit. The event detection unit 20 of the imaging system 10-3 performs photoelectric conversion of the subject optical image for each pixel to generate a pixel signal. Furthermore, the event detection unit 20 detects, as an event, that a luminance change of a pixel exceeds a preset luminance threshold value on the basis of the pixel signal. The event detection unit 20 generates event detection information indicating a detection result of the event and proceeds to step ST27.

[0134] In step ST27, the imaging system 10-3 records the motion information and the event detection information. The information processing unit 40-3 of the imaging system 10-3 records the motion information supplied from the motion sensor 30 and the event detection information generated in step ST26 in the event information generation unit 43, and proceeds to step ST28.

[0135] In step ST28, the imaging system 10-3 outputs event information. The information processing unit 40-3 of the imaging system 10-3 generates event information including the event detection information, the movement information generated by the movement control unit 42c, and the motion information generated by the motion sensor 30, which are recorded in the event information generation unit 43, and outputs the event information to the image generation unit 50, the external device, and the like.

[0136] FIG. 11 illustrates an operation in a case of reducing the number of detected events. (a) of FIG. 11 illustrates an example of an event detected in a case where the event detection unit 20 moves in a direction of arrow MA. In the third embodiment, in a case where there is an event excess state, the movement control unit 42c moves the position of the subject optical image in the imaging unit 22 so that the detected events decrease. For example, as illustrated in (b) of FIG. 11, the movement control unit 42c moves the position of the subject optical image in a direction of the arrow MB to reduce number of the detected events as compared with (a) of FIG. 11.

[0137] According to such a third embodiment, in a case where there are many detected events, it is possible to reduce the number of detected events by moving the position of the subject optical image in the imaging unit 22. Furthermore, since the number of events can be reduced, it is possible to prevent the event information from being lost due to an information amount of the event information exceeding a communication capacity.

[0138] Next, the image generation unit 50 that generates a captured image using the event information generated in the above-described embodiments will be described. The image generation unit 50 may be provided in the above-described imaging system 10-1 (10-2, 10-3), or may be provided separately from the imaging system 10-1 (10-2, 10-3). Furthermore, in a case where the image generation unit 50 is provided in the imaging system 10-1 (10-2, 10-3), the image generation unit 50 may be provided separately from the event detection unit 20 and the information processing unit 40-1 (40-2, 40-3), or may be provided integrally with the information processing unit 40-1 (40-2, 40-3).

[0139] The image generation unit 50 generates an image signal of an image (integrated image) indicating a difference in luminance between subjects captured by the event detection unit 20 on the basis of the event information, and outputs the image signal to a display device and the like.

[0140] FIG. 12 illustrates the number of times of occurrence of an event in a case where edges having different luminance steps are moved by one pixel.

[0141] A subject OB moves by one pixel in a period from time T1 to time T4. Note that (a) of FIG. 12 illustrates a case where a luminance difference between the subject OB and a background BG is large, and (b) of FIG. 12 illustrates a case where the luminance difference between the subject OB and the background BG is smaller than that in (a) of FIG. 12.

[0142] In a case where the luminance difference between the subject OB and the background BG is large, for example, luminance of a pixel PT changes as illustrated in (c) of FIG. 12, and three events occur in the period from time T1 to time T4. Furthermore, in a case where the luminance difference between the subject OB and the background BG is small, for example, the luminance of the pixel PT changes as illustrated in (d) of FIG. 12, and one event occurs in the period from time T1 to time T4. Therefore, it is possible to generate a luminance image on the basis of the event information using the number of times of occurrence of the event.

[0143] In a case where there is no motion information of the event detection unit 20, the image generation unit 50 generates a luminance image by using estimation of luminance using DNN described in Non-Patent Document 2, a method of obtaining an optical flow and estimating luminance, and the like. For example, in a case where the position of the subject optical image is not moved as in the conventional case, a difference in luminance between backgrounds cannot be reproduced as illustrated in (c) of FIG. 3. However, as in the present technology, by moving the position of the subject optical image in a case where it is determined to be in an event non-detection state, a luminance image indicating a difference in luminance between the backgrounds can be generated as illustrated in (c) of FIG. 4.

[0144] Furthermore, in a case where the event information includes the motion information, the image generation unit 50 calculates an edge luminance step by using the number of times of occurrence of the event and the luminance threshold value, and adds the edge luminance step to the motion direction of the event detection unit 20 to generate a luminance image.

[0145] FIG. 13 illustrates generation of a luminance image in a case where the event information includes the motion information of the event detection unit. (a) of FIG. 13 illustrates subjects imaged by the event detection unit 20. The event detection unit 20 images, for example, a subject OBa and a subject OBb having luminance higher than that of the subject OBa. (b) of FIG. 13 illustrates an event detected when the event detection unit 20 moves rightward by one pixel. As illustrated in (c) of FIG. 13, the image generation unit 50 calculates luminance information in line order from a first line Li to a last line LH.

[0146] (d) of FIG. 13 illustrates the number of occurrences EC of the event in a line Ln in (c) of FIG. 13, in which, for example, two on-events occur in a left edge portion of the subject OBa and, for example, two off-events occur in a right edge portion thereof. Furthermore, for example, four on-events occur in a left edge portion of the subject OBb having luminance higher than that of the subject OBa, and, for example, four off-events occur in a right edge portion thereof. Furthermore, (e) of FIG. 13 illustrates an event integration result ET. Note that the number of occurrences of on-event is a positive value, and the number of occurrences of off-event is a negative value.

[0147] The image generation unit 50 performs calculation shown in Expression (1) and generates images indicating luminance of the subjects OBa and OBb. Expression (1) represents an event integration result ET (x, y) at a position (x, y). In Expression (1), in a case where the number of pixels of one line is “W” and the number of lines is “H”, variables x and y are “x=1, 2, . . . W, y=1, 2, . . . H”. In Expression (1), “EC (x, y)” indicates the number of occurrences of an event occurring at the position (x, y), “TB” indicates a preset basic luminance level, and “MVP” indicates a motion amount (the number of pixels) of the subject optical image.
ET(x,y)=ET(x−1,y)+EC(x,y)×TB/MVP  (1)

[0148] The image generation unit 50 calculates an event integration result for each line from the first line Li to the last line LH on the basis of position information of the event detection information generated by the event detection unit 20, and generates an image indicating a difference in luminance between the subjects OBa and OBb as illustrated in (f) of FIG. 13.

[0149] Furthermore, in a case where the position of the subject optical image is not moved, the boundaries RA, RB and the white line WL of the road cannot be reproduced as illustrated in (d) of FIG. 8 described above. However, as in the present technology, in a case where an event non-detection state is determined, a luminance image in which the boundaries and the white line are reproduced can be generated as illustrated in (e) of FIG. 8 by moving the position of the subject optical image.

[0150] In this manner, the image generation unit 50 can estimate the luminance with high accuracy and at low cost by using the event detection information, the motion information, and the like included in the event information, and can generate an image indicating a difference in luminance between the subjects.

[0151] An embodiment of the present technology may be a configuration and operation in which the above-described embodiments are combined. Furthermore, in the imaging unit 22 in the event detection unit 20, color filters (for example, mosaic filters of red, green, and blue) may be provided on the imaging surface. In this case, event information for each color component can be generated. Furthermore, the image generation unit 50 can generate a color image indicating a difference in luminance for each color component of a subject by using the event information for each color component.

[0152] Furthermore, in a case where the event detection unit 20 receives an infrared ray, a millimeter wave, and the like for distance measurement as incident light, the information processing unit 40-1 (40-2, 40-3) can generate event information indicating not only occurrence of a change in distance to a subject in which a motion has occurred but also occurrence of a change in distance to a subject in which a motion has not occurred. Furthermore, in a case where the event detection unit 20 receives an infrared ray for measuring temperature as the incident light, the information processing unit 40-1 (40-2, 40-3) can generate event information indicating not only occurrence of a change in temperature of the subject in which the motion has occurred but also occurrence of a change in temperature of the subject in which the motion has not occurred, and the like. Furthermore, if motion information of the event detection unit is included in the event information, the image generation unit 50 can generate a distance image indicating a difference between the distance to the subject in which the motion has occurred and the distance to the subject in which the motion has not occurred, and a temperature display image indicating a difference in temperature.

[0153] A series of processing described in the specification may be executed by hardware, software, or a composite configuration of both. In a case of executing the processing by the software, a program recorded with a processing sequence is installed and executed in a memory incorporated in dedicated hardware in a computer. Alternatively, the program can be installed and executed on a general-purpose computer capable of executing various processing.

[0154] For example, the program can be recorded in advance in a hard disk, a solid state drive (SSD), or a read only memory (ROM) as a recording medium. Alternatively, the program may be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a compact disc read only memory (CD-ROM), a magneto optical (MO) disk, a digital versatile disc (DVD), a Blu-ray Disc (BD) (registered trademark), a magnetic disk, a semiconductor memory, and the like. Such a removable recording medium can be provided as so-called package software.

[0155] Furthermore, in addition to being installed on the computer from the removable recording medium, the program may be transferred to the computer in a wired or wireless manner from a download site via a network such as a wide area network (WAN) represented by cellular, a local area network (LAN), the Internet, and the like. In the computer, it is possible to receive the program transferred in this manner and to install the same on a recording medium such as a built-in hard disk and the like.

[0156] Note that the effects described in this specification are merely examples and are not limited, and additional effects that are not described may be present. Furthermore, the present technology should not be construed as being limited to the above-described embodiments of the technology. The embodiments of this technology disclose the present technology in the form of illustration, and it is obvious that those skilled in the art may modify or replace the embodiments without departing from the gist of the present technology. That is, in order to determine the gist of the present technology, claims should be taken into consideration.

[0157] Furthermore, the information processing apparatus of the present technology can also have the following configuration.

[0158] (1) An information processing apparatus including:

• • [0159] a determination unit that determines whether no event is detected on the basis of event detection information indicating a detection result of an event generated by an event detection unit, the event detection unit detecting, as the event, that a luminance change of a pixel in an imaging unit that photoelectrically converts an optical image indicating a subject exceeds a preset threshold value; and
  • [0160] a movement control unit that moves a position of the optical image indicating the subject in the imaging unit in a case where the determination unit determines that no event is detected.

[0161] (2) The information processing apparatus according to (1), in which the determination unit determines that no event is detected in a case where there is an event non-detection region in which the number of events indicated by the event detection information is smaller than a preset threshold value.

[0162] (3) The information processing apparatus according to (2), in which the determination unit divides a region of the optical image and determines whether each divided region is the event non-detection region.

[0163] (4) The information processing apparatus according to any one of (1) to (3), in which the determination unit determines a direction of an edge in the optical image on the basis of the event detection information, and determines that no event is detected in a case where there is a direction in which the edge is not detected.

[0164] (5) The information processing apparatus according to any one of (1) to (4), in which the determination unit determines a motion of the event, and determines that no event is detected in a case where a there is a direction in which there is no motion.

[0165] (6) The information processing apparatus according to any one of (1) to (5), in which the determination unit determines that no event is detected in a case where capacity shortage in a communication path occurs in transmission of the event detection information.

[0166] (7) The information processing apparatus according to (6), in which in a case where the capacity shortage occurs, the movement control unit moves the imaging unit in a direction in which the detected events decrease.

[0167] (8) The information processing apparatus according to any one of (1) to (7), in which

• • [0168] the event detection unit includes the imaging unit and an imaging optical system that forms the optical image indicating the subject on an imaging surface of the imaging unit, and
  • [0169] the movement control unit moves the position of the optical image indicating the subject by moving a position of an optical axis of the imaging optical system in the imaging unit.

[0170] (9) The information processing apparatus according to (8), further including an event information generation unit that generates event information including the event detection information generated by the event detection unit and movement information regarding movement of the position of the optical image indicating the subject.

[0171] (10) The information processing apparatus according to (9), in which the movement control unit moves the imaging unit in a direction orthogonal to the optical axis of the imaging optical system.

[0172] (11) The information processing apparatus according to (10), in which the event information generation unit uses information indicating movement of the imaging unit as the movement information.

[0173] (12) The information processing apparatus according to (9), in which the movement control unit moves the optical axis of the imaging optical system with respect to the imaging unit.

[0174] (13) The information processing apparatus according to (12), in which the event information generation unit uses information indicating movement of the optical axis as the movement information.

[0175] (14) The information processing apparatus according to (9), in which the event information generation unit includes, in the event information, motion information generated by a motion sensor that detects a motion of a mobile body including the imaging unit.

• [0176] 10-1, 10-2, 10-3 Imaging system
• [0177] 20 Event detection unit
• [0178] 21 Imaging optical system
• [0179] 22 Imaging unit
• [0180] 23 Event detection processing unit
• [0181] 24 Movement processing unit
• [0182] 30 Motion sensor
• [0183] 40-1, 40-2, 40-3 Information processing unit
• [0184] 41a, 41b, 41c Determination unit
• [0185] 42a, 42b, 42c Movement control unit
• [0186] 43 Event information generation unit
• [0187] 50 Image generation unit
• [0188] 221 Image sensor
• [0189] 222 Event detection processing unit