Adaptive image sensor systems and methods

[0001] The present invention relates to imaging devices, and, more particularly, to adaptive image sensor systems and methods.

[0002] Modern electronic devices such as cellular telephones, cameras, and computers often use digital image sensors. Imagers (i.e., image sensors) often include a two-dimensional array of image sensing pixels. Each pixel typically includes a photosensor such as a photodiode that receives incident photons (light) and converts the photons into electrical signals.

[0003] In conventional imaging systems, a digital image sensor captures images of a scene at a fixed frame rate. However, it is often the case that many of the captured frames are unnecessary. Consider, as an example, an imaging system being used as a security camera in a low-traffic area. Many images captured by the imaging system will be identical (and therefore not useful) as movement in the low-traffic area is rare. However, the imaging system will still generally need to capture images at a high frame rate, to be able to adequately capture video of any events (such as a security breach) that do occur. While conventional imaging systems often reduce the frame rate of captured video in post-processing, in order to reduce the amount of storage space needed to archive the video; having the digital image sensor continually capture images at the fixed frame rate increases power consumption and results in other limitations on image sensor performance and cost.

[0004] It would therefore be desirable to be able to provide improved image sensor systems.

[0013] Digital camera modules are widely used in electronic devices. An electronic device with a digital camera module is shown in FIG. 1. Electronic device 10 may be a digital camera, a laptop computer, a display, a computer, a cellular telephone, or other electronic device. Device 10 may include one or more imaging systems such as imaging systems 12A and 12B (e.g., camera modules 12A and 12B) each of which may include one or more image sensors 14 and corresponding lenses. During operation, a lens focuses light onto an image sensor 14. The lens may have fixed aperture. The pixels in image sensor 14 include photosensitive elements that convert the light into digital data. Image sensors may have any number of pixels (e.g., hundreds or thousands or more). A typical image sensor may, for example, have millions of pixels (e.g., megapixels). In high-end equipment, sensors with 10 megapixels or more are not uncommon. In at least some arrangements, device 10 may include two (or more) image sensors 14, which may capture images from different perspectives. When device 10 includes two image sensors 14, device 14 may be able to capture stereo images.

[0014] Still and video image data from camera sensor 14 may be provided to image processing and data formatting circuitry 16 via path 26. Image processing and data formatting circuitry 16 may be used to perform image processing functions such as adjusting white balance and exposure and implementing video image stabilization, image cropping, image scaling, etc. Image processing and data formatting circuitry 16 may also be used to compress raw camera image files if desired (e.g., to Joint Photographic Experts Group or JPEG format).

[0015] In some arrangements, which is sometimes referred to as a system on chip or SOC arrangement, camera sensor 14 and image processing and data formatting circuitry 16 are implemented as a common unit 15 (e.g., on a common integrated circuit, or stacked together). The use of a single integrated circuit to implement camera sensor 14 and image processing and data formatting circuitry 16 can help to minimize costs. If desired, however, multiple integrated circuits may be used to implement circuitry 15. In arrangements in which device 10 includes multiple camera sensors 14, each camera sensor 14 and associated image processing and data formatting circuitry 16 can be formed on a separate SOC integrated circuit (e.g., there may be multiple camera system on chip modules such as modules 12A and 12B).

[0016] To save space and simplify assembly by a manufacturer of a camera or other electronic device in which imaging capabilities are desired, it may be desirable to stack a processor (e.g., processing circuitry 16, or a processor in camera sensor 14) and an imager (e.g., camera sensor 14) to form a preassembled image sensor unit 15. An image sensor unit 15 that is formed in this way has the image sensing capabilities of the imager and the processing capabilities of the processor in a single component. Additionally, in arrangements in which the processor 16 (sometimes referred to herein as an image co-processor iCP) has a smaller area than the imager 14, forming the processor 16 on a separate wafer may decrease the cost of the processor 16, by allowing more processors 16 to be formed on each wafer. Moreover, processors 16 and imagers 14 may be formed on different types of wafers, each of which may be optimized for their respective use. As an example, a wafer of image co-processors 16 may be optimized for analog circuitry (to optimize the performance of analog circuitry in the image co-processors such as sample and hold circuitry, line buffers, and analog-to-digital converters) or digital circuitry (to optimize the performance of digital circuitry in the image co-processors such as image formatting circuitry). Meanwhile, the imager wafer may be optimized for optical performance. Optimization of wafers for analog, digital, and/or optical performance may involve altering the materials and methods used in forming completed wafers and the design and arrangement of circuits in the wafers, as examples.

[0017] Circuitry 15 conveys data to host subsystem 20 over path 18. Circuitry 15 may provide acquired image data such as captured video and still digital images to host subsystem 20.

[0018] Electronic device 10 typically provides a user with numerous high level functions. In a computer or advanced cellular telephone, for example, a user may be provided with the ability to run user applications. To implement these functions, electronic device 10 may have input-output devices 22 such as projectors, keypads, input-output ports, and displays and storage and processing circuitry 24. Storage and processing circuitry 24 may include volatile and nonvolatile memory (e.g., random-access memory, flash memory, hard drives, solid state drives, etc.). Storage and processing circuitry 24 may also include processors such as microprocessors, microcontrollers, digital signal processors, application specific integrated circuits, etc.

[0019] Device 10 may include position sensing circuitry 23. Position sensing circuitry 23 may include, as examples, global positioning system (GPS) circuitry, radio-frequency-based positioning circuitry (e.g., cellular-telephone positioning circuitry), gyroscopes, accelerometers, compasses, magnetometers, etc.

[0020] An example of an arrangement for sensor array 14 is shown in FIG. 2. As shown in FIG. 2, device 10 may include an array 14 of pixels 28 coupled to image readout circuitry 30 and address generator circuitry 32. As an example, each of the pixels in a row of array 14 may be coupled to address generator circuitry 32 by one or more conductive lines 34. Array 14 may have any number of rows and columns. In general, the size of array 14 and the number of rows and columns in array 14 will depend on the particular implementation. While rows and columns are generally described herein as being horizontal and vertical rows and columns may refer to any grid-like structure (e.g., features described herein as rows may be arranged vertically and features described herein as columns may be arranged horizontally).

[0021] Address generator circuitry 32 may generate signals on paths 34 as desired. For example, address generator circuitry 32 may generate reset signals on reset lines in paths 34, transfer signals on transfer lines in paths 34, and row select (e.g., row readout) signals on row select lines in paths 34 to control the operation of array 14. If desired, address generator circuitry 32 and array 14 may be integrated together in a single integrated circuit (as an example).

[0022] Image readout circuitry 30 may include circuitry 42 and image processing and data formatting circuitry 16. Circuitry 42 may include sample and hold circuitry, analog-to-digital converter circuitry, and line buffer circuitry (as examples). As one example, circuitry 42 may be used to measure signals in pixels 28 and may be used to buffer the signals while analog-to-digital converters in circuitry 42 convert the signals to digital signals. In a typical arrangement, circuitry 42 reads signals from rows of pixels 28 one row at a time over lines 40. With another suitable arrangement, circuitry 42 reads signals from groups of pixels 28 (e.g., groups formed from pixels located in multiple rows and columns of array 14) one group at a time over lines 40. The digital signals read out by circuitry 42 may be representative of charges accumulated by pixels 28 in response to incident light. The digital signals produced by the analog-to-digital converters of circuitry 42 may be conveyed to image processing and data formatting circuitry 16 and then to host subsystem 20 (FIG. 1) over path 18.

[0023] If desired, some or all of camera sensor 14 may be used to detect changes in a scene being observed by camera sensor 14. Control circuitry in camera module 12 (or host subsystem 20) may then adapt the operation of the camera sensor 14 in response to the detected changes. As an example, the camera sensor 14 may detect an object moving at a speed across the scene above a given threshold and, in response, the control circuitry may increase the frame rate of the camera sensor 14 in order to adequately record the moving object. As another example, a relatively-low power secondary imaging sensor may be used to determine when a pre-set triggering condition is encountered and, in response, the control circuitry may activate a primary imaging sensor.

[0024] As shown in the examples of FIGS. 3 and 4, camera sensor 14 may include a primary imaging sensor 14A and a secondary imaging sensor 14B. The secondary imaging sensor 14B may have lower power requirements than the primary imaging sensor 14A. Accordingly, the secondary imaging sensor 14B may have generally lower performance capabilities than the primary imaging sensor 14A. As examples, the secondary imaging sensor 14B may be a monochrome imaging sensor and may have a lower imaging resolution than the primary imaging sensor 14A. If desired, the secondary imaging sensor 14B may also be optimized for high speed operations (e.g., high frame rate operations) so that the operations of the primary imaging system 14A can be rapidly adjusted in response to changing conditions and in response to the detection of triggering conditions.

[0025] FIG. 3 illustrates an embodiment in which the secondary imaging sensor 14B is formed from a subset of the imaging pixels that form primary imaging sensor 14A. While FIG. 3 illustrates the secondary imaging sensor 14B as being formed in a corner of sensor 14A, this is merely one potential arrangement. As other examples, the secondary imaging sensor 14B may be formed from pixels that are dispersed (randomly, uniformly, or with some other particular order) throughout the primary imaging sensor 14A. In one example, imaging sensor 14 may be formed from an array of pixels including a repeating block of red, clear, blue, and green, where the clear pixels form secondary imaging sensor 14B and the red, blue, and green pixels form the primary imaging sensor 14A.

[0026] FIG. 4 illustrates another embodiment in which the secondary imaging sensor 14B is formed from a separate imaging sensor than the primary imaging sensor 14A. As illustrated in FIG. 4, the secondary imaging sensor 14B may generally have a smaller footprint and/or smaller resolution than the primary imaging sensor 14A, although alternative arrangements may be implemented. If desired, primary and secondary imaging sensors 14A and 14B may be formed on separate integrated circuit chips or may be integrated together on a single integrated circuit chip.

[0027] A flowchart of illustrative steps involved in adaptively adjusting image sensor operating conditions such as a frame rate is shown in FIG. 5. As shown by step 50 in FIG. 5, camera module 12 may detect one or more changes in a scene being imaged by the camera module 12. The changes may include changes in the amount and velocity of movement in the scene. The changes may be detected from a primary imaging sensor such as sensor 14A, by a secondary imaging sensor such as sensor 14B, or by a combination of sensors 14A and 14B. As an example, whenever primary sensor 14A is in a low power standby mode, the secondary sensor 14B may look for changes in the scene in step 50. Further, when the primary sensor 14A is capturing images of the scene, the secondary sensor 14B may either continue to look for changes in the scene or may itself be placed into a low power standby mode, with the operations of step 50 being taken over by the primary sensor 14A. As part of step 50, image processing circuitry 16 and/or processing circuitry 24 in host subsystem may analyze image data from a monitoring image sensor, such as secondary sensor 14B, in order to identify the changes in the scene.

[0028] With some suitable arrangements, step 50 may involve the removal of any camera motion that would otherwise alter the observations of motion in the scene. As an example, if the monitoring image sensor is being held in a user's hand, the monitoring image sensor will generally experience at least some “hand shake.” This type of unwanted camera motion can, if desired, be detected and compensated for using any desired image stabilization technique, including image analysis, gyroscopic measurements, accelerometer measurements, etc.

[0029] In step 52, image sensor operating conditions may be adjusted in response to the changes detected in step 50. As an example, if the monitoring image sensor detected an increase in the amount of movement in the scene in step 50, step 52 might involve directing the primary imaging sensor 14A to increase the frame rate at which it captures images, such that the motion is captured without motion blur. In other words, the new framerate of the primary imaging sensor 14A may be a function of the magnitude of the motion detected in step 50. Similarly, if the monitoring image sensor detected a decrease in the amount of movement in the scene, step 52 might involve reducing the frame rate of the primary imaging sensor 14A. Increases in the frame rate may generally be performed immediately after detection of an increase in movement. Decreases in the frame rate may also be performed immediately after detection of a decrease in movement or, if desired, after some predetermined delay.

[0030] Other image sensor operating conditions of the primary imaging sensor 14A that may be adjusted based on information from the secondary imaging sensor 14B in step 52 include, as examples, integration time, gain settings, white-balance settings, auto-focus settings, etc. The changes in the operating conditions of the primary imaging sensor 14A may be made based on various changes detected in step 50. As examples, the changes detected in step 50 that result in adjustments in step 52 may include movement of identified interest points and/or objects in the scene, changes in the color of a scene, changes in the light intensity of the scene, etc. Interest points and/or objects may be defined dynamically, e.g., may be context sensitive. As an example, any movement observed by a security camera may be defined by a user to be relevant, whereas a traffic camera may be programmed to ignore movement outside of road and sidewalk space. Detailed explanations of the techniques for identifying interest points and areas of interest have been omitted for brevity.

[0031] With one suitable arrangement, step 52 may involve directing the primary imaging sensor 14A to record only a fraction of the scene being imaged. As an example, if the secondary imaging sensor 14B identifies an object of interested in the bottom half of the viewing range of the primary imaging sensor 14A in step 50, the primary sensor 14A may be directed to record only pixels that image the bottom half of the viewing range.

[0032] If desired, step 52 might involve reducing the frame rate of the primary imaging sensor 14A to zero, as long as certain conditions are met. For example, if the monitoring imaging sensor 14B detects that there is no movement in the scene, that any movement is not of interest (e.g., is outside of an area of interest, is not associated with an object identified as being of interest, is merely associated with movement of the camera system, etc.), and/or that any movement is below a predetermined threshold, the primary imaging sensor 14A may be powered down and placed into a low power standby mode. Once there is sufficient movement, the primary imaging sensor 14A may be restarted at a non-zero frame rate.

[0033] In step 54, the primary imaging sensor 14A may capture images at the new frame rate, determined in step 52. If the new frame rate is set to zero, step 54 may involve placing the primary imaging sensor 14A into a low power standby mode while the secondary imaging sensor 14B continues to look for changes in the scene (in step 50).

[0034] The operations of FIG. 5 may enable a real-time or near-real-time optimization of the amount and quality of data collected by the primary image sensor 14A, balanced with an optimization of the power consumption of the overall device 10, for a given set of imaging objectives. As an example, when device 10 is a security camera, the operations of FIG. 5 may generally reduce the amount of data coming from the camera module, by not capturing frames that do not contain interesting information; while increasing the quality of data, by capturing interesting moments with a high frame rate (e.g., by reducing motion blur); and while decreasing the power consumption of device 10, by allowing the primary imaging sensor 14A to be placed into a low power standby mode when not needed. Additionally and since the long-term bandwidth of images from the primary imaging sensor 14A may generally be lower, device 10 may be designed with smaller and cheaper storage buffers and communications links, relative to conventional fixed-frame rate cameras.

[0035] If desired, a monitoring imaging sensor such as secondary imaging sensor 14B may be used in triggering primary imaging sensor 14A. An example of an arrangement of this type is shown in FIG. 6.

[0036] In step 60, device 10 may obtain trigger criteria (e.g., criteria indicating the conditions that result in the primary imaging sensor 14A beginning image capture operations). The trigger criteria may be preprogrammed into device 10, may be obtained from a user of device 10, may be determined dynamically by device 10, and may be obtained by any other suitable means. As examples, the trigger criteria may be the detection of an interest point or object in the scene (e.g., a person's face, a person, a baseball, an airplane, wildlife, or any other object), the detection of a particular individual or object in the scene (e.g., a particular person's face or body, a particular species of animal, etc.), the detection of motion above a given threshold in the scene, the detection of motion in a predefined or user-defined region of the scene, the detection of a change in the color of the scene, the detection of a change in the light intensity of the scene, the detection of a change in the light intensity of an interest point, object, person, etc. If desired, the detection criteria may be context dependent. As an example, device 10 could be configured to trigger primary imaging sensor 14A at the start of a flower's blooming process. The trigger criteria may, in such an example, be based on movement or color change patterns (e.g., early-stage blooming movement) of the flower being imaged. The trigger criteria may, if desired, by combined in any desired manner.

[0037] In step 62, the secondary imaging sensor 14B may monitor the scene (e.g., the region being imaged by device 10) for the trigger criteria specified in step 60. The secondary imaging sensor 14B may continually monitor the scene for the trigger criteria. As an alternative and whenever the secondary imaging sensor 14B does not detect the trigger criteria, the secondary imaging sensor 14B may be shut down to conserve power. As illustrated by step 64, the secondary imaging sensor 14B may be restarted after an optional delay and the scene checked again for the presence of the trigger criteria in step 62.

[0038] When the trigger criteria are detected in step 62, the primary imaging sensor 14A may be triggered in step 66. If desired, information on the desired operating conditions may also be provided to the primary imaging sensor 14A in step 66. As an example, the secondary imaging sensor 14B may detect information concerning the level of light in the scene (auto-exposure), the color balance of light in the scene (auto-white balance), and focus information in the scene (auto-focus) and may provide this information to the primary imaging sensor 14A in step 66. By providing this type of information, the primary imaging sensor 14A may be able to begin image capture operations in an expedited manner.

[0039] The framerate and the resolution of the secondary imaging sensor 14B used in monitoring the scene for trigger criteria in step 62 may be determined based at least in part upon the desired resolution of motion or change detection as well as a desired reaction time. The framerate can, if desired, be programmable and/or dynamically adjustable (e.g., context sensitive). Higher framerates generally reduce the reaction time of the system and reduce the amount of time between the trigger criteria and the activation of the primary imaging sensor 14A. Higher resolutions generally enable the secondary imaging sensor 14B to detect smaller objects and/or smaller amounts of movement.

[0040] Prior to step 66, the primary imaging sensor 14A may be in a standby mode in step 68. In the standby mode, the power consumption of the primary imaging sensor 14A may be reduced.

[0041] In step 70, the primary imaging sensor 14A may begin image capture operations, when triggered in step 66. The primary imaging sensor 14A may, optionally, utilize information on the operating conditions (e.g., auto-white balance, exposure, and focus settings) obtained by the secondary imaging sensor 14B at least in the initial moments of image capture operations of step 70. At later points in time, the operating conditions may be determined using the primary imaging sensor 14A, the secondary imaging sensor 14B, another sensor, or a combination of these and other sensors.

[0042] In step 72, the scene may be monitored for stop conditions. As an example, the stop conditions may be the absence of the trigger criteria specified in step 60. As another example, the stop conditions may be distinct from the trigger criteria. In general, the stop conditions may be specified by the user or preprogrammed, in a fashion similar to that described above in connection with the trigger criteria as specified in step 60. Step 72 may involve the analysis of frames captured by the primary imaging sensor 14A, the secondary imaging sensor 14B, another sensor, or a combination of these and other sensors.

[0043] Once a stop condition is detected in step 72, the primary imaging sensor 14A may be placed into the standby mode of step 68, at least until trigger criteria are again detected in step 62.

[0044] With at least some suitable arrangements, the trigger criteria may be adjusted based on recent images (e.g., recent image context). In these types of arrangements, step 72 may include determining if the desired trigger criteria have changed and, as illustrated by line 74, altering the trigger criteria in step 60 accordingly. As an example and returning to the blooming flower example, step 72 may include determining that the flower has substantially completed its blooming process and determining that the trigger criteria should be adjusted to trigger primary imaging sensor 14A when motion indicative of the start of the death of the flower bloom is detected. With this type of arrangement, the imaging system may capture both the blooming process and the bloom's decay (e.g., the flower petals falling off).

[0045] The operations of FIGS. 5 and 6 may enable various implementations such as an imaging system with adaptive time lapse and an imaging system with adaptive slow motion. In the time lapse example, device 10 may be triggered to capture one frame (or a given number of frames) in response to the detection of trigger criteria such as an object moving a given amount. In this manner, device 10 may be able to capture a time lapse image of clouds, by capturing an image whenever the clouds move by a given amount. Device 10 will not, however, capture unnecessary images in which clouds have not moved relative to earlier images. In the slow motion example, device 10 may be triggered to capture video at a high framerate in response to the detection of trigger criteria. In this manner, device 10 may be able to capture slow motion video of a vehicle running a red light. These are merely illustrative examples.

[0046] FIG. 7 illustrates a simplified block diagram of imager 200 (e.g., an imager that may adaptively adjust image sensor operating conditions such as frame rate based on observation of a scene being imaged). Pixel array 201 includes a plurality of pixels containing respective photosensors arranged in a predetermined number of columns and rows. The row lines are selectively activated by row driver 202 in response to row address decoder 203 and the column select lines are selectively activated by column driver 204 in response to column address decoder 205. Thus, a row and column address is provided for each pixel.

[0047] CMOS imager 200 is operated by a timing and control circuit 206, which controls decoders 203, 205 for selecting the appropriate row and column lines for pixel readout, and row and column driver circuitry 202, 204, which apply driving voltages to the drive transistors of the selected row and column lines. The pixel signals, which typically include a pixel reset signal Vrst and a pixel image signal Vsig for each pixel are sampled by sample and hold circuitry 207 associated with the column driver 204. A differential signal Vrst−Vsig is produced for each pixel, which is amplified by amplifier 208 and digitized by analog-to-digital converter 209. The analog to digital converter 209 converts the analog pixel signals to digital signals, which are fed to image processor 210 which forms a digital image.

[0048] FIG. 8 shows in simplified form a typical processor system 300, such as a digital camera, which includes an imaging device such as imaging device 200 (e.g., an imager that may adaptively adjust image sensor operating conditions such as frame rate based on observation of a scene being imaged). Processor system 300 is exemplary of a system having digital circuits that could include imaging device 200. Without being limiting, such a system could include a computer system, still or video camera system, scanner, machine vision, vehicle navigation, video phone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system, and other systems employing an imaging device.

[0049] Processor system 300, which may be a digital still or video camera system, may include a lens such as lens 396 for focusing an image onto a pixel array such as pixel array 201 when shutter release button 397 is pressed. Processor system 300 may include a central processing unit such as central processing unit (CPU) 395. CPU 395 may be a microprocessor that controls camera functions and one or more image flow functions and communicates with one or more input/output (I/O) devices 391 over a bus such as bus 393. Imaging device 200 may also communicate with CPU 395 over bus 393. System 300 may include random access memory (RAM) 392 and removable memory 394. Removable memory 394 may include flash memory that communicates with CPU 395 over bus 393. Imaging device 200 may be combined with CPU 395, with or without memory storage, on a single integrated circuit or on a different chip. Although bus 393 is illustrated as a single bus, it may be one or more buses or bridges or other communication paths used to interconnect the system components.

[0050] Various embodiments have been described illustrating imaging devices. An imaging device may include a primary imaging sensor and a secondary imaging sensor. The secondary imaging sensor may monitor a scene being imaged by the imaging device for trigger criteria such as sudden movement, changes in the intensity of ambient light, and the presence of a particular object or person. Upon detection of relevant trigger criteria, the imaging device may activate the primary imaging sensor and capture one or more images of the scene with the primary imaging sensor. Operational parameters of the primary imaging sensor including framerate and exposure settings may be based on data from the secondary imaging sensor. The secondary imaging sensor may have a lower power consumption than the primary imaging sensor such that monitoring the scene with the secondary imaging sensor and only activating the primary imaging sensor upon detection of the relevant trigger criteria enables an efficient and adaptive imaging system.

[0051] The foregoing is merely illustrative of the principles of this invention which can be practiced in other embodiments.