METHOD FOR TIME OF FLIGHT MODULATION FREQUENCY DETECTION AND ILLUMINATION MODULATION FREQUENCY ADJUSTMENT

[0001] Acquiring 3D geometric information from real environments is an essential task for many applications in computer graphics. Prominent examples such as virtual and augmented environments and human machine interaction, e.g. for gaming, clearly benefit from simple and accurate devices for real-time range image acquisition. However, even for static scenes there is no low-price off-the-shelf system that provides full-range, high resolution distance information in real time. Laser triangulation techniques, which merely sample a scene row by row with a single laser device, are rather time-consuming and therefore impracticable for dynamic scenes. Stereo vision camera systems suffer from the inability to match correspondences in homogeneous object regions.

[0002] Time-of-Flight (ToF) technology, based on measuring the time that light emitted by an illumination unit requires to travel to an object and back to a detector, is used in LIDAR (Light Detection and Ranging) scanners for high-precision distance measurements. Recently, this principle has been the basis for the development of new range-sensing devices, so-called ToF cameras, which are realized in standard CMOS or CCD technology; in the context of photogrammetry, ToF cameras are also called Range Imaging (RIM) sensors. Unlike other 3D systems, the ToF camera is a very compact device which already fulfills most of the above-stated features desired for real-time distance acquisition. There are two main approaches currently employed in ToF technology. The first one utilizes modulated, incoherent light, and is based on a phase measurement. The second approach is based on an optical shutter technology, which was first used for studio cameras and was later developed for miniaturized cameras

[0003] Within the last three years, the number of research activities in the context of ToF cameras has increased dramatically. While the initial research focused on more basic questions like sensor characteristics and the application of ToF cameras for the acquisition of static scenes, other application areas have recently come into focus, e.g. human machine interaction and surveillance.

[0004] A variety of safety-enhancing automobile features can be enabled by digital signal processors that can sense and analyze the dynamic 3D environment inside and outside the vehicle. Safety features may include collision warning and avoidance, smart airbag deployment, obstacle detection such as backup warning, and parking assistance. Common to these applications is the need to detect, isolate, measure, locate, recognize, and track objects such as people, traffic, and roadside features.

[0005] It is often proposed to perform these tasks using conventional 2D imaging sensors and analysis software, but achieving cost-effective and reliable performance during all vehicular usage scenarios is a formidable challenge. The appearance of objects in a 2D image varies greatly, depending on illumination conditions, surface materials, and object orientation. These variations in the image complicate the task of software that must interpret the scene. On the other hand, the 3D shape of objects is invariant to those confounding effects.

[0006] Stereovision based 3D recovery is computationally complex and fails on un-patterned surfaces. RADAR, ultrasonic, scanning LADAR, and other ranging technologies are similarly proposed, but they have difficulty discriminating objects due to limited temporal or angular resolution; moreover, the need for specialized sensors for each safety function poses system integration challenges. A single high frame rate focal-plane-array 3D sensor is desirable because it can serve multiple safety and convenience functions simultaneously, allowing applications to jointly exploit shape and appearance information in a dynamic scene. The output of the sensor should be a sequence of 2D arrays of pixel values, where each pixel value describes the brightness and Cartesian X,Y,Z coordinates of a 3D point on the surface of the scene.

[0007] Growing government legislation, increasing liability concerns, and the inevitable consumer desire for improved safety make the introduction of new safety features a high priority for automakers. Today, various sensing technologies play a key role in delivering these features, detecting conditions both inside and outside of the vehicle in applications like parking assistance, adaptive cruise control, and pre-crash collision mitigation. Each of these applications is characterized by a unique customized technology (e.g. ultrasonic, RADAR, LADAR, digital image sensing, etc.), which generally provides either a ranging function or an object recognition function.

[0008] The need for investment in multiple disparate technologies makes it challenging to deploy individual safety features as quickly or as broadly as desired.

[0009] Future applications pose even more difficulties, as multiple features must be provided in a single vehicle. Plus, virtually all of the new sensing applications on automakers' roadmaps (e.g. pedestrian detection being planned in Europe and Japan) require both ranging and object recognition functions. Combining two incongruent technologies to accomplish this task (such as RADAR and digital image sensing) is expensive, difficult to implement, and poses the additional problem of inefficient development.

[0010] The use of vision gives added levels of discernment to the air bag systems by providing static or dynamic occupant classification and position sensing. Further, the addition of a vision system inside the cabin enables other value-added applications such as abandoned baby/pet detection, personalization, and security. Applications for vision-based sensing outside the car are blind spot detection, vehicle lane departure, safety in rear vision, proximity of other vehicles around the vehicle, and off road and heavy equipment proximity sensing. The benefits of vision sensors are two fold. They provide enhanced visual feedback to assist the driver in operating the vehicle. But more importantly, when vision sensors also provide range data, they provide the necessary information for advanced algorithms to achieve higher level of discernment and more accurate analysis of object motion dynamic. With such sensors, for instance, the system can use the shape differences between a person and a large box sitting in the front seat to deploy the air bag or not.

[0011] In addition to depth values, ToF cameras also provide intensity values, representing the amount of light sent back from a specific point.

[0012] Due to the periodicity of the modulation signal, ToF cameras have a defined non-ambiguous range. Within this range, distances can be computed uniquely. The range depends on the modulation frequency of the camera which defines the wave length of the emitted signal. As shown in FIG. 1, to compute distances the camera evaluates the phase shift between a reference (emitted) signal 101 and the received signal 102. The phase shift is proportional to the distance d.

[0013] Currently most ToF cameras operate at a modulation frequency of about 20 MHz for gaming, TV control gesture and digital signage etc. Then, a single wavelength is 15 meter, ant the unique range of these ToF cameras is approx. 7.5 meter. This frequency may be changed for automotive use where a car moves at 60 miles per hour or faster to obtain a longer range coverage. The range can be adjusted by adjusting the modulation frequency of the active illumination depending on automobile's speed.

[0014] Depth sensor technology is a cutting edge new technology. The TOF sensor's cross-interference from modulated lights has not been considered mainly because multiple TOF cameras have not been adopted to any great degree. As a result, TOF modulation frequencies have been manually reconfigured. However, for example, if TOF technology is widely applied to automobile applications e.g. for front and rear scene analysis, it is almost impossible to assign individually separate TOF modulation frequencies for each sensor product. The given technique allows a dynamic change of TOF modulation frequency by detecting other TOF sensors' modulation frequencies. Thus, the cross-interference of modulated illumination among multiple TOF sensors can be avoided.

[0018] A TOF (Time Of Flight) sensor delivers a very accurate depth measurement and can be widely applied to applications within various environments. However, TOF sensor technology intrinsically has a cross interference problem among TOF sensors due to IR illuminations from multiple TOF sensors employing the same modulation frequency. Since IR modulation frequency is directly related to a covered depth range of TOF, there is the likelihood that the same modulation frequency of TOF IR illumination will be selected among TOF sensors in place.

[0019] For example, for applications targeting a mid range depth map, in many cases a 20 MHz IR modulation frequency is chosen. Interference from TOF sensors with the same IR modulation frequency (20 MHz) can significantly impact on depth accuracy of each sensor.

[0020] The described invention provides a method to avoid IR interference among TOF sensors by combining the IR modulation frequency detector with logic for dynamic adjustment of the IR transmission modulation frequency.

[0021] As shown in FIG. 2, the invention consists of mainly two parts, the AIMD (Ambient IR Modulation-frequency Detection) module and TIMA (Transmission IR Modulation-frequency Adjustment) module.

[0022] The AIMD module detects all ambient IR modulation-frequencies 208 by IR frequency detector 201, and MCU 202 analyzes each modulation frequency and its duration time by IR Modulation Frequency Monitoring Routine 203. If its duration Δtime(f_r,IR) is longer than a predefined threshold Th_Δtime, the MCU triggers IR Modulation Frequency Adjusting Routine 207 to adjust the current IR transmission modulation frequency 209 of the associated TOF sensor. The predefined threshold can be dynamically re-configured to adjust the sensitivity of TOF to interference noise and the requirement for real-time processing.

[0023] A minor change of IR modulation frequency 209 (e.g. 20 Mh->20.1 Mhz) can significantly reduce the interference related noise. Thus, the interference of ambient IR modulation frequencies can be removed in the depth map acquisition of TOF sensor 205. As a result, TOF sensor raw data 210 can be captured without ambient cross-interference noise.