US 10,126,412 B2Grant
Optical phased array lidar system and method of using same
Issue Date:2018-11-13
•15 Claims
•6 Drawing Sheets
Abstract
A lidar-based system and method are used for the solid state beamforming and steering of laser beams using optical phased array (OPA) photonic integrated circuits (PICs) and the detection of laser beams using photodetectors. Transmitter and receiver electronics, power management electronics, control electronics, data conversion electronics and processing electronics are also included in the system and used in the method.
Laser pulses beamformed by the OPA PIC reflect from objects in the field of view (FOV) of said OPA, and are detected by a detector or a set of detectors.
A lidar system includes at least one lidar, and any subset and any number of complementary sensors, data processing/communication/storage modules, and a balance of system for supplying power, protecting, connecting, and mounting the components of said system.
Direct correlation between the 3D point cloud generated by the lidar and the color images captured by an RGB (Red, Green, Blue) video camera can be achieved by using an optical beam splitter that sends optical signals simultaneously to both sensors.
A lidar system may contain a plurality of lidar sensors, a lidar sensor may contain a plurality of optical transmitters, and an optical transmitter may contain a plurality of OPA PICs.
Metadata
Assignee
- Quanergy Systems, Inc.
Inventors
- Louay Eldada
- Tianyue Yu
- Angus Pacala
Application Information
Application Number:US 14/279,348
Filing Date:2014-05-16
Priority Date:2013-08-19
Art Unit:3645
Classifications
IPC:
G01C3/08G01S7/481G01S17/02G01S17/93G01S7/486G01S17/89G01S13/86
Field of Search:
G01S 17/023G01S 17/936G01S 7/4863G01S 7/4818G01S 7/4814
Patent Drawings (6 sheets)
Description
Priority Claim
[0001] The present application claims the benefit of priority from U.S. Provisional Application Ser. No. 61/867,574, filed Aug. 19, 2013.
References Cited
[0002]
| U.S. Patent Documents | |||
|---|---|---|---|
| 7,339,727 B1 | March 2008 | Rothenberg | |
| 7,406,220 B1 | July 2008 | Christensen | |
| 7,428,100 B2 | September 2008 | Smith | |
| 7,436,588 B2 | October 2008 | Rothenberg | |
| 7,489,870 B2 | February 2009 | Hillis | |
| 7,532,311 B2 | May 2009 | Henderson | |
| 7,555,217 B2 | July 2009 | Hillis | |
Field of the Invention
[0003] The present invention relates generally to the field of environment sensing, and more particularly to the use of Time of Flight (ToF) lidar sensors for real-time three-dimensional mapping and object detection, tracking, identification and/or classification.
Background of the Invention
[0004] A lidar sensor is a light detection and ranging sensor. It is an optical remote sensing module that can measure the distance to a target or objects in a scene, by irradiating the target or scene with light, using pulses (or alternatively a modulated signal) from a laser, and measuring the time it takes photons to travel to said target or landscape and return after reflection to a receiver in the lidar module. The reflected pulses (or modulated signals) are detected, with the time of flight and the intensity of the pulses (or modulated signals) being measures of the distance and the reflectivity of the sensed object, respectively.
[0005] Conventional lidar sensors utilize mechanically moving parts for scanning laser beams. In some systems, including certain systems used in automotive applications, such as advanced driver assist systems (ADAS) and autonomous driving systems, it is preferred to use solid state sensors for a variety of potential advantages including but not limited to higher sensor reliability, longer sensor lifetime, smaller sensor size, lower sensor weight, and lower sensor cost.
[0006] Radio frequency (RF) delay lines used for the creation of radar phased arrays were used several decades ago for the solid state steering of radar signals. Photonic integrated circuit (PIC) based delay lines combined with detectors and RF antenna arrays were used two decades ago to improve the precision of delays in the solid state steering of radar signals. PICs with microscale and nanoscale devices can be used to produce optical phased arrays (OPAs), comprising tunable optical delay lines and optical antennas, for the solid state steering of laser beams. Phased Arrays in the optical domain that are produced to date are complex, costly and/or have a different purpose than beam forming and beam steering; some combine spatial filters, optical amplifiers and ring lasers (U.S. Pat. No. 7,339,727), some involve a plurality of optical input beams (U.S. Pat. No. 7,406,220), some involve volume diffraction gratings and a plurality of input directions (U.S. Pat. No. 7,428,100), some combine beams of a plurality of wavelengths (U.S. Pat. No. 7,436,588), some have optical phase reference sources and gain elements (U.S. Pat. No. 7,489,870), some have predetermined areas in the field of view and a plurality of beam forming elements (U.S. Pat. No. 7,532,311), and some have multiple frequencies and multiple optical phase reference sources (U.S. Pat. No. 7,555,217).
Summary of the Invention
[0007] A lidar-based system and method are used for the solid state beamforming and steering of laser beams using OPA PICs and the detection of laser beams using photodetectors. Transmitter and receiver electronics, power management electronics, control electronics, data conversion electronics and processing electronics are also included in the system and used in the method.
[0008] Laser pulses beamformed by the OPA PIC reflect from objects in the field of view (FOV) of said OPA, and are detected by a detector or a set of detectors.
[0009] A lidar system includes at least one lidar, and any subset and any number of complementary sensors, data processing/communication/storage modules, and a balance of system for supplying power, protecting, connecting, and mounting the components of said system.
[0010] Direct correlation between the 3D point cloud generated by the lidar and the color images captured by an RGB (Red, Green, Blue) video camera can be achieved by using an optical beam splitter that sends optical signals simultaneously to both sensors.
[0011] A lidar system may contain a plurality of lidar sensors, a lidar sensor may contain a plurality of optical transmitters, and an optical transmitter may contain a plurality of OPA PICs.
Description of the Drawings
[0012] The following drawings are illustrative of embodiments of the present invention and are not intended to limit the invention as encompassed by the claims forming part of the application.
[0013] The schematic diagram of FIG. 1 provides a frontal view of a solid state lidar sensor 10 that can be implemented using the present invention, depicting an OPA-comprising transmitter 20, a receiver 30, a processor 40 and one or a plurality of printed circuit boards 50 comprising control electronics. A solid state lidar sensor 10 may contain a plurality of optical transmitters 20, and an optical transmitter 20 may contain a plurality of OPA PICs.
[0014] The schematic diagram of FIG. 2 provides an angled view of a solid state lidar sensor 10 that can be implemented using the present invention, depicting an OPA-comprising transmitter 20, a receiver 30, a processor 40 and one or a plurality of printed circuit boards 50 including control electronics.
[0015] The schematic diagram of FIG. 3 provides a top view of a solid state lidar sensor 10 that can be implemented using the present invention, depicting an OPA-comprising transmitter 20, a receiver 30, a processor 40 and one or a plurality of printed circuit boards 50 including control electronics.
[0016] The schematic diagram of FIG. 4 provides a side view of a solid state lidar sensor 10 that can be implemented using the present invention, depicting an OPA-comprising transmitter 20, a receiver 30, a processor 40 and one or a plurality of printed circuit boards 50 including control electronics.
[0017] The schematic diagram of FIG. 5 provides a view of a vehicle-mounted lidar system 60 that contains a plurality of lidar sensors 10.
Detailed Description of the Invention
[0018] A lidar-based system and method are used for the solid state beamforming and steering of laser beams using OPA PICs and the detection of laser beams using photodetectors. Transmitter and receiver electronics, power management electronics, control electronics, data conversion electronics and processing electronics are also included in the system and used in the method.
[0019] Microfabrication and/or nanofabrication techniques are used for the production of OPA PICs that include optical power splitters that distribute an optical signal from a laser, optical-fiber-coupled to the chip or integrated on the chip, to tunable optical delay lines for phase control, and said delay lines direct their output optical signals to optical antennas for out-of-plane coupling of light.
[0020] For each set of settings for the tuning elements (e.g., ohmic heating electrodes) of said delay lines, said optical antennas emit light beams with specific phase delays, forming a desired far-field radiation pattern through the interference of said emitted beams.
[0021] Settings of said tuning elements of said delay lines can be varied to generate by ‘random access’ any sequence of far-field radiation patterns. In a specific embodiment, the far-field radiation pattern essentially maintains its shape as it is moved to any desired sequence of locations; in a more specific embodiment, said far-field radiation pattern whose shape is kept essentially constant is swept in the far field to form a rastered line (e.g., a serpentine line).
[0022] In a TOF lidar application, a OPA-based lidar includes an optical transmitter (including laser, laser driver, laser controller, OPA PIC, and OPA controller), an optical receiver (including photodetector(s), photodetector driver(s), and receiver electronics), and electronics for power regulation, control, data conversion, and processing.
[0023] Photodetector types include avalanche photodiodes (APD) and PIN diodes (PIN diodes are positive-intrinsic-negative diodes, as they comprise a lightly-doped intrinsic semiconductor region between a a-type or positive-type semiconductor region and an n-type or negative-type semiconductor region).
[0024] Laser pulses beamformed by the OPA PIC reflect from objects in the field of view (FOV) of said OPA, and are detected by a detector or a set of detectors (including 1 D and 2D detector arrays). Detector arrays include staring arrays, staring-plane arrays, or focal-plane arrays (FPA), which consist of an array (typically 2D) of light-sensing pixels at the focal plane of a lens. The light-sensing pixels can be single-photon avalanche diodes (SPADs).
[0025] The OPA PIC is preferably compatible with a complementary metal-oxide-semiconductor (CMOS) process, and is preferably based on a silicon on insulator (SOI) structure. The OPA PIC may contain optical waveguiding elements composed of crystalline silicon, amorphous silicon and/or silicon nitride.
[0026] When the OPA PIC is based on a CMOS (complementary metal-oxide-semiconductor) process, it can be integrated with optoelectronics and/or electronics that are part of the same lidar (including but not limited to any number of lasers, laser drivers, laser controllers, optical amplifiers, optical detectors, receiver electronics, power regulation electronics, control electronics, data conversion electronics, data processing electronics) and are based on a CMOS process or can be hybridly integrated with CMOS technology.
[0027] A lidar system includes at least one lidar, and any subset and any number of the following:
-
- [0028] Complementary sensors
- [0029] GPS (Global Positioning System) or GNSS (Global Navigation Satellite System) receiver
- [0030] IMU (Inertial Measurement Unit)
- [0031] Wheel encoder
- [0032] Video camera (visible and/or IR)
- [0033] Radar
- [0034] Ultrasonic sensor
- [0035] Data processing/communication/storage modules
- [0036] Embedded processor
- [0037] Ethernet controller
- [0038] Cell modem
- [0039] Wi-Fi controller
- [0040] Data storage drive
- [0041] HMI (Human Machine Interface) e.g., display, audio, buzzer
- [0042] Balance of system
- [0043] Power supply
- [0044] Enclosure
- [0045] Cabling
- [0046] Mounting hardware
- [0028] Complementary sensors
[0047] Direct correlation between the 3D point cloud generated by the lidar and the color images captured by an RGB (Red, Green, Blue) video camera can be achieved by using an optical beam splitter that sends optical signals simultaneously to both sensors, simplifying the sensor fusion that generates a color point cloud or RGBD data (Red, Green, Blue and Depth). The OPA PIC, optical receiver and/or RGB video camera can be integrated on a single printed circuit board (PCB).
[0048] For reasons including but not limited to redundancy and widening the field of view, a lidar system may contain a plurality of lidar sensors, a lidar sensor may contain a plurality of optical transmitters, and an optical transmitter may contain a plurality of OPA PICs.
Claims
What is claimed is:
1. An apparatus, comprising:
a transmitter to transmit out-of-plane light, wherein the out-of-plane light has specific phase delays that form a desired far-field radiation pattern through the interference of emitted beams;
a receiver with an optical intensity beam splitter used to split and send optical signals simultaneously to an optical receiver assembly of a lidar and a video camera for direct correlation between a three-dimensional point cloud generated by the far-field radiation pattern and color images captured by the video camera to form red, green, blue and depth data; and
a printed circuit board hosting the transmitter, the receiver and an optical phased array photonic integrated circuit of the lidar.
2. The apparatus of claim 1 wherein said far-field radiation pattern is swept to form a rastered line.
3. The apparatus of claim 1 wherein the receiver includes photodetector drivers connected to photodetectors.
4. The apparatus of claim 3 wherein the photodetectors are avalanche photodiodes.
5. The apparatus of claim 3 wherein the photodetectors are positive-intrinsic-negative diodes.
6. The apparatus of claim 3 wherein the photodetectors form a staring array.
7. The apparatus of claim 3 wherein the photodetectors form a staring-plane array.
8. The apparatus of claim 3 wherein the photodetectors form a focal-plane array.
9. The apparatus of claim 3 configured as a silicon on insulator structure.
10. The apparatus of claim 1 wherein the optical intensity beam splitter is a visible and near-infrared beam splitter.
11. The apparatus of claim 10 wherein the visible and near-infrared beam splitter is a plate beam splitter.
12. The apparatus of claim 10 wherein the visible and near-infrared beam splitter is a cube beam splitter.
13. The apparatus of claim 10 wherein the visible and near-infrared beam splitter is a prism beam splitter.
14. The apparatus of claim 10 wherein the visible and near-infrared beam splitter is a pellicle beam splitter.
15. The apparatus of claim 10 wherein the visible and near-infrared beam splitter is a partially-metallized mirror.
Patent Citations (67)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US3781552(A) | 1973-12-01 | Kadrmas | Applicant |
| US5132843(A) | 1992-07-01 | Aoyama et al. | Applicant |
| US5210586(A) | 1993-05-01 | Grage et al. | Applicant |
| US5455669(A) | 1995-10-01 | Wetteborn | Applicant |
| US5543805(A) | 1996-08-01 | Thaniyavarn | Examiner |
| US5552893(A) | 1996-09-01 | Akasu | Applicant |
| US5682229(A) | 1997-10-01 | Wangler | Applicant |
| US5898483(A) | 1999-04-01 | Flowers | Applicant |
| US6765663(B2) | 2004-07-01 | Byren | Examiner |
| US6891987(B2) | 2005-05-01 | Ionov | Examiner |
| US7746449(B2) | 2010-06-01 | Ray et al. | Applicant |
| US7969558(B2) | 2011-06-01 | Hall | Examiner |
| US8125367(B2) | 2012-02-01 | Ludwig | Applicant |
| US8203115(B2) | 2012-06-01 | Hochberg et al. | Applicant |
| US8311374(B2) | 2012-11-01 | Hochberg | Examiner |
| US8731247(B2) | 2014-05-01 | Pollock | Examiner |
| US8836922(B1) | 2014-09-01 | Pennecot | Examiner |
| US8988754(B2) | 2015-03-01 | Sun | Examiner |
| US9014903(B1) | 2015-04-01 | Zhu | Applicant |
| US9069080(B2) | 2015-06-01 | Stettner et al. | Applicant |
| US9104086(B1) | 2015-08-01 | Davids et al. | Applicant |
| US9332243(B2) | 2016-05-01 | Klusza | Examiner |
| US9383753(B1) | 2016-07-01 | Templeton et al. | Applicant |
| US2005/0033497(A1) | 2005-02-01 | Stopczynski | Applicant |
| US2006/0091303(A1) | 2006-05-01 | Evans | Examiner |
| US2006/0197936(A1) | 2006-09-01 | Libeman et al. | Applicant |
| US2006/0239688(A1) | 2006-10-01 | Hillis et al. | Applicant |
| US2007/0052947(A1) | 2007-03-01 | Ludwig et al. | Applicant |
| US2007/0065002(A1) | 2007-03-01 | Marzell | Examiner |
| US2007/0181810(A1) | 2007-08-01 | Tan et al. | Applicant |
| US2008/0094607(A1) | 2008-04-01 | Bernard et al. | Applicant |
| US2008/0186470(A1) | 2008-08-01 | Hipp | Applicant |
| US2008/0204699(A1) | 2008-08-01 | Benz et al. | Applicant |
| US2008/0227292(A1) | 2008-09-01 | Miki | Applicant |
| US2009/0059201(A1) | 2009-03-01 | Willner | Examiner |
| US2009/0251680(A1) | 2009-10-01 | Farsaie | Examiner |
| US2009/0278030(A1) | 2009-11-01 | Deliwala | Applicant |
| US2010/0045964(A1) | 2010-02-01 | Jin et al. | Applicant |
| US2010/0187402(A1) | 2010-07-01 | Hochberg | Examiner |
| US2010/0187442(A1) | 2010-07-01 | Hochberg | Examiner |
| US2010/0253585(A1) | 2010-10-01 | Llorens del Rio et al. | Applicant |
| US2010/0271614(A1) | 2010-10-01 | Albuquerque et al. | Applicant |
| US2010/0290029(A1) | 2010-11-01 | Hata | Applicant |
| US2011/0216304(A1) | 2011-09-01 | Hall | Applicant |
| US2011/0222814(A1) | 2011-09-01 | Krill et al. | Applicant |
| US2011/0255070(A1) | 2011-10-01 | Phillips et al. | Applicant |
| US2011/0316978(A1) | 2011-12-01 | Dillon | Examiner |
| US2012/0013962(A1) | 2012-01-01 | Subbaraman | Examiner |
| US2012/0226118(A1) | 2012-09-01 | Delbeke et al. | Applicant |
| US2012/0286136(A1) | 2012-11-01 | Krill | Examiner |
| US2013/0027715(A1) | 2013-01-01 | Imaki et al. | Applicant |
| US2013/0044309(A1) | 2013-02-01 | Dakin | Examiner |
| US2013/0114924(A1) | 2013-05-01 | Loh et al. | Applicant |
| US2013/0127980(A1) | 2013-05-01 | Haddick | Applicant |
| US2013/0208256(A1) | 2013-08-01 | Mamidipudi | Examiner |
| US2013/0242400(A1) | 2013-09-01 | Chen | Examiner |
| US2013/0301976(A1) | 2013-11-01 | Saida et al. | Applicant |
| US2014/0152871(A1) | 2014-06-01 | Campbell | Applicant |
| US2014/0211194(A1) | 2014-07-01 | Pacala | Applicant |
| US2014/0240691(A1) | 2014-08-01 | Mheen | Applicant |
| US2014/0376001(A1) | 2014-12-01 | Swanson | Applicant |
| US2015/0192677(A1) | 2015-07-01 | Yu | Applicant |
| US2015/0293224(A1) | 2015-10-01 | Eldada | Applicant |
| US2015/0346340(A1) | 2015-12-01 | Yaacobi | Applicant |
| US2015/0378241(A1) | 2015-12-01 | Eldada | Applicant |
| US2016/0047901(A1) | 2016-02-01 | Pacala | Applicant |
| US2016/0049765(A1) | 2016-02-01 | Eldada | Applicant |
Non-Patent Literature (8)
- Guo et al., “InP Photonic Integrated Circuit for 2D Optical Beam Steering”, Photonics Conference, Oct. 9-13, 2011, Arlington, VA, IEEE, 2011, 3 pgs.Applicant
- Guo et al., “Two-Dimensional optical Beam Steering with InP-Based Photonic Integrated Circuits”, IEEE Journal of Selected Topics in Quantum Electronics, vol. 19, No. 4, Aug. 2013, pp. 1, 6, 8, 11.Applicant
- Hulme et al., “Fully Integrated Hybrid Silicon Free-Space Beam Steering Source with 32 Channel Phased Array”, SPIE, Mar. 26, 2014, pp. 898907-2, 898907-13.Applicant
- Taillaert et al., “An Out-Of-Plane Grating Coupler for Efficient Butt-Coupling Between Compact Planar Waveguides and Single-Mode Fibers”, IEEE Journal of Quantum Electronics, vol. 38, No. 7, Jul. 2002, pp. 951-953.Applicant
- International Search Report and Written Opinion issued to International Patent Application No. PCT/US15/44069, Nov. 12, 2015, 9 pgs.Applicant
- Van Acoleyen, Karel, “Nanophotonic Beamsteering Elements Using Silicon Technology for Wireless Optical Applications”, Ghent University, Dept. of Information Technology, Aug. 27, 2012, 180 pgs.Applicant
- Van Acoleyen, Karel, “Off-Chip Beam Steering with a One-Dimensional Optical Phased Array on Silicon-On-Insulator”, Optics Letters, vol. 34, No. 9, May 1, 2009, pp. 1477-1479.Applicant
- International Search Report and Written Opinion issued to international patent application No. PCT/US15/56516, Feb. 4, 2016, 7 pgs.Applicant