METHOD AND APPARATUS FOR INTENSITY CALIBRATION OF MULTI-BEAM LIDAR SENSORS

[0001] The present disclosure generally relates to a method and apparatus for intensity calibration of multi-beam Lidar sensors. More specifically, the present disclosure relates to a method and apparatus for intensity calibration of multi-beam Lidar sensors based on probability density estimation.

[0002] Lidar sensors are getting more sophisticated, often consisting of multiple pairs of laser emitters and receptors, each pair being referred to as a beam. Multi-beam Lidar sensors have significant utility for a variety of automotive applications that include safety applications and autonomous driving. Lidar sensors conventionally provide both a 3D spatial component and a reflectivity measurement of the material being struck by each laser beam. The reflectivity measurement may be a very useful source of information about the environment. However, empirical testing on commercially available Lidar units has shown that calibration of the reflectivity measurement is often poor.

[0003] This disclosure provides a method and/or apparatus for calibration of the reflectivity measurement of a multi-beam Lidar sensor based on probability density estimation.

[0014] According to an embodiment of the present disclosure, there is provided a method and apparatus for multi-beam Lidar intensity measurement calibration including determining position information of a moving platform within a global reference frame, collecting a plurality of data points, each data point including the position information and intensity information from a plurality of beams, segmenting the plurality of data points into a plurality of voxels based on the position information associated with each of the plurality of data points, finding a subset of voxels from among the plurality of voxels that include a point corresponding to a specific beam from among the plurality of beams and including a specific intensity value, estimating a probability density over the intensity values in all data points in the subset of voxels, and generating a calibration map for the specific beam based on the probability density.

[0015] Calibrated Lidar reflectivity measurements may be used for road-paint detection for lane markings, road-sign detection based on high reflectivity, or creation of reflectivity or intensity maps. These maps may be used for vehicle positioning or correction of poor GPS data via localization algorithms.

[0016] Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, FIG. 1 shows an illustration of a system including a Lidar sensor 101, mounted on a movable platform 103, and detecting a target 105.

[0017] FIG. 2 is a block diagram of a Lidar system. The Lidar system may include multiple beams 203. Each beam includes a laser emitter 205 and receptor 207 pair. The laser emitter 205 and receptor 207 pair are configured such that a laser signal transmitted by the laser emitter 205 is received by the laser receptor 207 after being reflected by an object 201. A controller 209 controls the functions of the beam 203. The controller also controls a positioning device 211 to determine position information corresponding to data collected from the beam 203.

[0018] When mounted on a moving platform 103, different Lidar beams encounter the same target 105 at different times. The Lidar produces a large number of Lidar points p. Each Lidar point includes a 3D coordinate p_x, p_y, p_z relative to a global coordinate frame, and an intensity measurement p_r. The moving platform 103 may compute its position information, so that the points generated by the Lidar are projected into the global coordinate frame. The position information may be computed using, e.g., a wheel odometry device, an inertial measurement device, a global positioning system (GPS), or any combination of these.

[0019] Due to inaccuracies and/or physical variations, the intensity returns are highly variable between two different beams, even when hitting the same physical object which is assumed to have relatively consistent reflective properties.

[0020] According to an embodiment of the present disclosure, to calibrate the intensity measurements, a large amount of data points from the moving platform 103 is collected and segmented into a large number of small 3D voxels.

[0021] Then, for each beam and each possible intensity, the corresponding set of voxels that contain this particular measurement are found, and the probability density over the intensities returned by all beams for those voxels are estimated. The data structures used to collect the data points may include a large 3D array of voxels, and a data structure that indexes into those voxels as explained next. The array of voxels may be constructed so that memory for any given voxel is allocated only when necessary (i.e., when there is a Lidar point that falls within that voxel). Further, as multiple points fall within a given voxel, summary statistics such as the mean intensity of all points currently in that voxel are collected. The data structure that indexes into the voxel array includes a mapping that returns the set of voxels that corresponds to each beam and discrete intensity value. For example, beam 32 and intensity value 127 has a set of voxels corresponding to the times that beam 32 has measured intensity 127. That set of voxels forms the statistics of the probability density used (in the basic probability density estimation algorithm as will be described later) for the calibration of beam 32, intensity 127.

[0022] Finally, given the estimated probability density and a prior density, for each beam independently, a calibration map from measured (input) to calibrated (output) intensities is generated. This mapping is the maximum likelihood of the posterior density. Different embodiments of this method are described next with reference to FIGS. 3-6.

[0023] FIG. 3 is an illustration of a single voxel 301 containing a collection of points 303. The set of all points pεP and all voxels vεV are processed to produce the sets P_v of Lidar points in each voxel v, and the sets V(l_i,r_j) of voxels that contain a point p from laser l_i with intensity r_j.

[0024] According to an embodiment of the present disclosure, the method used to estimate the probability density is a weighted kernel-based approach. This algorithm uses the set V_limrj of voxels that contain a point p from laser l_i with intensity rj to construct the density estimate. Each voxel v_k computes the average intensity r_vk of all points falling within it

$\begin{array}{cc}{r}_{v_k}=\frac{1}{P_{v_k}}\sum _{p\in P_{v_k}}p_r& \left(1\right)\end{array}$

which, along with a kernel bandwidth h, defines a kernel as a function K_h(x−r_vk) of x for that voxel. Following is an example code for determining the kernel bandwidth. The constants at the beginning of the code are selected based on the range of possible values (e.g., 0-255), and for maintaining smooth PDFs (i.e. having a standard deviation between min-stddev and max-stddev). kernel_density.h_ is the bandwidth and num_points is the total number of points collected in the corresponding voxels.

double min_stddev = 10.0;
double max_stddev = 100.;
double stddev_param = 0.4;
	if(num_points == 0) {
	kernel_density.h_ = max_stddev;
	} else {
	kernel_density.h_ = max_stddev * 1./ (pow((double)num_points
/ 1000, stddev_param));
	if(kernel_density.h_ < min_stddev) {
	kernel_density.h_ = min_stddev;
	}
	}

[0025] The basic probability density function (of x) for Laser l_i and intensity r_j is

$\begin{array}{cc}{p}_{l_i,r_j}\left(x\right)=\frac{1}{V_{l_i,r_j}}\sum _{v\in V_{l_i,r_j}}K_h\left(x-r_v\right)& \left(2\right)\end{array}$

[0026] FIG. 4 is a flowchart of an embodiment of the above-described method for estimating the probability density over calibrated (output) intensities for each beam l_i and measured (input) intensity r_j.

[0027] In step S401, a laser l_i and an intensity value r_j are selected.

[0028] In step S403, the set V_li,rj of voxels v_k that contain a point p from laser l_i with intensity r_j is determined.

[0029] In step S405, for each voxel v_k in V_ii,rj, the average intensity r_vk of all points falling within it are computed.

[0030] In step S407, for each voxel v_k in V_li,rj, a kernel with bandwidth h is defined as a function K_h(x−r_vk) of x.

[0031] In step S409, the average of kernels K_h(x−r_vk) of voxels v_k in V_li,rj is set as the density estimate p_li,rj(x).

[0032] The derived density estimates for neighboring input intensities (e.g. r_j and r_j±1 when intensities are measured in integers) are independent, unless some of the voxels used for these estimates are shared (e.g. unless V_ii,rj∩V_li,rj+1≠0). This independence may result in a calibration map that is relatively smooth, and may result in poor calibration in under-sampled regions of the intensity space.

[0033] The voxel size may be manually tuned to correspond to the range of the Lidar sensor and a typical minimum size of objects encountered in the environment. For example, for a Lidar that can report beyond 50 meters, a voxel size of 20 cm may be used in a road and/or parking lot environment.

[0034] According to another embodiment of the present disclosure, an extended algorithm is provided that generates an estimate based on weighted kernels from voxels corresponding to neighboring input intensities. The effective number of neighbors and the weights are parameters of the extended probability density estimation algorithm, specified by an additional kernel bandwidth h_n.

[0035] The extended probability density function makes use of two kernel bandwidths, one for neighboring input intensities h_n and one for each voxel's contribution h. The extended probability density function (of x) for Laser l_i and intensity r_j is

$\begin{array}{cc}{p}_{l_i,r_j}\left(x\right)=\frac{1}{R}\sum _{r\in R}K_{h_n}\left(r-r_j\right)\frac{1}{V_{l_i,r}}\sum _{v\in V_{l_i,r}}K_h\left(x-r_v\right)& \left(3\right)\end{array}$

[0036] FIG. 5 is a flowchart for extended probability density estimation, according to this embodiment.

[0037] In step S501, a laser l_i, an intensity r_j, and a set R of neighboring intensity values r are selected.

[0038] In step S503, for each intensity value r in R, the set V_li,r of voxels v that contain a point p from laser l_i with intensity r are determined.

[0039] In step S505, for each voxel v in each set V_li,r, the average intensity r_v of all points falling within voxel v are computed.

[0040] In step S507, for each voxel v in each set V_li,r, a kernel with bandwidth h is defined as a function K_h(x−r_v) of x.

[0041] In step S509, for each set V_li,r, the average of kernels K_h(x−r_v) of voxels v is set as the density estimate p_li,r(x).

[0042] In step S511, a kernel K_hn with bandwidth h_n and mean r_j is defined.

[0043] In step S513, p_li,rj(x) is set as the weighted average of p_li,r(x) over R, where the weighting of each p_li,r(x) is K_hn(r−r_j).

[0044] The calibration map may be generated based on the probability density estimates according to any of the above-described embodiments. The calibration map is a deterministic mapping M(i,j)→{circumflex over (r)}, with an entry for every laser l_i input intensity r_j pair to an output intensity {circumflex over (r)}. The calibration map is used to correct the reported intensity values as data is being read from the Lidar.

[0045] According to an embodiment of the present invention, to generate the calibration map, the estimated probability density is combined with a Gaussian prior using Bayes rule, and the resulting maximum likelihood density is chosen as the mapped output value. Integer intensity values in the range [0, 255] result in integer output intensity values in the same range.

[0046] An example of a prior density p(M(i,j)={circumflex over (r)}) is shown in FIG. 7. The x axis corresponds to the input intensity r_j, and the y axis corresponds to the output {circumflex over (r)}. The z axis is the maximum likelihood density. The same prior is used for all lasers l_i. According to an embodiment of the present disclosure, for laser l_i and input intensity r_j, the prior p(M(i,j)={circumflex over (r)}) is defined by starting with a Gaussian with mean r_j and standard deviation σ, cropping to the range of possible intensity values, and re-normalizing. The prior has a standard deviation of 50, and is cropped to the range [0, 255]. Applying Bayes rule for incorporating evidence e, gives:

p(M(i,j)={circumflex over (r)}|e)=αp(e|M(i,j)={circumflex over (r)})p(M(i,j)={circumflex over (r)}  (4)

where the probability density function p_li,rj(x) is used as evidence, and α is the normalization factor, resulting in:

p(M(i,j)={circumflex over (r)}|e)=αp_li,rj({circumflex over (r)})p(M(i,j)={circumflex over (r)}).  (5)

The calibration map is therefore:

$\begin{array}{cc}M\left(i,j\right)=\underset{\hat{r}}{\mathrm{argmax}}p_{l_i,r_j}\left(\hat{r}\right)p\left(M\left(i,j\right)=\hat{r}\right)& \left(6\right)\end{array}$

The normalization factor may be dropped as it is a constant for all i, j.

[0047] FIG. 6 is a flowchart for calibration map generation, according to an embodiment of the present disclosure.

[0048] In step S601, a laser l_i with intensity r_j are selected.

[0049] In step S603, a prior function p(M(i,j)={circumflex over (r)}) is defined as a Gaussian with mean and standard deviation σ.

[0050] In step S605, the prior p is cropped to the range of possible intensity values.

[0051] In step S607, the prior p is renormalized.

[0052] In step S609, the intensity value r that maximizes p_li,rj(r)p(M(i,j)=r) is determined.

[0053] FIGS. 8A-C show final density estimate p(M(i,j)={circumflex over (r)}|e) incorporating both prior and gathered evidence for three different beams. The x axis is the input intensity (as reported by the sensor), and the y axis is the output (corrected) intensity. The height corresponds to the likelihood. The final calibration map is generated directly from these probabilities.

[0054] The resulting data can be used in applications such as road-paint detection for lane markings, road-sign detection based on high-reflectivity returns, and the creation of intensity maps to be used for vehicle positioning and localization.

[0055] Next, a hardware description of the controller 209 according to exemplary embodiments is described with reference to FIG. 9. In FIG. 9, the controller 209 includes a CPU 900 which performs the processes described above. The process data and instructions may be stored in memory 902. These processes and instructions may also be stored on a storage medium disk 904 such as a hard drive (HDD) or portable storage medium or may be stored remotely. Further, the claimed advancements are not limited by the form of the computer-readable media on which the instructions of the inventive process are stored. For example, the instructions may be stored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other information processing device with which the controller 209 communicates, such as a server or computer.

[0056] Further, the claimed advancements may be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing in conjunction with CPU 900 and an operating system such as Microsoft Windows 7, UNIX, Solaris, LINUX, Apple MAC-OS and other systems known to those skilled in the art.

[0057] CPU 900 may be a Xenon or Core processor from Intel of America or an Opteron processor from AMD of America, or may be other processor types that would be recognized by one of ordinary skill in the art. Alternatively, CPU 900 may be implemented on an FPGA, ASIC, PLD or using discrete logic circuits, as one of ordinary skill in the art would recognize. Further, CPU 900 may be implemented as multiple processors cooperatively working in parallel to perform the instructions of the inventive processes described above.

[0058] The controller 209 in FIG. 9 also includes a network controller 906, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with network 999. As can be appreciated, the network 999 can be a public network, such as the Internet, or a private network such as an LAN or WAN network, or any combination thereof and can also include PSTN or ISDN sub-networks. The network 999 can also be wired, such as an Ethernet network, or can be wireless such as a cellular network including EDGE, 3G and 4G wireless cellular systems. The wireless network can also be WiFi, Bluetooth, or any other wireless form of communication that is known.

[0059] The controller 209 further includes a display controller 908, such as a NVIDIA GeForce GTX or Quadro graphics adaptor from NVIDIA Corporation of America for interfacing with display 910, such as a Hewlett Packard HPL2445w LCD monitor. A general purpose I/O interface 912 interfaces with a keyboard and/or mouse 914 as well as a touch screen panel 916 on or separate from display 910. General purpose I/O interface 912 also connects to a variety of peripherals 918 including printers and scanners, such as an OfficeJet or DeskJet from Hewlett Packard.

[0060] A sound controller 920 is also provided in the controller 209, such as Sound Blaster X-Fi Titanium from Creative, to interface with speakers/microphone 922 thereby providing sounds and/or music. The speakers/microphone 922 can also be used to accept dictated words as commands for controlling the controller 209 or for providing location and/or property information with respect to the target property.

[0061] The general purpose storage controller 924 connects the storage medium disk 904 with communication bus 926, which may be an ISA, EISA, VESA, PCI, or similar, for interconnecting all of the components of the controller 209. A description of the general features and functionality of the display 910, keyboard and/or mouse 914, as well as the display controller 908, storage controller 924, network controller 906, sound controller 920, and general purpose I/O interface 912 is omitted herein for brevity as these features are known.

[0062] The Lidar controller 928 connects the Lidar 930 with communication bus 926, so that the Lidar 930 may be controlled via CPU 900, and data collected by Lidar 930 may be stored in Disk 904. CPU 900 may use data collected by Lidar 930 to provide information via speaker 922 or display 910.

[0063] The positioning device controller 932 connects the positioning device 211 with communication bus 926, so that the positioning device 211 may be controlled via CPU 900, and position information may be stored in Disk 904 in association with the data collected by Lidar 930.

[0064] Initial calibration information or other settings for Lidar 930 may be provided to controller 209 via network 999, keyboard/mouse 914, touch screen 916, or peripherals 918.

[0065] In the above description, any processes, descriptions or blocks in flowcharts should be understood as representing modules, segments or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process, and alternate implementations are included within the scope of the exemplary embodiments of the present advancements in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending upon the functionality involved, as would be understood by those skilled in the art.

[0066] While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel methods, apparatuses and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods, apparatuses and systems described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.