Visual vehicle-positioning fusion system and method thereof

[0001] The present invention relates to a vehicle-positioning system and a method thereof, particularly to a visual vehicle-positioning fusion system and a method thereof.

[0002] With technological advancements, obtaining precise location information on the road has become a fundamental requirement for autonomous vehicles. In addition to the continuous development of positioning technology, related industries have emerged, including intelligent transportation and logistics. The conventional positioning technology includes global positioning system (GPS), inertial measurement unit (IMU), light detection and ranging (LiDAR), and cameras. However, the existing positioning technology has technical limitations and shortcomings. For example, the stability of the GPS can be easily affected by environmental interferences. The GPS may be shield in urban environments with clusters of buildings or in environments like tunnels. In adverse weather conditions, the signal may fail to be received properly. The IMUs may accumulate errors over time. LiDAR is associated with higher costs and prices and susceptible to weather conditions. For instance, in weather conditions such as heavy rain, snow, or dense fog, it is difficult for the LiDAR to position accurately. Although cameras are the mainstream sensors for autonomous driving functions, they are susceptible to weather and lighting conditions, easily leading to misjudgments.

[0003] In other words, relying on a single type of positioning sensor is susceptible to changes in the surrounding environment, thereby affecting reliability. Depending merely on a single type of positioning sensor often leads to misjudgments or inaccurate analyses in positioning sensing, failing to meet current usage demands. Furthermore, a single type of positioning sensor has its technical limitations, which means that autonomous driving functions may not cover everything, thereby potentially increasing risks and leading to accidents.

[0004] Accordingly, the present invention provides a visual vehicle-positioning fusion system and a method thereof to overcome the abovementioned problems.

[0005] The primary objective of the present invention is to provide a visual vehicle-positioning fusion system and a method thereof, which improve the accuracy and reliability of positioning vehicles and avoid using positioning sensors with high prices, thereby reducing the overall cost and improving industrial applications.

[0006] In order to achieve the foregoing objectives, the present invention provides a visual vehicle-positioning fusion system, which is suitable for the on-board system of a vehicle. The visual vehicle-positioning fusion system includes a storage device, a positioning device, and a processing device. The storage device is configured to store an image point cloud map. The positioning device is arranged on the vehicle and configured to receive the longitude and latitude measurement coordinates of the vehicle. The processing device is configured to convert the image point cloud map into a longitude and latitude database corresponding to the image point cloud map, use the longitude and latitude measurement coordinates as first location information, compare the longitude and latitude database with the longitude and latitude measurement coordinates to generate the initial location of the vehicle, perform feature matching on the initial location and the image point cloud map to generate second location information of the vehicle on the image point cloud map, and select and output the first location information or the second location information based on a positioning fusion rule.

[0007] In an embodiment, the visual vehicle-positioning fusion system further includes an image capturing device and a real-time dynamic sensing device. The image capturing device is arranged on the vehicle and configured to generate driving images. The real-time dynamic sensing device is arranged on the vehicle and configured to generate the real-time dynamic information of the vehicle. The processing device is configured to fuse the driving images and the real-time dynamic information to generate real-time images. The storage device has an image database that is configured to store the real-time images consecutively generated.

[0008] In an embodiment, the visual vehicle-positioning fusion system further includes an inertial measurement device is arranged on the vehicle and configured to generate the inertial measurement information of the vehicle. The processing device is configured to access the real-time images in the image database, respectively fuse each of the real-time images with the inertial measurement information, respectively extract a plurality of feature points of each of the real-time images, and perform local mapping and loop closure detection based on the plurality of feature points and the inertial measurement information to construct the image point cloud map.

[0009] In order to achieve the objectives of the present invention, the present invention provides a visual vehicle-positioning fusion method that is performed by a processing device. The visual vehicle-positioning fusion method includes: converting an image point cloud map stored in a storage device into a longitude and latitude database corresponding to the image point cloud map; using longitude and latitude measurement coordinates received by a positioning device as first location information; comparing the longitude and latitude database with the longitude and latitude measurement coordinates to generate the initial location of a vehicle; performing feature matching on the initial location and the image point cloud map to generate the second location information of the vehicle on the image point cloud map; and selecting and outputting the first location information or the second location information based on a positioning fusion rule.

[0010] To sum up, the visual vehicle-positioning fusion system and the method thereof of the present invention can improve the accuracy and reliability of the autonomous driving system, employ the global satellite positioning system to obtain low-precision longitude and latitude, and quickly find the high-precision initial location of the vehicle, so that the need for a single expensive sensor is eliminated to reduce costs and increase accuracy.

[0011] Below, embodiments are described in detail in cooperation with the attached drawings to make easily understood the characteristics and accomplishments of the present invention.

[0020] The present invention will be described in detail with embodiments and attached drawings below. Similar or identical elements are denoted with similar or identical symbols in the drawings. In the drawings, the shapes or thicknesses may be exaggerated intentionally to make presentation simpler and labeling easier. The elements that are well known by the persons skilled in the art are not necessarily described in the specification or depicted in the drawings. Further, the persons having ordinary knowledge in the field should be able to make modification or variation according to the specification of the present invention. However, it should be noted: any modification or variation made according to the spirit or technical contents disclosed in the specification of the present invention is to be also included by the scope of the present invention.

[0021] Please refer to FIG. 1. In a first embodiment, a visual vehicle-positioning fusion system 1A includes a storage device 10, a positioning device 20, and a processing device 30. The storage device 10 stores the image point cloud map 102 of a vehicle. The positioning device 20 is arranged on the vehicle and configured to receive the longitude and latitude measurement coordinates Pg of the vehicle. The processing device 30 converts the image point cloud map 102 into a longitude and latitude database 104 corresponding to the image point cloud map 102. The processing device 30 uses the longitude and latitude measurement coordinates Pg as first location information P1. The processing device 30 compares the longitude and latitude database 104 with the longitude and latitude measurement coordinates Pg to generate the initial location P₀ of the vehicle. The processing device 30 performs feature matching on the initial location P₀ and the image point cloud map 102 to generate the second location information P2 of the vehicle on the image point cloud map 102. The processing device 30 selects and outputs the first location information P1 or the second location information P2 based on a positioning fusion rule. The storage device 10 includes a longitude and latitude database 104 corresponding to the image point cloud map 102. The image point cloud map 102 includes a plurality of point cloud map data P₀ int. When the plurality of point cloud map data P₀ int are stored, a plurality of corresponding location coordinate points Pt are synchronously recorded. The processing device 30 transfers the location coordinate points Pt corresponding to the plurality of point cloud map data P₀ int into the longitude and latitude database 104, as shown in FIG. 2. The positioning device 20 may receive the global positioning system (GPS) signal of a global navigation satellite system (GNSS) to obtain the location information of the vehicle, such as a longitude, a latitude, an altitude, etc.

[0022] Please refer to FIG. 1 and FIG. 2. When the processing device 30 obtains the present longitude and latitude measurement coordinates Pg, the processing device 30 compares the location coordinate points Pt in the longitude and latitude database 104 with the longitude and latitude measurement coordinates Pg and queries the longitude and latitude database 104 to generate the initial location P₀ of the vehicle. For example, the positioning device 20 obtains the present longitude and latitude measurement coordinates Pg of the vehicle, which is represented by (N23.42955° E120.92491°). The processing device 30 queries the location coordinate points Pt (i.e., from serial number T1 to serial number T8) in the longitude and latitude database 104 and obtains the location coordinate points Pt, which is closest to the present longitude and latitude measurement coordinates Pg and used as the initial location P₀. The initial location P₀ is expressed by equation (1).

[0023] $\begin{array}{cc}{P}_0=\min \left(P_t,P_g\right)& \left(1\right)\end{array}$

[0024] As a result, the processing device 30 queries the longitude and latitude database 104 to obtain the initial location P₀ corresponding to serial number T4 (N23.42956° E120.92492°). According to the low-precision longitude and latitude measurement coordinates Pg, the high-precision initial location P₀ of the vehicle may be rapidly looked up.

[0025] Please refer to FIG. 1, FIG. 3A, FIG. 3B, and FIG. 4. The processing device 30 selects and outputs the first location information P1 or the second location information P2 as positioning output information P based on the positioning fusion rule, which is introduced as follows.

[0026] Firstly, the processing device 30 fuses the first location information P1 and the second location information P2 to obtain an estimated reference location Pr. As shown in FIG. 4, the processing device 30 may use a Kalman filter 32 and inputs the first location information P1 and the second location information P2 to the Kalman filter 32 to generate the estimated reference location Pr. Furthermore, the processing device 30 calculates a first difference value D1 between the estimated reference location Pr and the first location information P1 and calculates a second difference value D2 between the estimated reference location Pr and the second location information P2. The first difference value D1 is a difference between the first location information P1 and the estimated reference location Pr (i.e., the distance between P1 and Pr). The second difference value D2 is a difference between the second location information P2 and the estimated reference location Pr (i.e., the distance between P2 and Pr).

[0027] The positioning fusion rule defines to use the first location information P1 outputted by the processing device 30 as the positioning output information P when the first difference value D1 less than the second difference value D2 and defines to use the second location information P2 outputted by the processing device 30 as the positioning output information P when the second difference value D2 less than the first difference value D1. That is to say, the processing device 30 selects one of the first location information P1 and the second location information P2 as the positioning output information P. A difference between the estimated reference location Pr and one of the first location information P1 and the second location information P2 selected by the processing device 30 is minimal.

[0028] During the travel of the vehicle, the vehicle may not necessarily be able to move forward smoothly at constant speed. Therefore, the probability density function is used to represent the estimated possible location of the vehicle at a time point. By measuring the obtained location, it can assist in confirming the location of the vehicle. FIG. 3A shows a first waveform w1, a second waveform w2, and a third waveform w3. The first waveform w1 represents the estimated reference location Pr. The second waveform w2 represents the longitude and latitude measurement coordinates Pg, namely the first location information P1. The third waveform w3 represents the second location information P2. As shown in FIG. 3B, when the processing device calculates the first difference value D1 and the second difference value D2, the processing 30 may determines that D1<D2. As a result, the processing device 30 outputs the first location information P1 as the positioning output information P based on the positioning fusion rule.

[0029] Please refer to FIG. 5. The visual vehicle-positioning fusion system 1B of the present invention is used to construct the image point cloud map. The visual vehicle-positioning fusion system 1B is different from the visual vehicle-positioning fusion system 1A of the first embodiment in that the visual vehicle-positioning fusion system 1B includes an image capturing device 40, a real-time sensing device 50, and an inertial measurement device 60. The storage device 10 further includes an image database 106. The image capturing device 40 arranged on the vehicle and configured to generate driving images Img of the vehicle. The real-time sensing device 50 uses real time kinematic technology. The real-time sensing device 50 is arranged on the vehicle and configured to generate real-time dynamic information Prtk of the vehicle. The processing device 30 fuses the driving images Img and the real-time dynamic information Prtk to generate real-time images RT-Img. The image database 106 is configured to store the real-time images RT-Img consecutively generated.

[0030] The visual synchronous positioning technology is used to generate the synchronous data of the real-time images RT-Img by align the real-time dynamic information Prtk to the driving images Img on the timeline. The frames of the consecutive real-time images RT-Img stored in the image database 106 in order of capturing image.

[0031] The inertial measurement device 60 has an inertial measurement unit (IMU). The inertial measurement device 60 is arranged on the vehicle and configured to generate the inertial measurement information IM of the vehicle. The inertial measurement information IM includes angle velocity information and acceleration information.

[0032] The processing device 30 accesses the real-time images RT-Img in the image database 106, respectively fuses each of the real-time images RT-Img with the inertial measurement information IM, respectively extracts a plurality of feature points of each of the real-time images RT-Img, and performs local mapping and loop closure detection based on the plurality of feature points and the inertial measurement information IM to construct the image point cloud map 102.

[0033] The visual simultaneous localizing and mapping (VSLAM) technology is used to construct the image point cloud map 102. Based on feature extraction algorithms like oriented features from accelerated segment test (FAST) and rotated binary robust independent elementary features (BRIEF) abbreviated as Oriented FAST and Rotated BRIEF (ORB), map points are constructed. The map is constructed using the established image database 106. The processing device 30 extracts the feature points of each of the real-time images RT-Img using the following methods: (1) extraction of descriptors such as ORB for each feature point; (2) feature matching; (3) detection of keyframes; and (4) interpolation of keyframes. The processing device 30 performs local mapping using the following methods: (1) interpolation of keyframes into the map and (2) removal of map points. The processing device 30 performs loop closure detection using the following methods: (1) determining whether there is a keyframe; (2) updating keyframes in a covisibility graph; and (3) fusing repeated map points.

[0034] Please refer to FIG. 5 and FIG. 6. Scale restoration is further performed in order to construct the image point cloud map 102. Using linear transformation, a local coordinate is converted into a latitude and a longitude. Scale restoration is expressed as follows. The processing device 30 selects the local coordinate P_l of the image point cloud map 102. The real-time dynamic information Prtk has a longitude and latitude trajectory Trtk. The processing device 30 selects the reference longitude and latitude coordinate P_v of the longitude and latitude trajectory Trtk, which corresponds to the local coordinate P_l. The processing device 30 calculates scale conversion parameters k_A and k_b based on the local coordinate P_l and the reference longitude and latitude coordinate P_v, as represented in equation (2).

[0035] $\begin{array}{cc}{P}_v=k_A{P}_l+k_b& \left(2\right)\end{array}$

[0036] In equation (2), the scale conversion parameters k_A and k_b are represented by

[0037] $k_A=\left[\begin{array}{cc}{a}_{11}& -a_{21}\\ a_{21}& a_{11}\end{array}\right]\mathrm{and}{k}_b=\left[\begin{array}{c}{b}_1\\ b_2\end{array}\right].$

[0038] As shown in FIG. 6, the processing device 30 obtains the local coordinates P_l1 and P_l2 and the corresponding the reference longitude and latitude coordinates P_v1 and P_v2 and interpolates the local coordinates P_l1 and P_l2 and the corresponding the reference longitude and latitude coordinates P_v1 and P_v2 into equation (2) to calculate the scale conversion parameters k_A (having coefficients a₁₁ and a₂₁) and k_b (having coefficients b₁ and b₂). Finally, the processing device 30 performs feature matching on the initial location P₀ and the image point cloud map 102 based on the scale conversion parameters k_A and k_b to generate the second location information P2.

[0039] Please refer to FIG. 7. The visual vehicle-positioning fusion system 1C includes a positioning device 20, a processing device 30, an image capturing device 40′ and an inertial measurement device 60. The processing device 30 includes a location identifying module 302 and a positioning fusion module 304. The recognition module 302 receives the real-time images RT-Img and inertial measurement information IM outputted by the image capturing device 40′ and the inertial measurement device 60 and extracts the feature points of the real-time images RT-Img to construct the map points and the image point cloud map. The location identifying module 302 performs matching on the feature points of the real-time images RT-Img and the constructed image point cloud map to obtain the local coordinate of the vehicle. Using linear transformation, the location identifying module 302 converts the local coordinate into longitude and latitude information, namely the reference longitude and latitude coordinate. The positioning fusion module 304 may determine the positioning output information P based on a voting rule. The positioning fusion module 304 uses a Kalman filter to estimate the optimal location information, namely the estimated reference location. Then, the positioning fusion module 304 compares the estimated reference location to the measurement values (i.e., the first location information P1 and the second location information P2) of the sensors (i.e., the positioning device 20, the image capturing device 40′, and the inertial measurement device 60) to determine the final positioning output information P.

[0040] The location identifying module 302 performs image recognition and map construction. The positioning fusion module 304 performs positioning fusion and sensing fusion. The location identifying module 302 and the positioning fusion module 304 may independently operate. Alternatively, the location identifying module 302 and the positioning fusion module 304 may be integrated in the same system, but the present invention is not limited thereto.

[0041] Although the concept of the visual vehicle-positioning fusion method has been explained concurrently with the visual vehicle-positioning fusion system of the present invention. FIGS. 8-11 show the flowcharts of the visual vehicle-positioning fusion method for clarity.

[0042] Please refer to FIG. 8. The visual vehicle-positioning fusion method, performed by the processing device, includes the following steps:

[0043] In Step S11, an image point cloud map stored in a storage device is converted into a longitude and latitude database corresponding to the image point cloud map.

[0044] In Step S12, longitude and latitude measurement coordinates received by a positioning device are used as first location information.

[0045] In Step S13, the longitude and latitude database is compared with the longitude and latitude measurement coordinates to generate the initial location of a vehicle.

[0046] In Step S14, feature matching is performed on the initial location and the image point cloud map to generate the second location information of the vehicle on the image point cloud map.

[0047] In Step S15, the first location information or the second location information is selected and outputted to serve as the final positioning output information based on a positioning fusion rule.

[0048] Please refer to FIG. 8 and FIG. 9. Step S15 may include the following steps:

[0049] In Step S151, the first location information and the second location information are fused to obtain an estimated reference location.

[0050] In Step S152, a first difference value between the estimated reference location and the first location information is calculated and a second difference value between the estimated reference location and the second location information is calculated.

[0051] In Step S153, the process determines whether the first difference value is less than the second difference value. If the result is yes, the process proceeds to Step S154. In Step S154, the first location information as the positioning output information is outputted. If the result is no, the process proceeds to Step S155. In Step S155, the second location information as the positioning output information is outputted.

[0052] Please refer to FIG. 8 and FIG. 10. The visual vehicle-positioning fusion method may further include the step of constructing the image point cloud map before Step S11 in FIG. 8. The step of constructing the image point cloud map may include the following steps:

[0053] In Step S101, driving images generated by an image capturing device and real-time dynamic information generated by a real-time dynamic sensing device are fused to generate real-time images.

[0054] In Step S102, the real-time images consecutively generated are stored into the image database of the storage device.

[0055] In Step S103, the real-time images are accessed in the image database.

[0056] In Step S104, each of the real-time images is respectively fused with the inertial measurement information of the vehicle generated by an inertial measurement device.

[0057] In Step S105, a plurality of feature points of each of the real-time images are respectively extracted and local mapping and loop closure detection are performed based on the plurality of feature points and the inertial measurement information to construct the image point cloud map.

[0058] Please refer to FIG. 8 and FIG. 11. Step S14 in FIG. 8 may further include the following steps:

[0059] In Step S141, the local coordinate of the image point cloud map is selected and the reference longitude and latitude coordinate of the longitude and latitude trajectory of the real-time dynamic information corresponding to the local coordinate is selected.

[0060] In Step S142, scale conversion parameters are calculated based on the local coordinate and the reference longitude and latitude coordinate.

[0061] In Step S143, feature matching is performed on the initial location and the image point cloud map based on the scale conversion parameters to generate the second location information.

[0062] In conclusion, the visual vehicle-positioning fusion system and the method thereof of the present invention can construct the image point cloud map, convert the image point cloud map into the longitude and latitude database, retrieve the longitude and the latitude (e.g., a first location) of a global positioning system (GPS) on the vehicle, and compares the first location with the longitude and latitude database to rapidly obtain the present rough location of the vehicle. After obtaining the present rough location of the vehicle, feature matching is performed to obtain the present precise location (e.g., a second location) of the vehicle on the image point cloud map. Finally, the first location or the second location is determined and outputted based on the voting rule. This way, the present invention can improve the accuracy and reliability of the autonomous driving system so that the need for a single expensive sensor is eliminated to reduce costs.

[0063] The embodiments described above are only to exemplify the present invention but not to limit the scope of the present invention. Any equivalent modification or variation according to the shapes, structures, characteristics and spirit disclosed in the specification is to be also included by the scope of the present invention.