Monocular computer vision aided road vehicle driving for safety

[0002] The present invention is a monocular computer vision technique. The technique is used to aid road vehicle driving for safety. A camera (e.g., camera or camcorder) is installed in a vehicle facing to the road in front of the vehicle to capture a sequence of road/scene images for analyzing the driving situation. The situation includes the distance to a front obstacle or vehicle, the speed of the vehicle, and the left/right location of the vehicle in a road lane. If a danger situation is arisen, the sound and/or light alarm will arise to warn the driver for safety.

[0003] Safe traffic is important to the property of an individual and the stability of human society. Driving a car out to work or travel is trivial nowadays. Sometimes, persons lose their attention during driving a car; especially, for a long-distance or a high-speed driving, many danger situations are then arisen. If there is a mechanism to alarm the driver and provide some useful information to judge by the driver in these situations, the danger is therefore avoided.

[0004] Several techniques have been proposed for the purposes of safe driving. In most previous proposed techniques, an ultrasonic range finder or laser radar was equipped on a car to detect the obstacles in front of the car. The equipments are expensive, have only a special function, and are not easily used. Other techniques had equipped a pair of cameras on a car and then utilized the stereo vision method to guide the navigation; however, this vision system is still complicated, working slowly, and expensive. If we can only equip a camera to detect the obstacles in front of the car and to guide the navigation, the vision system will be simpler, cheaper, and easy to practice for safe driving.

[0005] The left/right location of a vehicle in a road lane is also important to the driver during driving on a road; especially for that the driver can't concentrate his/her attention on the driving. To the detection of the left/right location in a lane, someone had proposed a method by equipping one camera on each side of the car to detect lines on both sides of the current lane; however, such equipments are still complicated and more expensive. It had better detect the left/right location of a vehicle in a road lane only using a camera.

[0006] The present invention is used for aiding road vehicle driving to improve the driving safety, to reduce the traffic accident, and to avoid the loss of lives and property. The invention technique only installs a camera in a vehicle facing to the front of the vehicle to capture a sequence of road/scene images, then utilizes the monocular computer vision method to acquire the driving situation. The situation includes the distance to a front obstacle or vehicle, the speed of the vehicle, and the left/right location of the vehicle in a road lane.

[0007] In order to attach the mentioned purposes, the present invention proposes a monocular computer vision technique to aid road vehicle driving for safety. The invention technique only installs a camera in a vehicle facing to the front of the vehicle and then processes the following steps:

[0008] i. using the camera to capture a sequence of road/scene images;

[0009] ii. processing and analyzing the sequence of images using the monocular computer vision method; and

[0010] iii. acquiring the driving situation for the vehicle based on the analysis results of the computer vision method.

[0011] As the described monocular computer vision technique for aiding road vehicle driving, the camera is a camera, TV camera, digital camera, camcorder, or digital camcorder. The capture device connects a computer and directly transmits the captured images or video into the computer for computer vision analysis.

[0012] As the described monocular computer vision technique for aiding road vehicle driving, the vehicle is a car, bus, or truck.

[0013] As the described monocular computer vision technique for aiding road vehicle driving, the camera is fixed on the front of a vehicle, the device is facing to the road in front of the vehicle, and the contents of the images are the road and scene in front of the vehicle.

[0014] As the described monocular computer vision technique for aiding road vehicle driving, the procedure for the monocular computer vision method contains:

[0015] i. detecting the distance to a front obstacle or vehicle by analyzing the image sequence;

[0016] ii. estimating the vehicle speed based on a continuous image sub-sequence of the sequence; and

[0017] iii. determining the left/right location of the vehicle in the current lane.

[0018] As the described monocular computer vision technique for aiding road vehicle driving, the detection of the distance to a front obstacle or vehicle consists of the following steps:

[0019] i. from each image to extract the lines on both sides of the current lane and then find the intersection of these two lines; the intersection point is just the vanishing point of the lane;

[0020] ii. from the vanishing point and the known focal length of the camera lens to find the pitch and yaw angles of the lane lines with respect to the camera coordinate system;

[0021] iii. from the pitch angle and the height of the camera to calculate the distance from the camera location to the point which is the intersection of the camera optical axis and the road plane;

[0022] iv. from each image to find horizontal lines to indicate the intersections of the rear wheels of front vehicles and the road, select a nearest horizontal line that overlaps the current lane, and judge that the horizontal line is located above or below the image center;

[0023] v. from the known camera focal length, camera height, pitch angle, and the vertical distance from the horizontal line to the image center to find the road distance from the camera location to the rear wheel of the front vehicle or obstacle.

[0024] As the described monocular computer vision technique for aiding road vehicle driving, the vehicle speed is computed based on the detection of the distance to a front vehicle. At first, a terminal point of a dashed lane line is found in any continuous image sub-sequence. For the images, the distances from the camera location to the terminal point are calculated respectively. The vehicle speed is just derived from the distance difference dividing by the time difference for the images.

[0025] As the described monocular computer vision technique for aiding road vehicle driving, the left/right location of the vehicle in a road lane is computed from the ratio of two distances which are from the midpoint of the image lower border to the intersection points of the extended image lower border and the two lines of the current lane.

[0026] As the described monocular computer vision technique for aiding road vehicle driving to acquire the driving situation, the situation includes the distance to a front obstacle or vehicle, the speed of the vehicle, and the left/right location of the vehicle in a road lane.

[0027] FIG. 1. The diagram of the invention vision system. (a) Side view. (b) Front view.

[0028] FIG. 2. The left-handed coordinate system and the camera coordinate system used in the proposal. (a) The left-handed coordinate system. (b) The camera coordinate system.

[0029] FIG. 3. The image center point and a vanishing point.

[0030] FIG. 4. The side view of the relationship between the camera and the road plane.

[0031] FIG. 5. The horizontal lines in an image.

[0032] FIG. 6. The side view of the relationship between the camera and image horizontal lines.

[0033] FIG. 7. The front view for indicating the right/left location of the vehicle in the current lane.

[0034] FIG. 8. The top view of a camera coordinate system in a lane.

[0035] The present invention installs an image capture device on the front of a vehicle as an artificial eye to capture a sequence of road/scene images for detecting the distance to a front obstacle or vehicle, estimating the speed of the vehicle, determining the left/right location of the vehicle in a road lane, and providing sound and/or light to warn the driver to aid the driver for safety.

[0036] FIG. 1 is the equipment diagram of the present invention. A vehicle 11 navigates on a road 13. A camera 12 is fixed near the center of the windshield and the upper border of the windshield; the camera faces to the road in front of the vehicle.

[0037] The theories and the formulas for the present invention techniques are detailedly described as follows:

[0038] A. Detecting the Distance to a Front Obstacle or Vehicle

[0039] We use the left-handed coordinate system to define the camera coordinate system and the world coordinate system as shown in FIG. 2. The rotation angle for the coordinate system is measured clockwise. The optical axis of the camera is defined as the z-axis of the camera coordinate system; the image is just the xy-plane of the camera coordinate system. The world coordinate system is just used for helping explanation of the proposed theory and we only care its orientation; thus only the direction of the coordinate system is defined. Assume the road being a plane. The world coordinate system is defined by the road plane and the lane lines of the current lane. Let the road plane be the XZ-plane of the world coordinate system and one lane line be the Z-axis of the world coordinate system.

[0040] Let (x_o, y_o) be the coordinates of the image center and (x_v, y_v) be the coordinates of the vanishing point of the current lane as shown in FIG. 3. According to the theory of vanishing point, the 3-dimensional direction of the lane lines with respect to the camera coordinate system is (x_v, y_v, q), where q is the focal length of the camera lens. If the vanishing point is not located on the central vertical line of the image, the camera must have a yaw rotation to the world coordinate system. The contained angle φ′ of Z-axis of the world coordinate system (i.e., the lane line) and x-axis of the camera coordinate system can be calculated by the formula $\begin{array}{cc}\cos {\phi }^{\prime }=\frac{\left[\begin{array}{ccc}1& 0& 0\end{array}\right]·\left[\begin{array}{ccc}{x}_v& y_v& q\end{array}\right]}{\sqrt{x_v^2+y_v^2+q^2}}.& \left(1\right)\end{array}$

[0041] The acute angle φ of the lane line and the yz-plane (i.e., the yaw angle of the camera coordinate system) is then calculated as $\begin{array}{cc}\phi =90°-{\cos }^{-1}\left(\frac{\mathrm{xv}}{\sqrt{x_v^2+y_v^2+q^2}}\right).& \left(2\right)\end{array}$

[0042] The positive φ means that the camera coordinate system turns left and the negative φ means to turn right. The component of the lane line on the yz-plane is [0 Y_vq], Thus the acute angle of the lane line and the camera optical axis (i.e., z-axis) on the yz-plane is θ, $\begin{array}{cc}\cos \theta =\frac{\left[\begin{array}{ccc}0& y_v& q\end{array}\right]·\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}{\sqrt{y_v^2+q^2}}\Rightarrow \theta ={\cos }^{-1}\left(\frac{q}{\sqrt{y_v^2+q^2}}\right).& \left(3\right)\end{array}$

[0043] From Eqs.(2) and (3), we can find the pitch angle θ and yaw angle φ of the lane lines with respect to the camera coordinate system. There are something in the formulas should be cared: the units of the coordinates [x_v, y_v] and q must be the same; the pixels in images may be not square pixels; thus the image should be calibrated before it is used for analysis.

[0044] The location under the camera and on the road is called the camera location. Assume the distance from the camera center to the camera location (i.e., the height of the camera) being h as shown in FIG. 4. If the camera location to the intersection point of the optical axis and the road plane is d, then by the relation of the right triangle, we have $\begin{array}{cc}\frac{h}{d}=\tan \theta \Rightarrow d=h\cot \theta .& \left(4\right)\end{array}$

[0045] Now we want to compute the distance from the camera location to the road position at which a horizontal line in the image indicates; examples are shown in FIG. 5.

[0046] Assume that a horizontal line is located below the image center with vertical distance a₁as shown in FIG. 6. The road position of the horizontal line to the camera location is d₁, the view angle with respect to the optical axis is δ₁; that is the view angle with respect to the road plane is θ₁,

[0047] θ₁−θ=δ₁.  (5)

[0048] According to the relationship of a right triangle, δ₁can be computed for the formula $\begin{array}{cc}\frac{a_1}{q}=\tan {\delta }_1\Rightarrow {\delta }_1={\tan }^{-1}\left(\frac{a_1}{q}\right).& \left(6\right)\end{array}$

[0049] Besides, from FIG. 6, we have

[0050] d₁=h cot θ₁.  (7)

[0051] If we substitute Eqs.(5) and (6) into Eq.(7), we have $\begin{array}{cc}{d}_1=h\cot {\theta }_1=h\cot \left(\theta +{\delta }_1\right)=h\cot \left(\theta +{\tan }^{-1}\left(\frac{a_1}{q}\right)\right).& \left(8\right)\end{array}$

[0052] In other words, the distance from the camera location to a road position at which a horizontal line in the image indicates can be computed from known camera height h, focal length q of the camera lens, the view angle θ of the optical axis with respect to the road plane, and the distance a₁from the image center to the horizontal line. If the horizontal line indicates the intersection of the rear wheel of a front vehicle and the road plane, then we just find the distance to the front vehicle.

[0053] We can also derive other formulas to compute the distance to a front vehicle. The derivation is started from Eq.(6) $\begin{array}{cc}\frac{a_1}{q}=\tan {\delta }_1\Rightarrow \cot {\delta }_1=\frac{q}{a_1}\Rightarrow \cot \left({\theta }_1-\theta \right)=\frac{q}{a_1}.& \left(9\right)\end{array}$

[0054] From Eqs.(4) and (7), we have $\begin{array}{cc}\begin{array}{c}d-d_1=h\left(\cot \theta -\cot {\theta }_1\right)=h\frac{\cot {\mathrm{\theta cot\theta }}_1+1}{\cot \left({\theta }_1-\theta \right)}\\ \Rightarrow d-d_1=h\frac{\cot \mathrm{\theta cot}{\theta }_1+1}{\frac{q}{a_1}}=\frac{a_{1}h}{q}\left(\cot \mathrm{\theta cot}{\theta }_1+1\right)\\ \Rightarrow d_1=d-\frac{a_{1}h}{q}\left(\cot \theta \cot {\theta }_1+1\right).\end{array}& \left(10\right)\end{array}$

[0055] That is, the distance from the camera location to a road position at which a horizontal line in the image indicates can be computed from known camera height h, focal length q of the camera lens, the view angle θ of the optical axis with respect to the road plane, the distance a₁from the image center to the horizontal line, the pitch angle θ of the camera, and the distance d from the camera location to the intersection point of the optical axis and the road plane.

[0056] With the same principle, a horizontal line is located above the image center with vertical distance a₂as shown in FIG. 6 and the road position of the horizontal line to the camera location is d₂, then the view angles have the relation

[0057] θ−θ₂=δ₂.  (11)

[0058] According to the relationship of the right triangle, view angle δ₂can be calculated by $\begin{array}{cc}\frac{a_2}{q}=\tan {\delta }_2\Rightarrow {\delta }_2={\tan }^{-1}\left(\frac{a_2}{q}\right).& \left(12\right)\end{array}$

[0059] Moreover, we have

[0060] d₂=h cot θ₂.  (13)

[0061] If we substitute Eqs.(11) and (12) into Eq.(13), we have $\begin{array}{cc}{d}_2=h\cot {\theta }_2=h\cot \left(\theta -{\delta }_2\right)=h\cot \left(\theta -{\tan }^{-1}\left(\frac{a_2}{q}\right)\right).& \left(14\right)\end{array}$

[0062] With other derivation from Eq.(12), we have $\begin{array}{cc}\frac{a_2}{q}=\tan {\delta }_2\Rightarrow \cot {\delta }_2=\frac{q}{a_2}\Rightarrow \cot \left(\theta -{\theta }_2\right)=\frac{q}{a_2}.& \left(15\right)\end{array}$

[0063] From Eqs.(4) and (13), we have $\begin{array}{cc}{d}_2-d=h\left(\cot {\theta }_2-\cot \theta \right)=h\frac{\cot \mathrm{\theta cot}{\theta }_2+1}{\cot \left(\theta -{\theta }_2\right)}\Rightarrow d_2-d=h\frac{\cot \mathrm{\theta cot}{\theta }_2+1}{\frac{q}{a_2}}=\frac{a_{2}h}{q}\left(\cot \mathrm{\theta cot}{\theta }_2+1\right)\Rightarrow d_2=d+\frac{a_{2}h}{q}\left(\cot \mathrm{\theta cot}{\theta }_2+1\right).& \left(16\right)\end{array}$

[0064] The process steps based on the proposed theory are described as follows:

[0065] i. from an image to find a horizontal line to represent the touch location of the rear wheel of a front vehicle in the current lane and judge the line is located above or below the image center;

[0066] ii. extract the lines of the current lane and find the vanishing point (x_v, y_v);

[0067] iii. from the vanishing point and the focal length q of the camera lens to find the pitch angle θ and yaw angle φ of the lane with respect to the camera coordinate system;

[0068] iv. from the pitch angle θ and the known camera height h to find the distance d using Eq:(4);

[0069] v. find the distance a₁(or a₂) from image center to the horizontal line, then calculate the front-vehicle distance d₁(or d₂) using Eq.(8) (or Eq.(14)).

[0070] B. Estimating the Vehicle Speed

[0071] We can use the above “detecting the distance to a front obstacle or vehicle” method to detect a terminal point of a dashed lane line of the current lane in a continuous image sub-sequence as one example shown in FIG. 5. For the images, the distances from the camera location to the terminal point are calculated respectively. The vehicle speed is just derived from the distance difference dividing by the time difference for the images.

[0072] C. Determining the Left/Right Location of the Vehicle in a Road Lane

[0073] If the optical axis of the camera is on the YZ-plane of the world coordinate system, then the vanishing point of the current lane must locate on the vertical central line of the image. The factor is independent to the left/right shift of the camera. In other words, only the camera coordinate system having a yaw rotation with respect to the world coordinate system can result in the factor that the vanishing point is not located on the vertical central line of the image.

[0074] Now we use an image as shown in FIG. 7 and a top-view diagram as shown in FIG. 8 to describe the relationship of the yaw rotation of the camera coordinate system and the lane lines. If c is the midpoint of the image lower border shown in FIG. 7; c is also shown in the current lane in the top-view diagram. In the image, we draw a line to link point c and the vanishing point; the drawn line is just the vehicle location line shown in FIG. 8. From the viewpoint of the 3-dimensional geometric meanings, the line is parallel to the lane lines. In the image, we can compute a distance ratio on any horizontal line located below the image center. We track a horizontal line from the vehicle location line to both lane lines to get two distances. The ratio of these two distances is invariant to the location of the horizontal line and the ratio is the same as the ratio of point c to both lane lines, L/R as shown in FIG. 7; moreover, the ratio is also the same as the ratio l/r shown in FIG. 8. Ratio l/r is just the distance ratio of the camera to the lane lines; ratio l/r is independent to the yaw rotation of the camera coordinate system. From the ratio, we can obtain the right/left location of the vehicle in the current lane.

[0075] If we know the vehicle width, we can judge whether the vehicle has deviated from the current lane and driven into the neighboring lane or has not. Assume that the lane width is R_w, the vehicle width is V_w, and the camera is fixed at the center of the windshield. If the ratio l/r is less than V_w/(2R_w−V_w) or greater than (2R_w−V_w)/V_w, then the vehicle has deviated from the current lane.

[0076] As described about, the present invention is a monocular computer vision technique to aid road vehicle driving for safety. The technique uses a camera which is installed in a vehicle and faces to the road in front of the vehicle to capture a sequence of road/scene images to acquire the distance to a front obstacle or vehicle, the speed of the vehicle, and the left/right location of the vehicle in a road lane. If a danger situation arises, the sound and/or light alarm will arise to warn the driver for safety. The present invention utilizes the monocular computer vision technique to reduce the cost and provide multiple safety functions. The present technique is an advanced, practical, and novel invention. The above description is not intended to be limiting, however, the invention instead being defined by the appended claims.