DEVICE AND METHOD FOR MONITORING VEHICLE SURROUNDINGS

[0001] The present invention relates to a device and a method for monitoring vehicle surroundings, which enable a driver to recognize obstacles or the like present around a vehicle.

[0002] There has conventionally been known a driving support system which enables a driver to recognize a possibility of contact with an obstacle or a pedestrian during low-speed traveling such as moving-back or parking by monitoring the surroundings of his own vehicle through a camera to easily check dead angles of the vehicle surroundings which cannot be directly checked visually.

[0003] For this driving support system, various technologies have been proposed. An example is a rear-view camera device which arranges a camera in a car body rear part such as a rear trunk part, and displays a camera video imaged by the camera with a vehicle rear part set as an imaging direction on a display monitor of a dashboard to be viewed by the driver. Regarding this rear-view camera device, there has been available a type which overlay-displays a predicted track in a camera video according to steering angles of own vehicle front wheels so that a positional relation between a traveling direction of the own vehicle and an obstacle can be easily determined by the camera video. Conventionally, in addition to the system for visually supporting driving by displaying the camera video, there has been known a system which uses an ultrasonic sensor or the like for detecting an obstacle present around a vehicle by reflection of an ultrasonic wave to give a warning about presence of the obstacle, a positional relation between the own vehicle and the obstacle, or a distance from the own vehicle through a warning sound or a light emitted from a light emitting diode (LED).

[0004] Recently, as disclosed in Japanese Patent Application Laid-Open No. 2001-315603, a device has been proposed which uses an image synthesis technology to display a bird's-eye image looking down on an own vehicle from a virtual camera above the own vehicle, and displays a virtual object in a detection range of an obstacle detection sensor, a safe area having no obstacle according to a distance to an obstacle, or a road surface boundary of the object in this bird's-eye image. In Japanese Patent Application Laid-Open No. 2001-315603, a warning sound is issued when the obstacle is detected by the obstacle detection sensor.

[0005] However, as described above, when the bird's-eye image is displayed by combining the detection range of the obstacle detection sensor with the bird's-eye image, and the warning is simultaneously given, if the obstacle detected by the obstacle detection sensor is present in space away from a ground surface, irrespective of actual presence of the obstacle within the detection range of the obstacle detection sensor, the obstacle is displayed in a manner of not being present within the detection range of the obstacle detection sensor in the bird's-eye image, giving uncomfortable feelings.

[0006] The present invention has been proposed with the aforementioned situation in mind, and objects of the invention are to provide a device and a method for monitoring vehicle surroundings, which display, when an obstacle detection range is displayed in a view point converted image, the obstacle detection range without giving any uncomfortable feelings in a positional relation between the obstacle detection range and an obstacle.

[0007] A vehicle surrounding monitoring device of the present invention includes at least one camera installed in an own vehicle to image a video around the own vehicle, obstacle detection means installed in the own vehicle to detect an obstacle in an obstacle detection area, image processing means for converting a camera image imaged by the camera into a view point converted image seen from a virtual view point above the own vehicle, and display means for displaying the view point conversion image converted by the image processing means. To solve the aforementioned problem, a warning is given by a warning sound upon entry of an obstacle within an obstacle detection range of the obstacle detection means, and simultaneously the image processing means combines an image of the obstacle detection range of the obstacle detection means with an obstacle image which is imaged by the camera and included in the view point converted image so that the both images are to be in contact.

[0008] A vehicle surrounding monitoring method according to the present invention includes imaging a video around an own vehicle by a camera installed in the own vehicle, detecting an obstacle within an imaging range of the camera, and to solve the aforementioned problem, when presence of an obstacle is detected, giving a warning by a warning sound upon entry of the obstacle within an obstacle detection range of obstacle detection means, and simultaneously synthesizing an image of the obstacle detection range of the obstacle detection means to be in contact with an obstacle imaged by the camera and included in a view point converted image.

[0026] Below, the embodiments of the present invention will be described with reference to the drawings.

[0027] The present invention is applied to a vehicle surrounding monitoring device configured, for example, as shown in FIG. 1. In the vehicle surrounding monitoring device, at least one or more cameras 2-1, 2-2, . . . , 2-N (simply referred to as “camera 2” generically hereinafter), a plurality of obstacle detection sensors (obstacle detection means) 3-1, 3-2, . . . , 3-N (simply referred to as “obstacle sensor 3” generically hereinafter), and a display device 4 are connected to an image processing unit (image processing means) 1. The cameras 2-1, 2-2, . . . , 2-N are arranged in different positions of an own vehicle, and imaging directions are different from one another. As in the case of the camera 2, the obstacle sensors 3-1, 3-2, . . . , 3-3 are arranged in different positions of the own vehicle, and obstacle detection ranges are different from one another.

[0028] Specifically, as shown in FIG. 2, the cameras 2-1, 2-2, . . . , 2-N are, for example, super-wide-angle cameras where horizontal field angles are 180°, and include four cameras, i.e., a camera 2F for imaging a front side of the vehicle, a camera 2B for imaging a rear side, a camera 2R for imaging a right side, and a camera 2L for imaging a left side. Accordingly, the cameras 2F, 2B, 2R, and 2L include, by the image processing unit 1 described below, as in the case of a bird's-eye image (view point converted image) 100 subjected to image synthesis processing (view point conversion processing) to be displayed by the display unit 4 shown in FIG. 3, a front camera display area 101F, a rear camera display area 101B, a right camera display area 101R, and a left camera display area 101L, and can form images for understanding all the surroundings of the own vehicle.

[0029] The bird's-eye image 100 is an image subjected to view point conversion processing as if to look down the own vehicle from above by using camera image data imaged by the cameras 2F, 2B, 2R, and 2L shown in FIG. 2. An image drawn roughly in the center of the bird's-eye image is an own vehicle overlay image 102 having a size corresponding to a floor area of the own vehicle stored beforehand in an overlay image storage unit which is not shown. Black beltlike partition lines are arranged between the display areas of the bird's-eye image 100. Note that each display area can be allocated differently from the example according to an actual imaging area.

[0030] For the obstacle sensors 3-1, 3-2, . . . , 3-3, ranges (obstacle detection ranges) provided with widths of angle of about 90° to 140° from installation positions in fan shapes to detect obstacles present in a distance of about 1 m from the own vehicle are set. As shown in FIG. 2, the obstacle sensors 3-1, 3-2, . . . , 3-3 include four obstacle sensors, i.e., an obstacle sensor 3FR for detecting obstacles of front right direction, an obstacle sensor 3FL for detecting obstacles of front left direction, an obstacle sensor 3BR for detecting obstacles of rear right direction, and an obstacle sensor 3BL for detecting obstacles of rear left direction. These obstacle sensors 3FR, 3FL, 3BR, and 3BL are configured so that in FIG. 3, obstacles present in a front camera display area 101F can be detected by the obstacle sensors 3FR and 3FL, obstacles present in a rear camera display area 101B can be detected by the obstacle sensors 3BR and 3BL, obstacles present in a right camera display area 101R can be detected by the obstacle sensors 3FR and 3BR, and obstacles present in a left camera display area 101L can be detected by the obstacle sensors 3FL and 3BL.

[0031] As shown in FIG. 2, a case where the cameras 2-1, 2-2, . . . , 2-N consist of cameras 2F, 2B, 2R, and 2L respectively, and the obstacle sensors 3-1, 3-2, . . . , 3-3 consist of obstacle sensors 3FR, 3FL, 3BR, and 3BL respectively will be described below.

[0032] As shown in FIG. 1, the image processing unit 1 enters camera image data of the vehicle surroundings imaged by the cameras 2F, 2B, 2R, and 2L to frame buffers 11-1, 11-2, . . . , 11-N (simply referred to as “frame buffer 11” generically hereinafter) which are input buffers corresponding to the cameras 2F, 2B, 2R, and 2L to temporarily store the data. A plurality of camera image data stored in the frame buffer 11 are used for forming a bird's-eye image 100 by image synthesis processing of a pixel synthesis unit 13.

[0033] The image processing unit 1 enters sensor signals from the obstacle sensors 3FR, 3FL, 3BR, and 3BL to a single obstacle determination unit 12. The obstacle determination unit 12 calculates presence of an obstacle around the own vehicle, and a position and a distance when an obstacle is present based on each entered sensor signal. The obstacle determination unit 12 outputs the calculated presence of the obstacle, and the calculated position and distance of the obstacle to a table switching unit 14.

[0034] The table switching unit 14 switches a table (address conversion table) to be referred to in the image synthesis processing of the pixel synthesis unit 13 to one of a plurality of address conversion tables among the plurality of address conversion tables stored in a table storage unit 15.

[0035] In the address conversion table stored in the table storage unit 15, correspondence relationship between each pixel of the bird's-eye image 100 which is an output image output to the display device 4 and information for specifying image data to be stored in the pixel is described. Specifically, in the address conversion table, whether to store camera image data in each pixel of the bird's-eye image 100 which is the output image, whether to store an overlay image indicating the own vehicle or the obstacle, which of addresses in which of the frame buffers 11 image data is taken out from when the camera image data is stored, and which of pieces of color information is stored when the overlay image is stored are described. In other words, in the address conversion table, contents of image synthesis processing for which of the frame buffers or overlay image storage units not shown the camera image to form the bird's-eye image 100 is taken out from, and which of the address in the output buffer of a video signal generation unit 16 the taken-out camera image data or the overlay image is stored are described.

[0036] The pixel synthesis unit 13 refers to the address conversion table selected by the video signal generation unit 16 to take out the camera image data stored in the frame buffer 11 or the overlay image indicating the own vehicle or the obstacle, and stores the image in the output buffer of the video signal generation unit 16. Accordingly, the pixel synthesis unit 13 stores the bird's-eye image 100 obtained by synthesizing the plurality of camera image data imaged by the cameras 2F, 2B, 2R, and 2L in the video signal generation unit 16.

[0037] Upon storage of the bird's-eye image 100 of one frame in the output buffer by the pixel synthesis unit 13, the video signal generation unit 16 outputs the bird's-eye image 100 to the display device 4 to display the image therein. Output timing of the bird's-eye image 100 from the output buffer of the video signal generation unit 16 to the display device 4 has been preset in advance together with timing of the image synthesis processing of the pixel synthesis unit 13.

[0038] A warning output unit 17 for outputting, when presence of an obstacle within the obstacle detection range of the obstacle sensor 3 is detected by the obstacle determination unit 12, a warning sound indicating the detection is connected to the obstacle determination unit 12. This warning output unit 17 includes a speaker for outputting a predetermined warning sound to the driver.

[0039] Next, a specific example where the vehicle surrounding monitoring device configured in the aforementioned manner causes the display device 4 to display the bird's-eye image 100 will be described.

[0040] For example, as shown in FIGS. 4A and 4B, when a pole 110 is present as an obstacle in a rear right side of the own vehicle, the bird's-eye image 100 includes, as shown in FIG. 4C, an obstacle image 120 in the rear camera display area 101B below a right side of the own vehicle overlay image 102. As in the case of the pole 110, when the obstacle stands vertically in contact with the ground surface, in the bird's-eye image 100, a contact surface between the pole 110 and the ground surface can be displayed in a correct position with respect to the own vehicle. When the vehicle is moved back straight in a state of FIG. 4A, as shown in FIGS. 5A and 5B, contact of the own vehicle with the pole 110 can be understood. Additionally, as shown in FIG. 5C, the image processing unit 1 forms an obstacle image 120′ where the pole 110 of the rear camera display area 101B is displayed to be distorted so that the obstacle image 120′ can be displayed in contact with the own vehicle overlay image 102.

[0041] As shown in FIGS. 6A to 6C and FIGS. 7A to 7C, when the other vehicle 130 stopped in the position of the pole 110 is present in FIGS. 4A to 4C and FIGS. 5A to 5C, an obstacle present in a position closest to the own vehicle is a bumper of the other vehicle 130, and present at a predetermined height from the ground surface different from the pole 110.

[0042] When the obstacle is present above the ground surface as in the aforementioned case, depending on a positional relation with attaching heights of the cameras 2F, 2B, 2R, and 2L, a phenomenon of positional shifting, i.e., shifting of a positional relation between the own vehicle and the other vehicle 130 in the bird's-eye image 100, occurs. In other words, as shown in FIGS. 6A and 6B, when the own vehicle is moved back straight, irrespective of a positional relation of potential contact with the other vehicle 130, as shown in FIG. 6C, there is a possibility of forming a bird's-eye image 100 where the own vehicle is not in contact with the other vehicle 130. Even when a stage is changed from the state of FIGS. 6A and 6B, by straight movement-back of the own vehicle, to a state where the own vehicle is actually brought into contact with the other vehicle 130 as shown in FIGS. 7A and 7B, there is a possibility of forming a bird's-eye image 100 which includes an other vehicle image 140′ which is not in contact as shown in FIG. 7C.

[0043] The phenomenon of the shifting of the positional relation between the own vehicle and the other vehicle 130 in the bird's-eye image 100 is caused, as shown in FIG. 8, by a principle of generating a bird's-eye image 100 itself where depending on a relation between a ground height H1 of the camera 2 and a ground height H2 of the obstacle, as for the obstacle above the ground, a distance from the camera 2 to the obstacle is imaged as if the obstacle is present in a position more away from the camera 2 than that in an actual case. In other words, in reality, irrespective of only a distance D1 from the camera 2 to the obstacle, the obstacle is imaged at a distance D2 within a camera imaging range, and thus the bird's-eye image 100 includes the obstacle in the position of the distance D2 greater than the distance D1.

[0044] More specifically, a case where the other vehicle 130 is actually present as an obstacle within an obstacle detection range 3a of the obstacle sensor 3BR as shown in a diagram of FIG. 9A where the own vehicle is seen from above and a diagram of FIG. 9B where the own vehicle is seen from an oblique rear side will be considered. In this case, when an installation position of the camera 2 is higher than an installation position of the obstacle sensor 3, the obstacle detection range 3a imaged by the camera 2 includes, as shown in FIG. 10, not only an area 3b where the obstacle detection range 3a is projected on the ground surface but also an area 3c of a range passed from the installation position of the camera 2 through an end of the obstacle detection range 3a and projected on the ground surface.

[0045] Accordingly, in reality, even if the other vehicle 130 is present within the obstacle detection range 3a, even when an obstacle detection range image 150 indicating the area 3b where the obstacle detection range 3a is projected vertically on the ground surface is synthesized with the bird's-eye image 100 shown in FIG. 11A, as shown in FIG. 11B, in a bird's-eye image 100 imaged by the camera 2 and subjected to view point conversion, contact of an other vehicle image 140 within the obstacle detection range 3a cannot be presented.

[0046] On the other hand, in the case of the vehicle surrounding monitoring device to which the present invention is applied, when an obstacle enters the obstacle detection range 3a of the obstacle sensor 3, a warning sound is output from the warning output unit 17, an image of the obstacle detection range 3a of the obstacle sensor 3 is simultaneously synthesized with an image of an obstacle imaged by the camera 2 and included in the bird's-eye image 100 to be contact therewith, and the bird's-eye image 100 is displayed by the display device 4.

[0047] The image of the obstacle detection range 3a of the obstacle sensor 3 is, as shown in FIG. 10, an overlay image (referred to as obstacle detection range image hereinafter) indicating the area 3c of a range where the obstacle detection range 3a is projected on the ground surface with the attaching position of the camera 2 set as a view point when the obstacle detection range 3a is imaged by the camera 2.

[0048] Such a vehicle surrounding monitoring device directly displays the bird's-eye image 100 processed by the pixel synthesis unit 13 at a normal time when the obstacle determination unit 12 detects no obstacle, and synthesizes an obstacle detection range image with the bird's-eye image 100 to display it thereon when the obstacle determination unit 12 detects an obstacle. Accordingly, in the vehicle surrounding monitoring device, upon detection of the obstacle, if the obstacle is present above and thus not in contact with the ground surface, presence of the obstacle within the obstacle detection range 3a of the obstacle sensor 3 is recognized in the bird's-eye image 100 subjected to the view point conversion.

[0049] Specifically, in the case of the positional relation between the own vehicle and the other vehicle 130 shown in FIGS. 9A and 9B, when the obstacle determination unit 12 detects presence of an obstacle in the rear right side of the own vehicle, the bird's-eye image 100 shown in FIGS. 11A and 11B is switched to displaying shown in any one of FIGS. 12 to 14. When the obstacle determination unit 12 detects no more obstacle, the displaying is returned to the bird's-eye image 100 of FIGS. 11A and 11B.

[0050] A bird's eye image 100 shown in FIG. 12 is an image obtained by displaying an other vehicle image 140 which is an obstacle detected in the rear right side of the own vehicle in the rear camera display area 101B, and synthesizing semi-transparently an obstacle detection range image 151 indicating an area 3c of a range passed from the installation position of the camera 2 through the end of the obstacle detection range 3a and projected on the ground surface with a predetermined area in the rear side of the own vehicle.

[0051] In the case of executing such displaying, the obstacle determination unit 12 notifies the detection of the obstacle in the rear right side of the own vehicle to the table switching unit 14. The table switching unit 14 selects an address conversion table to display the obstacle detection range image 151 in a rear side of the own vehicle overlay image 102. Accordingly, the pixel synthesis unit 13 can form a bird's-eye image 100 where not the camera image data but the obstacle detection range image 151 is displayed in the rear side of the own vehicle overlay image 102. Simultaneously, the obstacle determination unit 12 operates the warning output unit 17 to give a warning about presence of the other vehicle 130 within the obstacle detection range 3a.

[0052] A bird's-eye image 100 shown in FIG. 13 is an image obtained by synthesizing a steric obstacle detection range image 152 with the obstacle detection range 3a of the obstacle sensor 3BR which has detected the other vehicle 130. This obstacle detection range image 152 is configured by setting an area a (area 3b of FIG. 10) where the obstacle detection range 3a is projected on the ground surface seen from above as a bottom surface, and connecting planes b and c corresponding to the area 3c shown in FIG. 10 to the area a of the bottom surface. These planes b and c are planes extending vertically from a boundary of the area where the obstacle detection range 3a of the obstacle sensor 3 is projected on the ground surface with the attaching position of the camera 2 as a view point.

[0053] An end point d of the obstacle detection range image 152 is, as shown in FIG. 10, a point passed from the installation position of the camera 2 through one upper end part of the obstacle detection range 3a and projected on the ground surface. An end point e of the obstacle detection range image 152 is, as shown in FIG. 10, a point passed from the installation position of the camera 2 through the other upper end part of the obstacle detection range 3a and projected on the ground surface. In this case, the table switching unit 14 selects an address conversion table for displaying the steric obstacle detection range image 152 in the rear side of the own vehicle overlay image 102. Steric surfaces b and c of the steric obstacle detection range image 152 are preferably displayed by different colors to facilitate differentiation of the steric surfaces b and c from each other.

[0054] Thus, by displaying the steric obstacle detection range image 152 as the image of the obstacle detection range 3a, presence of the other vehicle 130 within the obstacle detection range 3a of the obstacle sensor 3 in its separated state from the ground surface can be presented.

[0055] Additionally, a bird's-eye image 100 shown in FIG. 14 is an image obtained by synthesizing a guide line 153 in a position where the other vehicle image 140 present away from and above the ground surface is projected on the ground surface with the obstacle detection range image 150 of the area (area 3b of FIG. 10) where the obstacle detection range 3a is projected on the ground surface. The guide line 153 is a line where a car body part and a bumper part as ends of the other vehicle 130 are projected on the ground surface, and displayed in contact with the obstacle detection range image 150.

[0056] In this case, the table switching unit 14 selects an address conversion table for displaying the obstacle detection range image 150 in the rear side of the own vehicle overlay image 102. The pixel synthesis unit 13 forms a bird's-eye image 100 by synthesizing the guide line 153 with a position of a distance calculated by the obstacle determination unit 12 where a part of the other vehicle 130 away from the ground surface is present. The pixel synthesis unit 13 prestores distance coordinates from the obstacle sensor 3 in the bird's-eye image 100, and synthesizes the guide line 153 with the position of the distance to the other vehicle 130 calculated by the obstacle determination unit 12 in the bird's-eye image 100. Accordingly, the guide line 153 is displayed in a form where the other vehicle image 140 is projected on the ground surface.

[0057] Thus, the guide line 153 is synthesized with the bird's-eye image 100 in a form of being stacked on the obstacle detection range image 150, whereby the presence of the other vehicle 130 within the obstacle detection range 3a can be presented to the driver.

[0058] Furthermore, as another method for enabling understanding of shifting of a positional relation between the own vehicle overlay image 102 and the other vehicle image 140 in the bird's-eye image 100 from an actual positional relation between the own vehicle and the other vehicle 130, as shown in FIGS. 15 to 17, the display form of the own vehicle overlay image 102 drawn roughly in the center of the bird's-eye image 100 may be changed.

[0059] In a bird's-eye image 100 shown in FIG. 15, the other vehicle image 140 is displayed in a rear right side, overlay images 160A and 160B are formed in left and right sides of the own vehicle overlay image 102, and the overlay image 160B in a direction where the other vehicle image 140 is present is displayed by a warning color. The table switching unit 14 selects an address conversion table for displaying respectively the semitransparent overlay image 160A and the warning color overlay image 160B on both sides of the own vehicle overlay image 102. Accordingly, an area of the own vehicle overlay image 102 is set larger than a floor area of the own vehicle.

[0060] In a bird's-eye image 100 shown in FIG. 16, the own vehicle overlay image 102 is changed to a virtual own vehicle overlay image 102′ enlarged in a horizontal direction. In this case, the table switching unit 14 selects an address conversion table for displaying an own vehicle overlay image 102′ where a width of the own vehicle overlay image 102 is increased.

[0061] Furthermore, as shown in FIG. 17, the own vehicle overlay image 102 is enlarged horizontally and sterically to be changed to a virtual own vehicle overlay image 102′. In this case, the table switching unit 14 selects an address conversion table for increasing a width of the own vehicle overlay image 102, and displaying the steric own vehicle overlay image 102′.

[0062] Thus, when the obstacle is detected, by enlarging the own vehicle overlay image 102 of the size corresponding to the floor area of the own vehicle in the width direction, a positional relation between the own vehicle and the other vehicle image 140 which are brought into contact with each other when the own vehicle is moved back straight can be understood by the driver.

[0063] As shown in FIGS. 15 to 17, when the own vehicle overlay image 102 is enlarged to be displayed, by the warning output unit 17, the presence of the other vehicle 130 within the obstacle detection range 3a may be understood by a warning sound. Accordingly, in reality, there is a possibility of contact between the obstacle detected by the obstacle sensor 3 and the own vehicle, and this possibility can be surely presented to the driver.

[0064] The aforementioned embodiment is only an example of the present invention. The invention is not limited to the embodiment. Needless to say, even other embodiments can be employed, and various changes can be made according to design or the like without departing from the technical ideas of the invention.

[0065] In other words, in the aforementioned display example of the bird's-eye image 100, the obstacle detection range image is displayed in the semitransparent manner. Not limited to this, however, the obstacle detection range image may be flashed.

[0066] According to the present invention, when an obstacle is detected, a warning is given, and images are synthesized so that an image of the obstacle is brought into contact with the obstacle detection range. Thus, even when the obstacle is present away from the ground surface, and even when positional shifting of the obstacle occurs in a view point converted image, displaying can be realized where no uncomfortable feelings are given in a positional relation between the obstacle detection range and the obstacle.