Method for the displaying information in a motor vehicle

[0002] The present invention concerns a method and apparatus for displaying information in a motor vehicle.

[0003] In present-day motor vehicles, absolutely no automated functions exist to support the driver in recognizing road signs. Such a function has, however, a high comfort and safety value, for example, in warning the driver of speed limits or prohibitions against passing and/or in informing the driver of temporarily applicable limitations to his request. As a practical matter, with the existing traffic infrastructure, road signs can only be recognized and analyzed with the aid of image sensors since the information is only available for analysis in optical form. Owing to the high comfort and safety relevance of road sign recognition, work is being done on video-based road sign recognition. All of the known approaches are based on a limited number of road signs to be recognized. Based on the number of signs to be recognized, the image recorded by the camera undergoes image segmenting to recognize and classify the relevant road signs. With the appropriate classification, the information in the road sign is obtained, for example, a 70 km/h speed limit. The classified road sign information is reported to the driver via an appropriate display area in which, for example, the number combination “70” is displayed. The method of analyzing the information in the image obtained by the sensor for road signs and displaying the information to the driver may be problematic because the classification must be made very reliably. Otherwise it cannot be ruled out that the driver may be given incorrect information. It is believed that incorrect classifications may not only reduce the acceptance of such a system, as well as its reliabilty. If for example, a driver is supplied with an incorrect speed limit resulting in his exceeding the maximum speed limit, the question arises as to who will assume the liability for damages resulting from this. Based on the related art, incorrect classifications cannot be ruled out in the recognition of road signs.

[0004] In contrast, the method according to an exemplary embodiment of the present invention for the display of information has the advantage that a segment of the image obtained by the sensor is displayed directly without an analysis of the content of the road sign and a classification being made. The analysis of the camera's image searches for the outlines of the road signs and presents this image segment directly to the driver. The driver himself performs the classification; the road sign is merely pointed out to him and it is presented to him as a support. Should an error occur in the analysis, the driver is only presented an image segment without a road sign which he can simply ignore.

[0005] In an exemplary embodiment, information in addition to the road sign can be presented if the analysis unit additionally receives information from other vehicle sensors besides the camera, and both the time as well as the distance since the last recognition of a road sign can be displayed.

[0006] The exemplary method, method can also record road signs with supplemental information, for example, “only when wet” in the case of speed limits. The road sign is displayed with the additional sign, the presentation in the display area not actually being able to resolve the lettering but pointing out the presence of the supplemental sign.

[0007] It is also believed that with additional warning of the driver based on a road sign classification, the transparency of the method is preserved. The driver can easily compare the acoustic, haptic, or optical warning to the driver based on the information processing by comparing it with the image in the display area.

[0008] FIG. 1 shows a vehicle with an image sensor.

[0009] FIG. 2 shows an exemplary road scene.

[0010] FIG. 3 shows an exemplary road scene with an exemplary display.

[0011] FIG. 4 shows a road scene with an exemplary display.

[0012] FIG. 5 shows an exemplary road scene with an exemplary display.

[0013] FIG. 6 shows a schematic representation of the sensor technology in the motor vehicle.

[0014] FIG. 1 shows the positioning of image sensor 2 within vehicle 1. Image sensor 2 is attached as high as possible in a central position behind the windshield, the viewing direction being in alignment with the direction of travel. The sensor should still be located in the area of the windshield wipers. FIG. 2 shows a roadway 3, which, for example, is bordered by reflector posts 9. A vehicle 8 approaching from the opposite direction is shown schematically on the opposite roadway. A traffic sign combination is installed on the right edge of the roadway. Road sign 4 has a limiting additional sign 5. In addition, road sign 6 for a “Rough Road” is also installed. From the view of the driver, the instrument panel 10 of his own vehicle can be seen with steering wheel 12. A screen 11 is fitted into instrument panel 10 in the field of view of the driver, and image segment 7 with the relevant road signs is recorded by the image sensor and separated out by the image processing being displayed on screen 11.

[0015] FIG. 3 shows a scene as described in FIG. 2, information 13 being additionally displayed in an area of the instrument panel. FIG. 4 shows an additional display form in which distance information 14 is shown in addition. FIG. 5 shows the display form with time information 15. FIG. 6 shows an exemplary implementation of the sensor configuration for driver support. Image sensor 2 is connected via a cable to an electronic analysis circuit 16. Electronic analysis circuit 16 has additional connections to additional sensors 19, for example, a touch key via which a driver's intentions can be communicated to the electronic analysis circuit, and vehicle sensors 18 which detect, for example, vehicle speed, distance or time measurements. Electronic analysis circuit 16 is connected to control unit 17 which, in turn, is connected to a display area 11. Image sensor 2 is positioned in the direction of vehicle travel and records video images of the vehicle surroundings. These images are sent to the electronic analysis circuit via the cable. Software must be used in the electronic analysis circuit which can separate the road signs needed for driver support from other image components. In general, not all road signs are separated out. This depends, however, on the selected type of driver support. If it is desired, for example, to support the driver only with regard to the particularly significant class of road signs that limit vehicle speed, the software will be configured to recognize only those road signs that indicate a limitation of maximum speed. The software recognizes road signs on the basis of their outline structure 7, on the basis of colors or on the basis of optical structures located on the sign. Suitable algorithms for the segmenting of road signs are provided in “Detektion von Verkehrszeichen aus monochronen Bildfolgen [Detection of Road Signs from Monochrome Image Sequences],” Master's thesis by Holger Janssen, Institute for Theoretical Information Technology, University of Hanover 1997. If a road sign necessary for driver support has been segmented in a sensor image, it can be segmented again in each subsequent image and tracked over time until it leaves the field of view of the sensor. By repeated segmenting and tracking over time, it is possible to achieve a stabilization of the recognition of relevant road signs. After segmenting, a classification must be carried out in a suitable manner since it must be decided whether a road sign relevant for driver support is present. In doing so, not only a road sign relevant for driver support is shown with its informational content but rather the image segment containing the road sign is presented directly to the driver via a display area 11. If the recognition and segmenting of road signs is incorrect, which cannot be completely ruled out with the complexity of road sign recognition, a randomly and incorrectly segmented scene segment without road signs is presented to the driver in his display area. He can detect the malfunction of the system immediately and act accordingly. The system does not mislead the driver into a false sense of security.

[0016] Various improvements to the system for support of the driver are additionally possible. The image quality of the image recorded by the image sensor can be improved under certain circumstances with the aid of algorithmic methods. Such methods include image brightening and/or darkening, contrast intensification and edge sharpening. This method merely improves the image for the driver's view without, however, distorting or interpreting its information content.

[0017] An exemplary embodiment of the system allows the most recently segmented road sign from the large number of road signs relevant to the level of driver support desired at the time to be continuously presented to the driver in display area 11. On expressways on which primarily speed limitations, passing prohibitions and cancellations of such prohibitions occur, the driver could be informed of the most recent applicable driving limitation by presentation of the most recent segmented road sign.

[0018] An additional exemplary embodiment of the system makes it possible to retain a certain number of the most recently segmented road sign images in an electronic memory. In this case, an input device via a keypad 19 must be provided for the driver which makes it possible for him to specify which sign he would like to have displayed to him. A sequential presentation of the stored road sign images, starting with the most recently segmented sign, is particularly practical. By the input of a signal, for example, via suitable touch keys or an acoustic input via a suitable input sensor 19, the driver can switch to the next previously segmented road sign. It is thus possible for the driver to be reminded of the most recently recognized and segmented road signs as he wishes. In addition to the exclusive presentation of the segmented road signs, an additional character interpretation may be practical. In this way the driver can be provided with symbolic information via defined characters 13, see FIG. 3. In this case, by comparing the displayed characters with the simultaneously presented image segment, the driver can immediately determine if the automatic road sign interpretation is incorrect. In this case also, the function of the system is transparent so that the responsibility remains exclusively with the driver.

[0019] In a further exemplary embodiment, warnings are output to the driver which can be obtained on the basis of information from additional vehicle sensors. Thus, for example, a speed signal can be combined with the output of a warning. If the driver is driving faster than is permitted by the maximum allowable speed according to the road sign, the analysis unit recognizes this discrepancy and generates an acoustic warning signal. In doing so, it is possible to activate a warning tone, an acoustic speech output or even an optical warning display.

[0020] In a further exemplary embodiment, lane recognition is implemented via the image sensor and the analysis unit. With the aid of such image-based lane tracking, it is possible for the analysis unit to detect when the vehicle pulls out into the opposite lane for the purpose of passing, for example. In connection with a previously recognized passing prohibition, it is then possible to emit a warning signal again.

[0021] In addition to the presentation of the road sign on display area 11, it is also possible to supply the driver with distance information indicating the distance between the present vehicle position and the position at which the road sign was detected. Additional information 14 is presented in the display area—see FIG. 4. Consequently, the driver has at least a limited possibility to check if the displayed road sign is at all still relevant to him. In this connection, the distance information can be obtained directly from the distance measuring equipment present in every vehicle or also by associating vehicle speed information with the time passed since the road sign was detected. The sensor signals are evaluated in analysis unit 16 which brings about a possible output in display area 11 via control unit 17.

[0022] As an alternative to a presentation of the distance information, time information can be displayed to the driver simultaneously with the presentation of the road sign. For this purpose, the time difference between detection of the road sign and the actual time must be recorded for the driver via control unit 17.

[0023] For image detection, a CMOS sensor having a non-linear wall characteristic is advantageous for road sign detection. Road signs generally have painted, metallic surfaces or plastic surfaces that are smooth and readily reflect incident light. With unfavorable light incidence or night driving with headlight illumination, it is thus possible for extreme brightness dynamics caused by reflection of light or other effects to occur. Such brightness differences can be controlled with the aid of a non-linear image sensor, making it possible to supply a simple preparation of an analyzable video image to the electronic analysis circuit. In this connection, an image sensor with a logarithmic characteristic is particularly advantageous since it ensures a constant contrast resolution over the entire brightness range.