Information processing device, storage medium storing computer program for information processing, and information processing method

[0001] This application claims priority to Japanese Patent Application No. 2023-078114 filed May 10, 2023, the entire contents of which are herein incorporated by reference.

[0002] The present disclosure relates to an information processing device, a storage medium storing a computer program for information processing, and an information processing method.

[0003] An automatic control system mounted in a vehicle creates a navigation route for the vehicle based on the current location of the vehicle, the destination location of the vehicle, and a navigation map. The automatic control system estimates the current location of the vehicle using the map information and controls the vehicle to travel along the navigation route.

[0004] The automatic control system also uses a camera to acquire front images representing the environment ahead of the vehicle, and detects lane marking lines (or lane boundary lines) represented in the front images. The automatic control system identifies traffic lanes partitioned by the lane marking lines, and controls the vehicle so that the vehicle travels in a lane.

[0005] When a branching location exists where a branching road branches from the traveling road on which the vehicle is traveling, the traveling road and branching road appear in the front images. The automatic control system detects the lane marking lines of the traveling road and the lane marking lines of the branching road from the front images.

[0006] When the vehicle is scheduled to proceed on the traveling road, the automatic control system selects the lane marking line of the traveling road and controls the vehicle. On the other hand, when the vehicle is scheduled to proceed on the branching road, the automatic control system selects the lane marking line of the branching road and controls the vehicle.

[0007] The automatic control system determines whether or not the branching road appears ahead of the vehicle based on the front images.

[0008] Japanese Unexamined Patent Publication No. 2017-123009 proposes a technique for recognizing the traveling lane in which the vehicle is traveling and an adjacent lane adjacent to the traveling lane, based on images, and determining whether or not a branching situation exists in which a branching lane branches from the traveling lane, based on differences in the change tendencies in the lane widths of the traveling lane and the adjacent lane.

[0009] In the technique described in Japanese Unexamined Patent Publication No. 2017-123009, however, it has not been possible to determine the existence of a branching situation until the adjacent lane appears next to the traveling lane in which the vehicle is traveling.

[0010] It is therefore an object of the present disclosure to provide an information processing device that can rapidly detect a branching road that branches from a road in which a vehicle is traveling, based on a front image representing the environment ahead of the vehicle.

[0011] (1) According to one embodiment of the present disclosure there is provided an information processing device. The information processing device has a processor configured to detect a vanishing point where two lane marking lines in a front image representing environment ahead of a vehicle intersect, detect a moving direction of the vehicle from a predetermined reference location to the vanishing point in the front image, determine whether or not each of two moving directions has a different orientation in excess of a predetermined reference range with respect to the previously detected moving direction when the two moving directions have been detected, and determine that a branching road that branches from a road in which the vehicle is traveling has appeared in the front image, when it has been determined that either of the two moving directions has a different orientation.

[0012] (2) In the information processing device of (1) above, the processor is further configured to determine whether or not the vehicle will exit from the branching road based on a driving plan representing the scheduled traveling trajectory of the vehicle, which is generated based on determination results wherein it has been determined that the branching road has appeared, and remove the moving direction determined to have a different orientation from the previously detected moving direction, when it has been determined that the vehicle will not exit to the branching road.

[0013] (3) In the information processing device of (2) above, the processor is further configured to set the moving direction determined to have a different orientation to be the previously detected moving direction, when it has been determined that the vehicle will exit from the branching road.

[0014] (4) In the information processing device according to any one of (1) to (3) above, the processor is further configured to determine whether or not the vehicle is scheduled to exit to the branching road which branches from the traveling road on which it is traveling based on a vehicle navigation route, within a predetermined range from the current location of the vehicle in a direction of travel, and remove lane marking lines that have a vanishing point, representing the moving direction determined to have a different orientation, from among the lane marking lines detected in the front image, when it has been determined that the branching road has appeared and it has been determined that the vehicle is not scheduled to exit from the traveling road to the branching road.

[0015] (5) In the information processing device according to any one of (1) to (4) above, the predetermined reference location in the front image represents the location of the vehicle.

[0016] (6) According to another embodiment, a storage medium storing a computer program for information processing is provided. The computer program for information processing causes a processor to execute a process and the process includes detecting a vanishing point where two lane marking lines in a front image representing environment ahead of a vehicle intersect, detecting a moving direction of the vehicle from a predetermined reference location to the vanishing point in the front image, determining whether or not each of two moving directions has a different orientation in excess of a predetermined reference range with respect to the previously detected moving direction when the two moving directions have been detected; and determining that a branching road that branches from a road in which the vehicle is traveling has appeared in the front image, when it has been determined that either of the two moving directions has a different orientation.

[0017] (7) According to yet another embodiment of the present disclosure, an information processing method is provided. The information processing method is carried out by an information processing device and the method includes detecting a vanishing point where two lane marking lines in a front image representing environment ahead of a vehicle intersect, detecting a moving direction of the vehicle from a predetermined reference location to the vanishing point in the front image, determining whether or not each of two moving directions has a different orientation in excess of a predetermined reference range with respect to the previously detected moving direction when the two moving directions have been detected; and determining that a branching road that branches from a road in which the vehicle is traveling has appeared in the front image, when it has been determined that either of the two moving directions has a different orientation.

[0018] The information processing device of the present disclosure can rapidly detect a branching road that branches from a road in which a vehicle is traveling, based on a front image representing the environment ahead of the vehicle.

[0019] The object and aspect of the present disclosure will be realized and attained by the elements and combinations particularly pointed out in the claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the present disclosure, as claimed.

[0032] FIG. 1 and FIG. 2 are diagrams showing operation of the information processing device 12 according to the embodiment in overview. Operation relating to information processing by the information processing device 12 disclosed herein will now be described in overview with reference to FIG. 1 and FIG. 2.

[0033] The vehicle 10 has an automatic control device 11 and an information processing device 12. The automatic control device 11 controls operation of the vehicle 10 based on front images taken by the front camera 2. The environment ahead of the vehicle 10 is represented in the front images. The vehicle 10 may also be a self-driving vehicle. Based on the front images, the information processing device 12 generates information for control of the vehicle 10 by the automatic control device 11.

[0034] In the example shown in FIG. 1, the vehicle 10 is traveling on a road 50. The road 50 has a left lane marking line 51, a right lane marking line 52, and a lane 53 delineated by the left lane marking line 51 and right lane marking line 52. The vehicle 10 is traveling on the lane 53.

[0035] The information processing device 12 detects two lane marking lines 101, 102 in a front image 100 taken by the camera 2. The lane marking line 101 corresponds to the left lane marking line 51, and the lane marking line 102 corresponds to the right lane marking line 52. The information processing device 12 also detects a vanishing point F1 where the two lane marking lines 101, 102 intersect in the front image 100.

[0036] The information processing device 12 further detects a moving direction V1 for movement of the vehicle 10 from a predetermined reference location R to the vanishing point F1 in the front image 100. The reference location R may be, for example, the location corresponding to the center of the front image 100 in the widthwise direction of the vehicle 10. The widthwise direction of the vehicle 10 is the direction perpendicular to the longitudinal direction of the vehicle 10. The reference location R is situated at the lower end center of the front image 100.

[0037] Since only one moving direction V1 has been detected, the information processing device 12 decides to set the lane marking line 101 and lane marking line 102, which define the vanishing point F1 corresponding to the moving direction V1 in the front image 100, as lane marking lines delineating the region in which the vehicle 10 is traveling.

[0038] The information processing device 12 notifies the automatic control device 11 of the lane marking line 101 and lane marking line 102 in the front image 100.

[0039] The automatic control device 11 controls operation of the vehicle 10 based on the lane marking line 101 and lane marking line 102 in the front image 100. For example, based on the front image 100, the automatic control device 11 drives the vehicle 10 so as to travel in the region 103 between the lane marking line 101 and lane marking line 102. The vehicle 10 is thus controlled to travel in the lane 53 delineated by the left lane marking line 51 and right lane marking line 52 of the road 50.

[0040] In the example shown in FIG. 2, the vehicle 10 is traveling on a road 50. A road 60 branches off from the road 50 ahead of the vehicle 10. The road 60 has a left lane marking line 61, a right lane marking line 62, and a lane 63 delineated by the left lane marking line 61 and right lane marking line 62.

[0041] The information processing device 12 detects two lane marking lines 101, 102 in a front image 110. The lane marking line 101 corresponds to the left lane marking line 51 of the road 50, and the lane marking line 102 corresponds to the right lane marking line 52 of the road 50. The information processing device 12 also detects a vanishing point F1 where the two lane marking lines 101, 102 intersect in the front image 110.

[0042] The information processing device 12 also detects two lane marking lines 104, 105 in the front image 110. The lane marking line 104 corresponds to the left lane marking line 61 of the road 60, and the lane marking line 105 corresponds to the right lane marking line 62 of the road 60. The information processing device 12 also detects a vanishing point F2 where the two lane marking lines 104, 105 intersect in the front image 110.

[0043] The information processing device 12 further detects a moving direction V1 for movement of the vehicle 10 from a reference location R to the vanishing point F1 in the front image 110. The information processing device 12 further detects a moving direction V2 for movement of the vehicle 10 from a reference location R to the vanishing point F2 in the front image 110. The moving direction V2 is a newly detected moving direction.

[0044] Since the moving direction V2 has a different orientation in excess of a predetermined reference range with respect to the previously detected moving direction V1, the information processing device 12 determines that the road 60 branching from the road 50 on which the vehicle 10 is traveling has appeared in the front image 110.

[0045] When the road 60 is shown to branch from the road 50 in the front image 110, the information processing device 12 can determine that the road 60 has appeared based on the moving direction V2 representing the road 60. The information processing device 12 can rapidly determine that the road 60 has appeared, although the timing at which the road 60 appears in the front image 110 will depend on the visual field of the camera 2 and the mounting angle of the camera 2 on the vehicle 10.

[0046] The information processing device 12 decides to set the lane marking line 101 and lane marking line 102, which define the vanishing point F1 corresponding to the moving direction V1 in the front image 110, as lane marking lines of the road 50 in which the vehicle 10 is currently traveling.

[0047] The information processing device 12 also decides to set the lane marking line 104 and lane marking line 105, which define the vanishing point F2 corresponding to the moving direction V2 in the front image 110, as lane marking lines of the branching road 60 which branches from the road 50 in which the vehicle 10 is currently traveling.

[0048] The information processing device 12 notifies the automatic control device 11 of the lane marking line 101 and lane marking line 102 in the front image 110, as lane marking lines of the road 50 in which the vehicle 10 is currently traveling.

[0049] The information processing device 12 also notifies the automatic control device 11 of the lane marking line 104 and lane marking line 105 in the front image 110, as lane marking lines of the road 60 which branches.

[0050] When the vehicle 10 is to progress on the road 50 without exiting from the road 50 to the road 60, the automatic control device 11 controls the operation of the vehicle 10 based on the lane marking line 101 and lane marking line 102 of the front image 110. For example, based on the front image 110, the automatic control device 11 drives the vehicle 10 so as to travel in the region 103 between the lane marking line 101 and lane marking line 102. The vehicle 10 is thus controlled to travel in the lane 53 delineated by the left lane marking line 51 and right lane marking line 52 of the road 50.

[0051] When the vehicle 10 is to exit from the road 50 to the road 60, on the other hand, the automatic control device 11 controls operation of the vehicle 10 based on the lane marking line 104 and lane marking line 105 of the front image 110, after the vehicle 10 has exited from the road 50. For example, based on the front image 110, the automatic control device 11 drives the vehicle 10 so as to travel in the region 106 between the lane marking line 104 and lane marking line 105. The vehicle 10 is thus controlled to travel in the lane 63 delineated by the left lane marking line 61 and right lane marking line 62 of the road 60.

[0052] As explained above, the information processing device 12 can rapidly detect a branching road that branches from a road in which a vehicle is traveling, based on front images representing the environment ahead of the vehicle 10.

[0053] By having the information processing device 12 notify the automatic control device 11 of the lane marking lines of the road 50 on which the vehicle 10 is traveling and the road 60 which branches, the automatic control device 11 can generate a driving plan which properly takes into account the region in which the vehicle 10 is traveling.

[0054] FIG. 3 is a hardware configuration diagram for a vehicle 10 in which the information processing device 12 of the embodiment is mounted. The vehicle 10 has a front camera 2, an automatic control device 11, and an information processing device 12. The vehicle 10 may also have a LiDAR sensor, as another distance sensor (not shown) for measurement of the distance of the vehicle 10 to surrounding objects.

[0055] The front camera 2, automatic control device 11 and information processing device 12 are connected in a communicable manner via an in-vehicle network 13 conforming to the Controller Area Network standard.

[0056] The camera 2 is mounted inside the vehicle 10 and directed toward the front of the vehicle 10. The camera 2, for example, takes a front image in which the environment of a predetermined region ahead of the vehicle 10 is shown, at a predetermined cycle. The front image can show the road in the predetermined region ahead of the vehicle 10, and road features such as surface lane marking lines on the road. The front image may also include other vehicles which are located ahead of the vehicle 10. The camera 2 has a 2D detector composed of an array of photoelectric conversion elements with visible light sensitivity, such as a CCD or C-MOS, and an imaging optical system that forms an image of the photographed region on the 2D detector.

[0057] Each time a front image is taken, the front camera 2 outputs the front image and camera imaging time at which the front image was taken to the automatic control device 11 and information processing device 12 via the in-vehicle network 13. The front image is also used for processing at the automatic control device 11 to estimate the location of the vehicle 10. At the automatic control device 11, the front image is also used for processing to detect objects surrounding the vehicle 10. The front image is also used for information processing by the information processing device 12. The vehicle 10 may also have another camera mounted in the vehicle 10 and directed toward the rear of the vehicle 10.

[0058] The automatic control device 11 controls operation including traveling of the vehicle 10. Based on the navigation map information, the destination location of the vehicle 10, and positioning information representing the current location of the vehicle 10 input from a GNSS receiver (not shown), the automatic control device 11 creates a navigation route from the current location to the destination location of the vehicle 10. The automatic control device 11 outputs the current location of the vehicle 10 and the navigation route to the information processing device 12 via the in-vehicle network 13.

[0059] The automatic control device 11 also estimates the location and orientation of the vehicle 10 at the camera imaging time, based on lane marking lines represented in high-precision map information, and lane marking lines surrounding the vehicle 10 which are represented in front images taken by the camera 2. The automatic control device 11 also estimates the road traveling lane where the vehicle 10 is located, based on the lane marking lines represented in the high-precision map information and on the estimated location and estimated declination of the vehicle 10.

[0060] The automatic control device 11 further detects objects around the vehicle 10, and their types, based on the front images. Objects also include other vehicles traveling around the vehicle 10.

[0061] The automatic control device 11 selects a traffic lane on the road on which the vehicle 10 is traveling, within the nearest driving zone (for example, 10 km) selected from the navigation route, based on the high-precision map information, the navigation route and the current location of the vehicle 10, and creates a traveling lane plan representing the scheduled traveling lane for traveling of the vehicle 10.

[0062] The automatic control device 11 also creates a driving plan representing the scheduled traveling trajectory of the vehicle 10 up until a predetermined time (for example, 5 seconds), based on the front images, the traveling lane plan, the high-precision map information, the current location of the vehicle 10, the surrounding environment information and the vehicle status information. The surrounding environment information includes the locations of other vehicles traveling around the vehicle 10. The vehicle status information includes the current location of the vehicle 10, and the vehicle speed, acceleration and traveling direction. The automatic control device 11 generates a driving plan for control of operation including steering, actuation and braking of the vehicle 10, so that the vehicle 10 travels through the region between the lane marking lines detected in the front images.

[0063] The automatic control device 11 also controls each unit of the vehicle 10 based on the driving plan. For example, the automatic control device 11 outputs the control signal based on the driving plan, to an actuator (not shown) that controls the steering wheel, a drive unit (not shown), or the brake (not shown), via the in-vehicle network 13.

[0064] The information processing device 12 carries out detection processing, determination processing, decision processing and removal processing. For this purpose, the information processing device 12 has a communication interface (IF) 21, a memory 22 and a processor 23. The communication interface 21, memory 22 and processor 23 are connected via signal wires 24. The communication interface 21 has an interface circuit to connect the information processing device 12 with the in-vehicle network 13.

[0065] The memory 22 is an example of a memory unit, and it has a volatile semiconductor memory and a non-volatile semiconductor memory, for example. The memory 22 stores an application computer program and various data to be used for information processing carried out by the processor 23 of each device. The memory 22 also stores internal parameters such as the optical axis direction of the camera 2 and the focal length and viewing angle of the imaging optical system, and external parameters such as the mounting position and orientation of the camera 2.

[0066] All or some of the functions of the information processing device 12 are functional modules driven by a computer program operating on the processor 23, for example. The processor 23 has a detecting unit 231, a determining unit 232, a deciding unit 233 and a removing unit 234. The processor 23 comprises one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may also have other computing circuits such as a logical operation unit, numerical calculation unit or graphics processing unit. Alternatively, the functional module of the processor 23 may be a specialized computing circuit in the processor 23. The information processing device 12 is an electronic control unit (ECU), for example. Operation of the information processing device 12 will be described in detail below.

[0067] For FIG. 3, the automatic control device 11 and the information processing device 12 were described as separate devices, but all or some of the devices may also be constructed as a single device.

[0068] FIG. 4 is an example of an operation flow chart for information processing by the information processing device 12 of the embodiment. Information processing by the information processing device 12 will be described below with reference to FIG. 4. The information processing device 12 carries out information processing according to the operation flow chart shown in FIG. 4, at an information processing time set with a predetermined cycle.

[0069] First, the detecting unit 231 detects a vanishing point where the two lane marking lines intersect in the front images taken by the camera 2 (step S101). The location of the vanishing point is represented on an image coordinate system. The image coordinate system has the top left corner of the front image as the origin, with the x-axis extending horizontally to the right from the origin and the y-axis extending downward from the origin, perpendicular to the x-axis. The detecting unit 231 is an example of the vanishing point detector. Processing whereby the detecting unit 231 detects a vanishing point will be described below with reference to FIG. 5 to FIG. 10.

[0070] The detecting unit 231 then detects a moving direction for the vehicle 10 from a predetermined reference location to the vanishing point in the front image (step S102). The detecting unit 231 is an example of the direction detector. The reference location may be the location corresponding to the vehicle 10 in the front images. For example, the reference location may be a location at the center in the widthwise direction of the vehicle 10. In embodiments, the reference location is situated at the lower end in the front images. The reference location is represented on the image coordinate system. The location of the reference location in the front images may be decided based on the internal parameters and external parameters of the camera 2. For example, when the camera 2 is mounted at the center location in the widthwise direction of the vehicle 10 in an orientation with the optical axis toward the front of the vehicle 10, the reference location is the center location at the lower end in the front images.

[0071] The moving direction of the vehicle 10 is represented in the image coordinate system, as the direction of a vector with the reference location as the initial point and the vanishing point as the terminal point. When two or more vanishing points have been detected in a front image, the moving direction of the vehicle 10 is detected with respect to each vanishing point.

[0072] The determining unit 232 then determines whether or not two moving directions have been detected by the detecting unit (step S103).

[0073] When two moving directions have been detected (step S103—Yes), the determining unit 232 determines whether or not each of the two moving directions has a different orientation in excess of a predetermined reference range with respect to the previously detected moving direction (step S104). The previously detected moving direction may be the most recent average moving direction (within 3 seconds, for example). The determining unit 232 is an example of the first determining unit.

[0074] First, the vector representing the current detected moving direction and a vector representing the previously detected moving direction are superimposed on the image coordinate system with their respective initial points at the reference location. The angle of the vector representing the current detected moving direction with respect to the direction of the vector representing the previously detected moving direction is calculated. It is determined whether or not this angle exceeds a predetermined reference angle. The reference angle may be 30 milliradians (mil), for example. Both edges of the road with a width of 3.5 m will be at about 30 milliradians at a distance of 100 m ahead of the vehicle 10.

[0075] When the direction of the vector representing the current detected moving direction is in excess of the reference angle with respect to the direction of the vector representing the previously detected moving direction, in the image coordinate system, the determining unit 232 determines that the direction of the moving direction is different. When the direction of the vector representing the current detected moving direction indicates a branching road that branches from the road in which the vehicle 10 is currently traveling, then it is determined that the orientation of the moving direction is different. The direction of the vector representing the current detected moving direction thus signifies the appearance of a branching road.

[0076] On the other hand, when the direction of the vector representing the current detected moving direction is not in excess of the reference angle with respect to the direction of the vector representing the previously detected moving direction in the image coordinate system, the determining unit 232 determines that the orientation of the moving direction is not different. The direction of the vector representing the current detected moving direction may be said to coincide with the direction of the vector representing the previously detected moving direction. When the direction of the vector representing the current detected moving direction indicates the road in which the vehicle 10 is currently traveling, then it is determined that the orientation of the moving direction is not different. The direction of the vector representing the current detected moving direction is therefore within the reference angle with respect to the direction of the vector representing the previously detected moving direction.

[0077] When the branching road is shown to branch from the traveling road in a front image, the information processing device 12 can determine that the branching road has appeared based on the moving direction representing the branching road. The information processing device 12 can rapidly determine that the branching road has appeared, although the timing at which the branching road appears in the front image will depend on the visual field of the camera 2 and the mounting angle of the camera 2 on the vehicle 10.

[0078] A smaller reference angle allows the information processing device 12 to determine more rapidly that the branching road has appeared. However, a smaller reference angle may also result in erroneous determination of branching roads. In embodiments, the size of the reference angle is therefore decided in consideration of precision.

[0079] When it has been determined that either of the two moving directions has a different orientation (step S104—Yes), the determining unit 232 determines that a branching road which branches from the road on which the vehicle 10 is traveling has appeared in the front image (step S105). When it has been determined that either of the two moving directions has a different orientation, it is possible that the front image shows lane marking lines of the traveling road in which the vehicle 10 is currently traveling, and lane marking lines of a branching road. The determining unit is an example of a second determining unit.

[0080] The vector representing the same direction as the previously detected moving direction corresponds to the traveling road in which the vehicle 10 is currently traveling. The vector representing a different direction from the previously detected moving direction corresponds to the branching road.

[0081] Next, the deciding unit 233 decides on the lane marking lines of the traveling road in which the vehicle 10 is currently traveling and the lane marking lines of the branching road, in the front image (step S106). The two lane marking lines defining the vanishing point which represents the terminal point of the vector representing the same direction as the previously detected moving direction are the lane marking lines of the traveling road. The two lane marking lines defining the vanishing point which represents the terminal point of the vector representing the different direction from the previously detected moving direction are the lane marking lines of the branching road.

[0082] The deciding unit 233 then notifies the automatic control device 11 of information representing the locations of the lane marking lines in the front image, and the series of processing steps is complete (step S107). Specifically, the deciding unit 233 notifies the automatic control device 11 of the front image, together with information representing the locations of the two lane marking lines delineating the traveling lane of the traveling road in which the vehicle 10 is currently traveling, and information representing the locations of the two lane marking lines delineating the lane of the branching road.

[0083] When the vehicle 10 is to proceed on the traveling road without exiting from the traveling road to the branching road, the automatic control device 11 generates a driving plan for the vehicle 10 so that it travels in the region between the two lane marking lines delineating the lane of the traveling road, based on the front image. The automatic control device 11 may also generate a driving plan by converting the front image to a bird's eye view based on the internal parameters and external parameters of the camera 2.

[0084] When the vehicle 10 is to exit from the road 50 to the road 60, on the other hand, the automatic control device 11 generates a driving plan for the vehicle 10 so that it travels in the region between the two lane marking lines delineating the lane of the branching road, based on the front image.

[0085] When two moving directions are not detected (step S103—No), or when it has been determined that neither of the two moving directions has a different orientation, (step S104—No), the deciding unit 233 decides on the lane marking lines of the traveling road in which the vehicle 10 is currently traveling, in the front image (step S108). The two lane marking lines defining the vanishing point which represents the terminal point of the vector representing the same direction as the previously detected moving direction are the lane marking lines of the traveling road. For example, when the vehicle 10 is traveling on a road with two or more lanes, it is determined that neither of the two moving directions has a different orientation.

[0086] The deciding unit 233 then notifies the automatic control device 11 of the front image and the information representing the locations of the lane marking lines in the front image (step S107), and the series of processing steps is complete. Specifically, the deciding unit 233 notifies the automatic control device 11 of the front image, together with information representing the locations of the two lane marking lines delineating the traveling lane of the traveling road in which the vehicle 10 is currently traveling. The automatic control device 11 generates a driving plan for the vehicle 10 so that it travels in the region between the two lane marking lines delineating the lane of the traveling road, based on the front image.

[0087] Processing by which the detecting unit 231 detects a vanishing point in a front image in step S101 will now be explained with reference to FIG. 5 to FIG. 10.

[0088] FIG. 5 is a drawing showing an example of a front image. The front image 200 shown in FIG. 5 corresponds to the front image 100 of FIG. 1. The detecting unit 231 detects regions 201, 202 corresponding to the two lane marking lines represented in the front image 200. The detecting unit 231 has a classifier trained to identify lane marking lines represented on road surfaces from images. By inputting the front image 200 into the classifier, the detecting unit 231 detects a region 201 and a region 202 respectively corresponding to the two lane marking lines represented in the front image 200. The positions of the pixels in the region 201 and region 202 are represented in the image coordinate system.

[0089] FIG. 6 is a diagram showing detection of a vanishing point and moving direction from a front image 200. The detecting unit 231 finds a polynomial representing the lane marking lines based on the positions of the pixels representing the lane marking lines, for both the region 201 and region 202. For example, the detecting unit 231 finds a polynomial using the least-square method.

[0090] The detecting unit 231 sets the vanishing point for both the region 201 and region 202 to be the intersection F1 between curves P1 and P2 represented by the polynomial. When the road is straight, the lane marking line is nearly a straight line, and when the road is curved, the lane marking line is nearly a curve.

[0091] The detecting unit 231 then calculates a vector V1 with the initial point being the reference location R set at the center at the lower end of the front image 200, and the terminal point being the vanishing point F1. The vector V1 represents the moving direction of the vehicle 10.

[0092] Two lane marking lines were detected in the example described above, but if three or more lane marking lines are detected, the detecting unit 231 calculates the vanishing point and moving direction for two adjacent lane marking lines.

[0093] FIG. 7 is a drawing showing another example of a front image. The front image 300 shown in FIG. 7 corresponds to the front image 110 of FIG. 2. The detecting unit 231 detects regions 301, 302, 303, 304 corresponding to each of the four lane marking lines represented in the front image 300. The positions of the pixels in the regions 301, 302, 303, 304 are represented in the image coordinate system.

[0094] FIG. 8 is a diagram showing detection of a vanishing point and moving direction from a front image 200. The detecting unit 231 finds a polynomial representing the lane marking lines based on the positions of the pixels representing the lane marking lines, for each of the regions 301, 302, 303, 304.

[0095] The detecting unit 231 sets the vanishing point for the adjacent regions 301, 302 to be the intersection F1 between curves P3 and P4 represented by the polynomial. The detecting unit 231 also sets the vanishing point for the adjacent regions 303, 304 to be the intersection F2 between curves P5 and P6 represented by the polynomial. When scanning the front image along the x-axis direction at a predetermined position on the y-axis, two adjacently situated regions without other intervening regions are considered to be adjacent regions.

[0096] The detecting unit 231 then calculates a vector V1 with the initial point being the reference location R set at the center at the lower end of the front image 200, and the terminal point being the vanishing point F1. The vector V1 represents the moving direction of the vehicle 10. The detecting unit 231 then calculates a vector V2 with the initial point being the reference location R set at the center at the lower end of the front image 200, and the terminal point being the vanishing point F2. The vector V2 represents another moving direction of the vehicle 10.

[0097] The method by which the detecting unit 231 calculates the vanishing point is not limited to the method described above. For example, the detecting unit 231 may detect the vanishing point by the following method in the front image shown in FIG. 9.

[0098] FIG. 9 is a drawing showing another example of a front image. The front image 300 shown in FIG. 9 corresponds to the front image 110 of FIG. 2. The detecting unit 231 may apply an edge detection filter such as a Sobel filter to the front image 300 to detect edge pixels, and it may detect pixels having brightness above a predetermined threshold value among the detected pixels, as candidate points. The locations of the candidate points are represented on the image coordinate system.

[0099] The detecting unit 231 groups the candidate points within a predetermined reference number of pixels from the lower end to the upper end of the front image 300, to generate regions 401, 402, 403. The reference number of pixels may be from 5 to 10, for example.

[0100] FIG. 10 is a diagram showing detection of a vanishing point and moving direction from a front image 300. The detecting unit 231 finds polynomials based on the respective candidate points for regions 401, 402, 403.

[0101] The detecting unit 231 sets the vanishing point for the adjacent regions 402, 403 to be the intersection F1 between curves P8 and P9 represented by the polynomials. The detecting unit 231 also sets the vanishing point for the adjacent regions 401, 403 to be the intersection F2 between curves P7 and P9 represented by the polynomials.

[0102] The detecting unit 231 then calculates a vector V1 with the initial point being the reference location R set at the center at the lower end of the front image 200, and the terminal point being the vanishing point F1. The vector V1 represents the moving direction of the vehicle 10. The detecting unit 231 then calculates a vector V2 with the initial point being the reference location R set at the center at the lower end of the front image 200, and the terminal point being the vanishing point F2. The vector V2 represents another moving direction of the vehicle 10.

[0103] As explained in detail above, the information processing device can rapidly detect a branching road that branches from a road in which a vehicle is traveling, based on front images representing the environment ahead of the vehicle.

[0104] By having the information processing device notify the automatic control device of the lane marking lines of the traveling road on which the vehicle is traveling and the branching road, the automatic control device can generate a driving plan which properly takes into account the region in which the vehicle is traveling.

[0105] Modified Example 1 and Modified Example 2 of the information processing device 12 of this embodiment will now be explained with reference to FIG. 11 and FIG. 12.

[0106] FIG. 11 is an example of an operation flow chart for information processing according to Modified Example 1, by the information processing device 12 of the embodiment.

[0107] This modified example differs from the operation flow chart shown in FIG. 4 in that the processing of steps S209 to S212 is added. The processing in steps S201 to S208 is the same as in steps S101 to S108 described above.

[0108] For this modified example, the automatic control device 11 generates a driving plan for the vehicle 10 as described above, after the automatic control device 11 has been notified of the information representing the locations of lane marking lines in the front image (step S207).

[0109] When the vehicle 10 is to proceed on the traveling road without exiting from the traveling road to the branching road, the automatic control device 11 generates a driving plan for the vehicle 10 so that it travels in the region between the two lane marking lines delineating the lane of the traveling road, based on the front image.

[0110] When the vehicle 10 is to exit from the road 50 to the road 60, the automatic control device 11 generates a driving plan for the vehicle 10 so that it travels in the region between the two lane marking lines delineating the lane of the branching road, based on the front image.

[0111] The automatic control device 11 also notifies the information processing device 12 of the driving plan, via the network 13.

[0112] The determining unit 232 acquires the driving plan from the automatic control device 11 (step S209). Based on the driving plan, the determining unit 232 then determines whether or not the vehicle 10 is to exit into the branching road (step S210). The driving plan represents a scheduled traveling trajectory for the vehicle 10 until a predetermined time ahead. The determining unit 232 is an example of a third determining unit. In embodiments, the cycle for the information processing time is longer than the cycle for generating the driving plan by the automatic control device 11.

[0113] When the scheduled traveling trajectory of the vehicle 10 is indicated as traveling in the region between the two lane marking lines delineating the lane of the branching road, it is determined that the vehicle 10 will exit into the branching road.

[0114] When the scheduled traveling trajectory of the vehicle 10 is indicated as traveling in the region between the two lane marking lines delineating the lane of the traveling road, on the other hand, it is determined that the vehicle 10 will not exit into the branching road.

[0115] When it has been determined that the vehicle 10 will not exit into the branching road (step S210—No), the removing unit 234 removes the moving direction determined to have a different orientation from the previously detected moving direction (step S211), and the series of processing steps is complete. The removing unit 234 is an example of the first removing unit.

[0116] Then when a branching road has been represented in the front image during the next information processing, this branching road is determined to be a newly appearing branching road.

[0117] When it has been determined that the vehicle 10 will exit into the branching road (step S210—Yes), on the other hand, the detecting unit 231 sets the previously detected moving direction as the moving direction determined to have a different orientation in excess of a predetermined reference range with respect to the previously detected moving direction (step S212), and the series of processing steps is complete. The removing unit 234 also removes the moving direction of the traveling road from the previously detected moving direction.

[0118] Therefore, during the next information processing, the branching road is treated as the traveling road in which the vehicle 10 is traveling.

[0119] In the modified example described in detail above, the information processing device changes the previously detected moving direction based on the vehicle driving plan. This allows the information processing device to properly determine the appearance of a branching road in the next information processing.

[0120] FIG. 12 is an example of an operation flow chart for information processing according to Modified Example 2, by the information processing device of the embodiment.

[0121] This modified example differs from the operation flow chart shown in FIG. 4 in that the processing of steps S309 to S311 is added. The processing in steps S301 to S308 is the same as in steps S101 to S108 described above.

[0122] After it has been determined that there has appeared in the front image a branching road which branches from the road on which the vehicle 10 is traveling (step S305), the determining unit 232 refers to the navigation route and determines whether or not the vehicle 10 is scheduled to exit from the traveling road into the branching road within a predetermined range from the current location of the vehicle 10 in the traveling direction (step S309). The predetermined range may be 1 km to 5 km, for example.

[0123] When it is not scheduled to exit into the branching road (step S309—No), the removing unit 234 removes the lane marking lines from the branching road (step S310). Specifically, the removing unit 234 removes the lane marking lines that had a vanishing point representing a moving direction determined to have a different orientation, from among the lane marking lines detected in the front image. The removing unit 234 is an example of the second removing unit. The removing unit 234 determines that lane marking lines that had a vanishing point representing a moving direction determined to have a different orientation delineate an adjacent lane which is adjacent to the lane in which the vehicle 10 is currently traveling.

[0124] Next, the deciding unit 233 decides on the lane marking lines of the traveling road in which the vehicle 10 is currently traveling in the front image (step S311).

[0125] The deciding unit 233 then notifies the automatic control device 11 of the front image and the information representing the locations of the lane marking lines in the front image (step S307), and the series of processing steps is complete. Specifically, the deciding unit 233 notifies the automatic control device 11 of the front image, together with information representing the locations of the two lane marking lines delineating the traveling lane of the traveling road in which the vehicle 10 is currently traveling.

[0126] When the vehicle is scheduled to exit into the branching road (step S309—Yes), however, processing proceeds to step S306. The processing in step S306 is the same as in step S106 described above.

[0127] In the modified example described in detail above, when the vehicle 10 is not scheduled to exit from the traveling road into the branching road within the predetermined range from the current location of the vehicle 10 in the traveling direction, the automatic control device 11 is notified of the front image from which the lane marking lines of the branching road have been removed.

[0128] The automatic control device 11 generates a driving plan for the vehicle 10 so that it travels in the region between the two lane marking lines delineating the traveling lane of the traveling road, based on the front image in which only the lane marking lines of the traveling road are shown. The automatic control device 11 can thus generate a safer driving plan.

[0129] The information processing device, computer program for information processing and information processing method of the embodiment described above according to the present disclosure may incorporate appropriate modifications that are still within the gist of the present disclosure. Moreover, the technical scope of the disclosure is not limited to these embodiments, and includes the present disclosure and its equivalents as laid out in the Claims.