Automotive infotainment system with spatially-cognizant applications that interact with a speech interface

[0001] This application is a division of U.S. application Ser. No. 17/539,508, filed Dec. 1, 2021, now U.S. Pat. No. 12,086,501, which claims the benefit of the Dec. 9, 2020 priority date of U.S. Provisional Applications 63/123,110, 63/123,114, 63/123,119, and the Feb. 10, 2021 priority date of U.S. Provisional Application 63/147,991, all of which are incorporated herein by reference.

[0002] A modern automobile often comes equipped a processing system that is used to receive a designated coordinate and to calculate an optimal route to that coordinate subject to certain constraints imposed by the available paths between the automobile's location and that coordinate. In some cases, the processing system connects to a receiver that receives a signal from which the vehicle's progress towards the designated coordinate can be determined.

[0003] It is particularly useful for the processing system to also connect to a speech interface so that audible announcements can be provided to the vehicle's operator. Such instructions assist the operator in following the calculated optimal route.

[0004] In one aspect, the invention features an automotive processing unit that is integrated into a vehicle having a cabin and a loudspeaker that radiates sound into the cabin. The automotive processing unit includes a navigator and an infotainment system. The infotainment system includes a speech interface, an application suite including one or more spatially-cognizant applications, and an automotive assistant that is configured to execute one or more of the spatially-cognizant applications. The speech interface receives a navigation announcement from the navigator and a touring announcement from one of the spatially-cognizant applications. In response, the speech interface then causes a spoken announcement to be made audible in the cabin. It does so by activating the loudspeaker. The spoken announcement includes content from at least one of the touring announcement and the navigation announcement.

[0005] Among the embodiments are those in which the spoken announcement is a hybrid announcement. A hybrid announcement is one that comprises content from both the touring announcement and the navigation announcement. In such embodiments, the speech interface combines content from the navigation announcement and the touring announcement to form the hybrid announcement.

[0006] Among the embodiments are those in which the speech interface includes a natural-language engine. In some of these embodiments, the natural language engine identifies, based on grammatical structure of one of the announcements, an insertion point for inserting content from into the other to form a hybrid announcement that is then made audible as the spoken announcement. In others, it identifies a sentence in one of the announcements and inserts content from the other announcement at an end of that sentence to form the spoken announcement. In still others, the natural-language engine inserts one announcement between first and second sentences of the other announcement, thereby forming a hybrid announcement that is made audible in the cabin as the spoken announcement. In still others, the natural-language engine identifies a sentence in one second announcement, selects, as an insertion point, a beginning of that sentence, and inserts content from the other announcement at the insertion point to thereby form a hybrid announcement that is made audible in the cabin as the spoken announcement.

[0007] Embodiments further include those in which the speech interface forms a hybrid announcement by inserting a word, a phrase, or a cue, for example, a syntactic cue, between the navigation announcement and the touring announcement. This hybrid announcement is then made audible in the cabin as the spoken announcement.

[0008] Embodiments also include those in which the speech interface combines touring content from the touring announcement and navigation content from the navigation announcement to form a hybrid announcement. Among these are embodiments in which the touring content and navigation content are delivered using different voices and those in which they are delivered using the same voice.

[0009] In still other embodiments, the speech interface includes a summarizer that receives the touring announcement and generates a summarized version thereof. This summarized version is then interleaved into the navigation announcement to form the spoken announcement.

[0010] Also, among the embodiments are those in which the infotainment system includes a time scaler that receives touring audio from an application in the application suite, changes a duration thereof, and includes the touring audio, with its duration having been changed, as a constituent of the spoken announcement.

[0011] In still other embodiments, the speech interface interleaves content from the touring announcement and the navigation announcement to form a hybrid announcement that includes content from both announcements. This hybrid announcement serving as the spoken announcement.

[0012] In some embodiments, the application suite includes an itinerary-generating application that generates an itinerary. The itinerary includes a path and a touring-announcement section along the path. In such embodiments, the spoken announcement, which includes content concerning a point-of-interest that is visible from the touring-announcement section, is made audible in the cabin while the vehicle is traversing the touring-announcement section.

[0013] In other embodiments, the application suite of the infotainment system includes an itinerary-generating application that simulates a journey by the vehicle along a path from which a point-of-interest is visible and, based on the simulation, to designate a section along the path to be a touring-announcement section. In these embodiments, the touring-announcement section is a section along the path that has the property that, when the vehicle traverses that section, content concerning the point-of-interest is made audible in the cabin as part of the spoken announcement.

[0014] In yet other embodiments, the application suite includes an itinerary-generating application that generates an itinerary and a touring application that accesses the itinerary and causes delivery of the spoken announcement while the vehicle is in a touring-announcement section defined in the itinerary. In such embodiments, the spoken announcement includes information about a point-of-interest that is visible from the touring-announcement section.

[0015] In still other embodiments, the application suite includes a touring application that accesses an itinerary that identifies a path that includes a touring-announcement section from which a point-of-interest is visible, that determines that the vehicle is in the touring-announcement section, and that postpones delivery of information that is scheduled to be delivered in a spoken announcement while the vehicle is in the touring-announcement section.

[0016] Also, among the embodiments are those in which the application suite includes a touring application that accesses an itinerary that identifies a path that includes a touring-announcement section from which a point-of-interest would be visible from the vehicle were the vehicle at the touring-announcement section. In some of these embodiments, the touring application receives certain and causes the touring-announcement section to be altered in response to that information. Embodiments include those in which the information is indicative of traffic conditions and those in which the information is from an occupant-monitoring system.

[0017] Among the embodiments that include an itinerary-generating application that generates an itinerary that includes a path to be traversed by the vehicle are those that control when announcements are permitted to occur. In some of these, the path is divided into sections that are reserved for one type of announcement only, in which case all other types are prohibited. In some of these embodiments, the choice depends on cognitive load, with announcements of one or both types being prohibited in sections that have high cognitive load.

[0018] Embodiments include those in which an itinerary-generating application generates an itinerary that includes a path having section that impose different cognitive loads on the driver of a vehicle as it traverses the path. In such cases, the itinerary-generating application avoids scheduling announcements during periods of high cognitive load. Thus, if a path has a first section having a first cognitive load and a second section having a second cognitive load that is less than the first, then given a choice, the application schedules the announcement during the second section. In some cases, the section during which the spoken announcement is permitted is an intersection-free section that is between two intersections. In other cases, the section is one having a sufficiently high radius-of-curvature. Thus, given two sections with different radii-of-curvature, the application will choose that having the larger of the two radii.

[0019] In other embodiments, an application designates a section of a generated path as being suitable for a touring announcement that includes information concerning a point-of-interest if that point-of-interest is visible from that section of the path. In others, the section designated as suitable is one for which a ray that extends between the point-of-interest and any point on that section does not pass through an obstruction. Conversely, sections designated as unsuited to the announcement are those in which the ray passes through an obstruction. In still other embodiments, prevailing weather conditions are taken into account. This is useful since visibility, in many cases, depends on weather. In such cases, prevailing weather conditions are used to identify a section of the path from which a point-of-interest is visible and to cause a touring announcement concerning the point-of-interest to be made while the vehicle is traversing that section of the path. In still other embodiments, the application takes into account time-of-day to identify a section of the path from which a point-of-interest is visible and to cause a touring announcement concerning the point-of-interest to be made while the vehicle is traversing that section of the path.

[0020] Also, among the embodiments are those in which an itinerary-generating application generates an itinerary that includes a path that comprises muted sections and unmuted sections. These sections are so designated based on constraints on delivering the touring announcement. In such embodiments, a touring announcement is permitted only while the vehicle is traversing an unmuted section.

[0021] In still other embodiments, an itinerary-generating application generates an itinerary that includes a path having a section in which there exists some impediment to delivery of the touring announcement. In such embodiments, the application permits a touring announcement to be made only when the vehicle is outside of this section.

[0022] Also, among the embodiments are those in which the automotive processing unit includes a navigator that has calculated a path between an origin and a destination and a detouring application that provides, after the vehicle has begun traversing the path, real-time touring announcements concerning points-of-interest. These points-of-interest are invisible from the calculated path.

[0023] In other embodiments, a detouring application detects a vehicle's approach to a decision point on a calculated path. This decision point has an associated salient alternative and a standard alternative. Upon determine that the salient alternative has been selected, the detouring application consults a feature database concerning the salient alternative and offers the driver one or more guided tours through the salient alternative.

[0024] In alternative embodiments, a detouring application detects that the vehicle has left a calculated route between an origin and a destination offers the driver a guided detour from among one or more guided detours that begin where the driver left the calculated route.

[0025] Also, among the embodiments are those in which a detouring application detects that a vehicle has left a calculated route between an origin and a destination, offers the driver a guided detour from among one or more guided detours that begin where the driver left the calculated route, and infers a selection of the guided detour based on a course steered by the driver.

[0026] In other embodiments, the application suite includes a detouring application that accesses data that identifies decision points along a path between an origin and a destination. At each of these decision points to choose between a standard alternative and a salient alternative. The standard alternative is one in which the vehicle remains on the path and the salient alternative is one in which the vehicle does not remain on the path. It stands to reason that there can only be one standard alternative and that there can be one or more salient alternatives. For example, in a four-way intersection of two-way streets, assuming no U-turn, there would be two salient alternatives.

[0027] Among the embodiments with a detouring application are those in which the detouring application dynamically creates decision points along a path being traversed by the vehicle. Each decision point provides a driver of the vehicle with an opportunity to choose between at most one standard alternative and one or more salient alternatives.

[0028] In still other embodiments, a detouring application interprets the vehicle's the direction, or orientation, of the vehicle's velocity vector and relies on it as a basis for inferring a selection, by a driver of the vehicle, a particular guided detour.

[0029] Yet other embodiments include those in which a detouring application poses a question to a driver of a vehicle upon detecting that the vehicle has left a calculated path between an origin and a destination and to interpret an acceleration of the vehicle as an answer to the question.

[0030] In some embodiments, the application suite comprises a touring application that is configured to receive an origin and destination and to identify a tour that comprises a path and an interest score. The path extends between the origin and destination and wherein the interest score depends at least in part on one or more points-of-interest along the path.

[0031] Among these are embodiments in which the touring application consults a point-of-interest database that includes information indicative of points-of-interest between the origin and the destination and that is located at the vehicle. In other embodiments, the touring application also consults a remote point-of-interest database that is maintained by a remote server. In such cases, the apparatus further comprises a network interface to provide communication between the touring application and the remote server. Among these are embodiments in which the remote server dynamically updates the remote point-of-interest database based on queries from occupants of a population of vehicles. As a result, it is possible to identify trending points-of-interest.

[0032] A variety of ways are available to determine a tour's interest score. In some embodiments, an interest score depends on how many points-of-interest are on the tour. In such embodiments, a score is assigned based on a first point-of-interest on the tour and then modified based on a second point-of-interest along the tour. As a result, the interest score depends, at least in part, on a number of points-of-interest in the tour. Another way to determine interest score is to assign an interest score to the tour based on a weighted sum of scores associated with each of a plurality of points-of-interest in the tour.

[0033] In some embodiments, too low an interest score disqualifies a path from rising to the level of a tour. In such embodiments, a touring identifies a tour between an origin and a destination only if that tour's interest score is above a threshold interest score.

[0034] Also, among the embodiments are those in which the application suite comprises a first touring application that is a constituent of a distributed touring system that comprises a second touring application and a remote server. The second touring application executes on a portable device. remote server switches seamlessly between the first and second touring applications thereby enabling a tour to include first and second tour segments. The first tour segment is one in which the remote server communicates with the first touring application. The second tour segment is one in which the remote server communicates with the second tour segment. A suitable portable device is a smart phone or portable telephone. This seamless switching permits a tour to be carried out via a vehicle and then on foot.

[0035] A variety of ways are available for causing the transition between the two tour segments. In some embodiments, the remote server receives information indicating that a mode of transportation of an occupant of the vehicle has changed, the occupant being one who carries the portable device. In others, the remote server receives information indicating that the portable device has moved independently of the vehicle or that a connection between the portable device and infotainment system has been interrupted. Either of these would be likely to occur if the vehicle has been parked and the occupant, who is carrying the portable device, has left the vehicle, and begun walking.

[0036] In some of the foregoing embodiments, the remote server provides data to the first touring application at a pace that is consistent with the velocity of the vehicle and provides data to the second touring application at a pace that is consistent with the velocity of the portable device.

[0037] Also, among the embodiments are those in which remote server transitions from providing data to the first tour-guide to providing data to the second tour-guide upon receiving information indicative of occurrence of a first event, and the remote server transitions from providing data to the second touring-application to providing the data to the first touring-application upon receiving information indicative of occurrence of a second event.

[0038] The method and apparatus as described and claimed herein are non-abstract. No description has been offered of any abstract implementations. Accordingly, the claims are to be construed as covering only non-abstract subject matter. Any person who construed them otherwise would be construing them incorrectly and without regard to the specification.

[0039] Applicant, acting as his own lexicographer, hereby defines “non-abstract” as the complement of “abstract” as that term has been defined by the courts of the United States as of the filing date of this application.

[0040] The method and apparatus as described herein is also one that has a technical effect. In many cases, the technical effect will be non-obvious. However, it exists. Therefore, any person who construes the claims as lacking a technical effect is merely displaying an inability to discern the technical effect as a result of its non-obviousness.

[0041] The processing system that executes the method is not a generic computer. It is a specialized digital electronic device that is specially adapted for operation in a passenger vehicle to accommodate the various technical constraints imposed by that environment including limited space and power and operation in environments of extreme heat or cold and resistance to shock and other forces that result from acceleration.

[0042] Additionally, though it is convenient to implement the method using software instructions, it is known that virtually any set of software instructions can be implemented by specially designed hardware, which is typically provided as an application-specific integrated circuit. The claims presented herein are also intended to cover such an implementation.

[0043] Other features and advantages of the invention are apparent from the following description, and from the claims.

[0060] FIG. 1 shows a vehicle 10 having a passenger cabin 12 that carries occupants 14. Among the occupants 14 is a driver 16 who operates the vehicle 10. In those embodiments in which the vehicle 10 is autonomous, no driver 16 is necessary.

[0061] Each occupant 14, 16 is seated in a seat 18. Each seat has an associated microphone 20 and a loudspeaker 22.

[0062] The vehicle 10 also includes one or more internal cameras 24. The internal cameras 24 are directed towards various portions of the cabin 12. An internal camera 24 therefore often has an occupant 14, 16 in its field of view.

[0063] The vehicle 10 also includes one or more external cameras 26. The external cameras 26 are directed towards the vehicle's environment. These external cameras 26 are therefore useful for identifying features in the vehicle's surroundings.

[0064] In those embodiments in which the vehicle 10 requires a driver 16, the internal cameras 24 and the microphones 20 communicate with an occupant monitor 28 that evaluates the driver's state. In some embodiments, the occupant monitor 28 does so by recognizing the driver's gaze direction or otherwise does so by collecting information indicative of the driver's receptiveness to delivery of information. The vehicle 10 further includes an on-board automotive-processing unit 30, details of which can be seen in FIG. 2.

[0065] Referring now to FIG. 2, the automotive-processing unit 30 features a navigator 32 that receives signals, processes them to output spatial coordinates, and displays a representation of the vehicle's spatial environment on a map selected from a map database 34. In addition, the navigator 32 has access to a feature database 36 that provides information concerning points-of-interest corresponding to various spatial coordinates.

[0066] The automotive-processing unit 30 also includes an infotainment system 38 having a head unit 40. The head unit 40 comprises a processor 42 and a memory 44. The infotainment system 38 interfaces with a peripheral-device set that includes one or more peripheral devices. Examples of peripheral devices include one or more of: the microphones 20, the loudspeakers 22, and the internal and external cameras 24, 26.

[0067] Referring now to FIG. 3, an automotive assistant 46 executing in the head unit 40 functions as the head unit's operating system. The automotive assistant 46 thus enables the head unit 40 to execute various applications among which is a speech interface 48. The speech interface 48 receives navigation announcements 49 from the navigator 32 and makes them audible in the cabin 12 through the loudspeaker 22. A navigation announcement 49 alerts the driver 16 to forthcoming changes in course.

[0068] The automotive assistant 46 also permits execution of one or more spatially-cognizant applications that are part of an application suite 50. Such applications are said to be “spatially cognizant” because their operation depends at least in part on information received from the navigator 32. The speech interface 48 receives touring announcements 51 from any of the applications in the application suite 50 and makes those touring announcements audible in the cabin 12 through the loudspeaker 22.

[0069] In some embodiments, the speech interface 48 uses the same voice to deliver navigation announcements 49 and touring announcements 51. In other embodiments, the speech interface 48 delivers touring announcements 51 using a voice that differs from that used to deliver navigation announcements 49.

[0070] As used herein, an “announcement” refers to information and not the particular form into which the information has been encoded. In their final form, the navigation announcements 49 and touring announcements 51 are made audible in the cabin 12 using the loudspeaker 22 and are hence encoded as sound waves. However, both existed prior to being made audible, albeit in different forms. Throughout operation of the infotainment system 38, the navigation announcements 49 and touring announcements 51 will manifest themselves in different forms while retaining their information during and after such transformations. Accordingly, both the navigation announcements 49 and touring announcements 51 will be referred to without reference to the form in which they exist and with the understanding that the form of an announcement is mutable even though the announcement itself remains the same.

[0071] Referring now to FIG. 4, a detouring application 52 from the application suite 50 defines a path 54 through a network of roads. In those cases in which the navigator 32 is in use, the detouring application 52 uses, as the defined path 54, the route computed by the navigator 32 between the vehicle's origin and its destination. In other cases, for example when the navigator 32 is not being used, the detouring application 52 uses, as the path 54, the particular road along which the vehicle 10 is traveling at the time.

[0072] A typical path 54 inherently has decision points 56 at which an opportunity to deviate from the path 54 exists. Examples of a decision point 56 include a highway exit or an intersection between roads. In some embodiments, the detouring application 52 accesses data that identifies decision points 56. In others, the detouring application 52 dynamically identifies decision points 56.

[0073] At each decision point 56, the driver 16 can choose to stay on the path 54 or leave the path 54. Thus, each decision point 56 has at least two alternatives: a standard alternative 60, in which the driver 16 chooses to remain on the path 54, and a salient alternative 62, in which the driver 16 chooses to leave the path 54.

[0074] As shown in FIG. 3, the detouring application 52 is in communication with detour data 64. Each decision point 56 is associated with a subset of the detour data 64. When the driver 16 chooses the salient alternative 62, the detouring application 52 uses the identity of the decision point 56 to consult a corresponding portion of the detour data 64.

[0075] The navigator 32 provides location information to the detouring application 52. The detouring application 52 is thus able to detect whether the vehicle 10 is either already on the salient alternative 62 or about to be on the salient alternative 62. Such detection can be carried out in a variety of ways. One method relies on geo-fencing. Another method relies on detecting a deviation from the path 54 by more than a threshold amount. Additional embodiments include those in which the detouring application 52 detects the choice of the salient alternative 62 using other clues, such as by detecting the operation of a turn signal just prior to a scheduled arrival at a decision point 56. Since one does not normally use the turn signal when staying on the path 54, operation of the turn signal is indicative of the driver's intent to deviate from the path 54.

[0076] The detour data 64 for a particular decision point 56 includes information drawn from the feature database 36 shown in FIG. 2. Such information includes information about attractions or points-of-interest in the vicinity. Such information also includes suggested detours, such as a long detour 66, a scenic detour 68, and a short detour 70 shown in FIG. 5, each of which begins at the decision point 56 and returns to the path 54, preferably downstream of the decision point 56.

[0077] In a process 72 as shown in FIG. 6, a vehicle 10 at a decision point 56 takes the exit that is marked for a particular national park (step 74). In response, the detouring application 52 detects the choice of the salient alternative 62 (step 76) and retrieves detour data 64 relevant to the national park (step 78). The detouring application 52 may then provide the driver with choices of guided detours through the national park, as shown in FIG. 5 (step 80).

[0078] The detouring application 52 then receives the driver's selection of a detour (step 82). The driver does so by either using the speech interface 48 or implicitly by turning in a direction that is consistent with the beginning of a particular detour. Then, the detouring application 52 provides suitably interesting narration delivered at appropriate spatial locations along the detour (step 84).

[0079] Referring back to FIG. 3, in some embodiments, the application suite 50 includes an itinerary-generating application 84 and a touring application 86. The itinerary-generating application 84 creates an itinerary 85 which the touring application 86 then uses as a basis for a tour of a region, a portion of which is shown in FIG. 7.

[0080] Like FIG. 4, FIG. 7 shows a path 54 to be traversed by the vehicle 10. As the vehicle 10 proceeds along the path 54, one or more point-of-interest 88 will come into view. During certain announcement periods 90, the touring application 86 delivers content concerning the point-of-interest 88, in spoken form, for the benefit of the vehicle's occupants 14.

[0081] Prior to the beginning the road trip, the itinerary-generating application 84 schedules the announcement periods 90 during which the touring application 86 is permitted to deliver content, i.e., to make a touring announcement 51. This schedule of announcement periods 90 becomes part of the itinerary 85.

[0082] The itinerary-generating application 84 faces certain complications when scheduling the announcement periods 90. These complications arise because the touring application 86, in addition to knowing about the various points-of-interest 88, faces certain constraints that exist along the path 54.

[0083] A first constraint is a visibility constraint. The point-of-interest 88 should be easily visible during the announcement period 90. It would not do to have an announcement period 90 while the point-of-interest 88 is receding into the distance or when the point-of-interest 88 is not even visible. Ideally, the vehicle 10 should be approaching the point-of-interest 88 or passing it so that the occupants 14 (and most especially, the driver 16) do not have to turn around to see it.

[0084] A second constraint is an interference constraint. The announcement period 90 must not interfere with speech carried out by any concurrently executing applications from the application suite 50 or elsewhere. For example, if the navigator 32 is in the midst of telling the driver 16 to make a turn, the touring application 86 should not barge in to discuss the historical importance of a nearby burial mound.

[0085] A third constraint is a cognitive constraint. As the vehicle 10 traverses the path 54, there will be certain periods that require high cognitive load. Such periods may occur, for example, while the vehicle 10 is negotiating a complex highway interchange or a rotary. For the touring application 86 to add to this cognitive load by delivering a lecture concerning some nearby point-of-interest 88 would, again, be poor form.

[0086] A fourth constraint is that the announcement period 90 inherently has some an announcement duration 92. The touring application 86 must have an announcement period 90 that is at least as long as the announcement duration 92. Otherwise, it will not be able to deliver its content without interruption. Thus, there must be a continuous section of the path 54 that is free of any other constraint for an interval that accommodates the announcement duration 92.

[0087] It is useful to consider the path 54 as being divided into muted sections and unmuted sections. When the vehicle 10 is in a muted section, the touring application 86 is forbidden from speaking. When the vehicle 10 is in an unmuted section, the touring application 86 is permitted to speak. The itinerary generator's challenge is to identify appropriate unmuted sections and to schedule the announcement periods 90 accordingly.

[0088] To assist it in scheduling announcement periods 90, the itinerary-generating application 84 associates each point-of-interest 88 with a footprint 94. As shown in FIG. 7, the path 54 goes through footprints 94 of various points-of-interest 88.

[0089] The size and shape of a footprint 94 depend on the prominence of its corresponding point-of-interest 88. In some embodiments, the footprint 94 is such that the point-of-interest 88 will be visible from a vehicle 10 that is within the footprint 94. Thus, a volcano rising from an otherwise flat plain will have a large footprint 94 whereas a graveyard tucked between skyscrapers in a city center will have a relatively small footprint 94.

[0090] The footprint's geometry also depends on weather. For instance, clear weather will tend to enlarge a footprint 94 whereas foggy weather or blizzard conditions will tend to shrink the footprint 94.

[0091] The footprint's geometry also depends on time-of-day. For example, the position of the sun may generate glare that makes the point-of-interest 88 difficult to see from certain angles at certain times-of-day. At night, many outdoor points-of-interest may be difficult to see unless either bright moonlight or intense boreal activity renders them visible, at least in silhouette.

[0092] The footprint 94 thus defines a region during which the point-of-interest 88 is likely to be visible. The itinerary-generating application 84 therefore attempts to schedule an announcement period 90 for that point-of-interest 88 while the vehicle 10 traverses its footprint 94.

[0093] The trip planning process thus includes a planning step in which the itinerary-generating application 84 designates each section on the path 54 as being muted or unmuted. This is a largely deterministic exercise that can be carried out before the tour actually begins.

[0094] For example, once a path 54 has been defined, it is possible to predict, with considerable accuracy, where the navigator 32 will be required to give directions. It is also possible to recognize, based on road topology and time-of-day, those portions of the path 54 that are likely to require significant cognitive load to negotiate. Additionally, if the tour is not too far in the future, weather information is available to assist in determining the sizes of the footprints 94.

[0095] FIG. 7 illustrates a simple scheduling exercise in which the path 54 features a first high-visibility section 96 and a second high-visibility section 98. The point-of-interest 88 is thus clearly visible on the vehicle's right from anywhere along these high-visibility sections 96, 98. As a result, absent any other constraints, both the first and second high-visibility sections 96, 98 would be suitable for an announcement period 90.

[0096] However, as a result of an imminent sharp turn to the right, a navigation-announcement section 100 that has been allocated to a navigation announcement 49 overlaps a significant portion of the first high-visibility section 96. What is left of the first high-visibility section 96 is too short for the touring application 86 to make all but the most perfunctory of announcements. After confirming that this remaining period is too short for the touring application 86 to deliver its content, the itinerary-generating application 84 rules out the first high-visibility section 96.

[0097] A high-cognitive-load section 102 spans much of the sharp right turn and overlaps a first portion of the second high-visibility section 98. The first portion of the second high-visibility section 98 is thus a muted section that is unavailable to the touring application 86. However, a second portion of the second high-visibility section 98 is unmuted. The itinerary-generating application 84 confirms that this second portion is long enough for the touring application 86 to deliver its content. Accordingly, the itinerary-generating application 84 schedules the second portion of the second high-visibility section 98 for this purpose.

[0098] FIG. 8 shows a portion of the path 54 seen in FIG. 7 but after having undergone a transformation that renders the path 54 as straight and causes the overlaps of the high-cognitive-load section 102 and the navigation-announcement section 100 with the high-visibility sections 96, 98 to be seen without unnecessary distraction.

[0099] To accommodate unexpected events that may occur during the tour, it is useful to permit the touring application 86 to execute short-term overrides after the tour has begun. For example, if traffic significantly slows down the vehicle's progress, a section of the path 54 that was previously deemed too short for an announcement period 90 can, as a result of the vehicle's reduced velocity, become long enough to accommodate the announcement period 90. The touring application 86 recognizes that, as a rule, a driver 16 sustains little cognitive load when stuck in traffic. Accordingly, the touring application 86 moves a scheduled announcement period 90 from a later section of the path 54 to this newly available section. This has the added benefit of entertaining the driver 16 at a time when the driver 16 is in most need of such entertainment.

[0100] In some cases, the driver 16 makes an unexpected turn. Depending on what the driver 16 does, this may require the touring application 86 to make significant adjustments that include moving and possibly even cancelling announcement periods 90.

[0101] In other cases, unexpected developments may substantially increase the driver's cognitive load. For example, just prior to a scheduled announcement period 90, an infant passenger may begin to wail or the driver 16 may become involved in a heated conversation with other occupants 14 in the vehicle 10. The above-mentioned occupant monitor 28, which has been unobtrusively observing the driver 16 all along, recognizes the telltale signs of such cognitive load and signals the touring application 86, which then postpones the scheduled announcement period 90 for a more opportune time.

[0102] Referring now to FIG. 9, a planning process 104 carried out by the itinerary-generating application 84 begins with the receipt of an origin and destination (step 106). This causes the path 54 to be determined (step 108). It also permits the completion of tasks that are independent of the points-of-interest 88, namely scheduling of navigation announcement 49 (step 110) and identifying high cognitive-load sections 102 of the path 54 (step 112).

[0103] The planning process 104 continues with the receipt of one or more points-of-interest 88 (step 114). This permits the identification of high-visibility sections 96, 98 along the path (step 116) and the scheduling of announcement periods 90 (step 118). The process 72 continues with a check to see if more points-of-interest 88 have been identified (step 120). If so, the loop is repeated. If not, the process ends (step 122).

[0104] Referring back to FIG. 3, the application suite 50 also includes a tour-guide application 124. The tour-guide application 124 receives current-location information from the navigator 32 and uses it to recognize when a point-of-interest 88 is sufficiently close to make a touring announcement 51. Upon recognizing the proximity of a point-of-interest 88, the tour-guide application 124 retrieves touring content from a touring database 126. The touring content comprises touring text that is to be spoken by the speech interface 48 and touring audio, such as music or sound effects. Together, these form the basis of a touring announcement 51.

[0105] A certain urgency is inherent in delivering a navigation announcement 49. To promote more effective execution of navigation instructions, the navigation announcement 49 should occur far enough in advance of a turn to be safely acted upon, but not so far that the driver 16 is likely to forget to act on it. Moreover, in complex situations, such as highway interchanges, the navigator 32 may deliver multiple navigation announcements 49 in rapid-fire succession.

[0106] Similarly, a touring announcement 51 is preferably delivered in a timely fashion. Ideally, the touring announcement 51 is delivered while the point-of-interest 88 is in view and certainly before a new point-of-interest has been reached.

[0107] Since navigation announcements 49 and touring announcements 51 must both be delivered in a timely fashion, a conflict is possible.

[0108] To permit navigation announcements 49 and touring announcements 51 to be delivered together, the speech interface 48 includes a natural-language engine 130 having a mixer 132 and a summarizer 134.

[0109] The speech interface 48 intercepts both the touring announcement 51 from the application suite 50 and the navigation announcement 49 from the navigator 32. The navigation announcement 49 is provided to the mixer 132. If no touring announcement 51 is also provided, the mixer 132 does nothing but forward the navigation announcement 49 to a text-to-speech converter 136. This results in the navigation announcement 49 being made audible within the cabin 12 over the loudspeaker 22.

[0110] Similarly, if no navigation announcement 49 is provided, then the touring announcement 51 is provided to the summarizer 134. The summarizer 134 carries out a summarization, if necessary, and forwards the resulting summarized touring announcement 51 to the mixer 132. The mixer 132 has no mixing to do and hence does nothing but forward the touring announcement 51 to the text-to-speech converter 136. This results in a touring announcement 51 being made audible within the cabin 12 over the loudspeaker 22.

[0111] In some cases, the mixer 132 receives both a navigation announcement 49 and a touring announcement 51. In such cases, the mixer 132 generates a hybrid announcement 138 that incorporates elements of both the navigation announcement 49 and the touring announcement 51. The mixer 132 then provides this hybrid announcement 138 to the text-to-speech converter 136. The hybrid announcement 138 is then made audible within the cabin 12 via the loudspeaker 22.

[0112] A hybrid announcement 138 simulates a single person who knows how to navigate while smoothly delivering a running commentary on points-of-interest along the way. This results in an integrated experience that feels, to the driver 16, like a knowledgeable companion who provides driving directions and talks about the various sights in between.

[0113] To generate the hybrid announcement 138, the natural-language engine 130 recognizes locations into which navigation announcement 49 can be unobtrusively inserted into touring announcement 51. Such indicators include certain punctuation marks or boundaries between certain semantic units, such as sentences or clauses, including dependent clauses, independent clauses, and subordinate clauses. The natural-language engine 130 also causes the mixer 132 to carry out the insertion of certain bridging words to create a smoother transition between touring announcement 51 and navigation announcement 49, essentially spackling over the seams that would otherwise be apparent to the driver 16 and more closely simulating the speech patterns of an actual person.

[0114] As an example, the vehicle 10 may be passing by a prominent church. As a result, the tour-guide application 124 would provide touring announcement 51 that says,

• • [0115] “The church on your left, with its two towers, one of which was replaced after a fire in 1985, was built in 1820.”

[0116] However, at about the same time, the vehicle 10 may also be approaching an intersection at which a right turn is required. As a result, the navigator 32 would provide the navigation announcement 49:

• • [0117] “In a quarter mile, turn right.”

[0118] Upon receiving the navigation announcement 49 and the touring announcement 51, the natural-language engine 130 causes the mixer 132 to insert the navigation announcement 49 into the touring announcement 51, thereby producing the hybrid announcement 138. An example of such hybrid announcement 138 would be:

• • [0119] “The church on your left, with its two towers, one of which was replaced after a fire in 1985, was built in 1820. By the way, you need to turn right in about a quarter mile. I'll tell you more about that statue in the church courtyard after you've turned.”

[0120] In forming the hybrid announcement 138, the natural-language engine 130 caused the mixer 132 to do more than simply concatenate the touring announcement 51 and the navigation announcement 49. It also caused the insertion of a syntactic cue, namely the words “By the way.” Such a cue draws the driver's attention to the directions, thus reducing the possibility that they will be overlooked.

[0121] A syntactic cue may prove insufficient. As a result, some embodiments offer the option of a more pronounced audio cue. Examples include those in which the mixer 132 marks the navigation announcement 49 to be delivered in a different voice or at a different volume to draw attention. Additional examples include an audible beep to alert the driver 16. However, using the same voice and volume for both the navigation announcement 49 and the touring announcement 51 is generally regarded as presenting a less jarring experience.

[0122] The natural-language engine 130 invokes the summarizer 134 when, as a result of exigent circumstances, such as an approaching turn, there is not enough time to deliver the touring announcement 51 in its entirety. The summarizer 134 identifies subjects and verbs in the touring announcement 51 so that the grammatical form can be recognized.

[0123] The summarizer 134 then deletes superfluous words in an effort to deliver the same information in less time. This is particular useful if the navigation announcement 49 is unusually long or if multiple instances of navigation announcement 49 are being provided.

[0124] In some embodiments, it is possible to adjust the overall length of the hybrid announcement 138 by having the text-to-speech converter 136 generate slightly shorter or longer syllables and slightly shorter or longer gaps between words and phrases. It has been found that variations of no more than ten percent in the overall length of an announcement are barely perceptible to the average listener.

[0125] As noted above, the touring announcement 51 can, in some cases, include touring audio or sound effects. In such cases, the infotainment system 38 includes a time scaler 140 that shortens or lengthens the touring audio to conform to a hybrid announcement 138 of the desired duration. The touring audio is then provided to a summer 142 so that it can be included as part of the hybrid announcement 138.

[0126] FIG. 10 shows a process 144 carried out by the speech interface 48 that begins with asking whether any text has been received from a source 124, 32 (step 146) and if so, whether any text has been received from another source 32, 124 (step 148). If not, the text from the first source 124 is simply provided to the text-to-speech converter 136 (step 150). Otherwise, the speech interface 48 composes hybrid announcement 138 using the text from the two sources 124, 32 (step 152) and provides this hybrid announcement 138 to the text-to-speech converter 136 (step 154).

[0127] Referring back to FIG. 3, embodiments further include those in which the application suite 50 features a touring application 156. The touring application 156 accesses local point-of-interest database 158. The touring application 156 operates in two different modes: a guide mode and an exploratory mode.

[0128] In the guide mode, the touring application 156 provides spoken instructions that guide the vehicle's driver 16 past various points-of-interest 88. As it does so, it uses the speech interface 48 to provide touring announcements 51. The touring announcements 51 feature narration calculated to hold the driver's interest.

[0129] Referring now to FIG. 11, the exploratory mode is intended to address a driver 16 who intends to travel between an origin 160 and destination 162. However, rather than travel via the shortest route, the driver 16 may have an urge to explore the landscape along the way. This driver 16 would thus like to know interesting places can be seen along the way without excessively compromising the length of planned journey between the origin 160 and the destination 162.

[0130] In the exploratory mode, the touring application 156 receives the driver's choice of an origin 160 and destination 162 and provides one or more diversions that encompass points-of-interest 88 that may interest the driver 16. This choice of origin 160 and destination 162 defines an excursion region 164 that extends is proximate to both the origin 160 and destination 162. As used herein, “proximate” also includes the case in which the excursion region 164 includes one or both of the origin 160 and the destination 162.

[0131] Typically, this excursion region 164 encompasses one or more paths 54. These paths 54 are those that would have been calculated by the navigator 32 in response to a request by the driver 16 for paths between the origin 160 and the destination 162.

[0132] The topology of the excursion region 164, and hence the meaning of “proximate,” depends on the driver's desire to avoid lengthening the planned voyage by an excessive amount. Thus, in some embodiments, the driver 16 also specifies the maximum extent by which a diversion would extend the planned voyage between the origin 160 and destination 162. A driver 16 who is has the time for a lengthy excursion would thus be presented with a larger excursion region 164. In contrast, a driver 16 who is in somewhat of a hurry may be presented with an excursion region 164 that takes the form of a narrow strip along the shortest path 54 between the origin 160 and the destination 162.

[0133] In defining the excursion region 164, the touring application 156 consults the local point-of-interest database 158 to identify points-of-interest 88 and the distances of those points-of-interest 88 from a calculated path 54. excursion region 164. Having done so, the touring application 156 cooperates with the navigator 32 to identify one or more tours 166 that traverse the excursion region 164, as shown in FIG. 12.

[0134] Each tour 166 comprises a path 54 in combination with an interest value that is associated with that path 54. The interest value depends on the distribution of points-of-interest 88 along the path 54. For example, a lengthy tour 166 whose sole point-of-interest 88 is a toxic waste site from a famous environmental disaster may have a relatively low interest value. In contrast, a short tour 166 along a brightly lit street lined with casinos may have a higher interest value. Based on their respective interest values, it is possible to rank tours 166 and to invite the driver 16 to choose one of the available tours 166 between the origin 160 and the destination 162.

[0135] In some embodiments, the touring application 156 displays the highest-ranking tour 166 and invites the driver to select it. In other embodiments, the touring application 156 provides information that has been selected to increase the probability that the driver 16 will select the tour 166. Other embodiments are those in which the touring application 156 displays a selection of tours 166 and invites the driver 16 to choose one of them.

[0136] In either case, once the driver 16 selects a tour 166, the touring application 156 begins operating in guide mode.

[0137] Referring now to FIG. 13, a rating method 168 carried out by the touring application 156 begins by determining the various paths 54 between the origin and the destination (step 170). For each such route, the touring application 156 identifies points-of-interest 88 along that route (step 172) and determines, for each such point-of-interest 88, a score (step 174). Once all the scores for the points-of-interest 88 have been determined, the touring application 156 adds them together to obtain the route's interest value (step 176). It then proceeds to rank them by interest value (step 178) for presentation to the user. In some practices, the touring application 156 excludes those routes that have an interest value below a pre-defined threshold.

[0138] FIG. 14 shows an implementation of the rating method 168 shown in FIG. 13 for calculating interest values for routes. The rating method 168 begins by setting an index to the first route (step 180) and initializing the interest value (step 182). An index is then set to inspect the first element of an array that holds the points-of-interest 88 on the route (step 184). If the first element is empty, the process inspects the next route (step 186). If it is not empty, the process increments the score for that route by the score of the point-of-interest 88 (step 188) and increments the index for the array that holds the points-of-interest 88 (step 190).

[0139] If no points-of-interest 88 remain on that route, the index for the routes is incremented (step 192) and a determination is made concerning whether there are any more routes (step 194). If there are, the procedure repeats (step 196). Otherwise, the routes are ranked (step 198).

[0140] Whether or not a point-of-interest 88 is considered along the path 54 depends in part on the prominence of the point-of-interest 88 and its distance from the route. For example, a large volcano may be considered to be along the path 54 even though it is dozens of miles from the actual path 54 simply because it is so prominent. In contrast, a sinkhole that is dozens of miles from the path 54 would most likely not be considered along the route because it cannot easily be seen from the path 54.

[0141] In some methods, the points-of-interest 88 all have the same score. Other embodiments include assigning different weights to different points-of-interest 88 and summing the weighted points-of-interest 88 along the path 54.

[0142] Also, among the embodiments are those that define a threshold interest value before a path 54 can merit identification as a tour 166. This eliminates the need to display paths 54 that have points-of-interest 88 that would not be expected to arouse nearly as much interest as other points-of-interest 88.

[0143] Also, among the embodiments are those that place an upper bound on the overall length of the path 54, thereby ensuring that proposed tour 166 does not excessively lengthen the journey between the origin 160 and the destination 48.

[0144] Referring now to FIG. 15, in some embodiments, the infotainment system 38 connects to a network interface 200 that communicates with a remote navigation server 202 in the cloud 204. The navigation server 202 maintains a remote point-of-interest database 208 that, like the local point-of-interest database 158, provides information concerning points-of-interest 88.

[0145] The remote navigation server 202 provides services to a population of vehicles, among which is the illustrated vehicle 10. Among these services is that of answering questions posed by occupants 14 of those vehicles concerning various features that those occupants 14 have encountered in their respective environments.

[0146] As an example, the remote navigation server 202 may find that it has received queries from occupants 14 of many vehicles 10 about an otherwise unremarkable landmark. Based on that vehicle's known location and any cues provided by the occupant 14 concerning the landmark of interest, the navigation server 202 infers the identity of this landmark that so many occupants 14 have inquired about. After having retrieve whatever information is available concerning that landmark, the navigation server 202 then communicates that information to the occupants 14 that have inquired.

[0147] As a result of having provided such services to a community of vehicles 10, the navigation server 202 is in the unique position of being able to detect a surge in interest in this otherwise unremarkable landmark. Based on such a surge in interest, the navigation server 202 would reasonably infer that this otherwise unremarkable landmark has become, for whatever reason, a legitimate point-of-interest 88. Since the navigation server 202 also maintains the remote point-of-interest database 208, it is able to use this information to update the remote point-of-interest database 208 to include this newly discovered point-of-interest 88 so that it can be included in a tour 166 when the touring application 156 operates in exploratory mode. As a result, it is possible to dynamically create new points-of-interest 88.

[0148] Conversely, the navigation server 202 detects a drop in the frequency of inquiries concerning the newly discovered point-of-interest 88. Accordingly, the navigation server 202 infers that this previously unremarkable landmark has, once again, become unremarkable. As such, the navigation server 202 removes it from the remote point-of-interest database 208.

[0149] In effect, this landmark has behaved in a manner analogous to a nova. It begins in obscurity. A surge in interest from occupants 14 of many vehicles 10 lifts it out of obscurity. As a result, this landmark becomes a point-of-interest 88 worthy of inclusion in the remote point-of-interest database 208. Then, after some time, the landmark fades back into obscurity.

[0150] Unlike the local point-of-interest database 158, the remote point-of-interest database 208 is dynamically updated based on the collective activity of the occupants 14 the aggregate vehicle population. As a result, in those cases when the infotainment system 38 is in communication with the navigation server 202, the navigator 32 will have access to two sources of information concerning points-of-interest 88. The local point-of-interest database 158 represents a static source of points-of-interest 88 and the remote point-of-interest database 208 represents a dynamic source of points-of-interest 88 that is continuously updated.

[0151] Referring now to FIG. 16, in some embodiments, the touring application 156 that operates in guide mode communicates with a touring server 210 via the cloud 204. Under such circumstances, the touring server 208 recognizes that the touring application 156 is one that is being executed within the infotainment system 38. Accordingly, the touring server 210 provides the touring application 156 with information consistent with a tour 166 that is being carried out in a moving vehicle 10.

[0152] However, there exist some points-of-interest 88 that the occupant 14 cannot enjoy from the confines of a vehicle 10. For example, in dense urban areas, many points-of-interest 88 abound. Yet, the vehicle 10 becomes a liability rather than an asset. Similarly, there are certain points-of-interest 88 that the vehicle 10 simply cannot reach. These include points-of-interest 88 in wilderness areas that are not accessible by road or points-of-interest 88 that can only be reached by ferry. In such cases, it is convenient for the occupant 14 to continue the tour 166 using a different mode of transportation. Examples of different modes include touring on foot, on a bicycle, on a boat, or via a suitable pack animal such as a horse or mule.

[0153] Of course, having left the vehicle 10 behind, the occupant 14 is technically no longer really an “occupant.” However, for convenience of exposition, and to avoid having to introduce a new noun, the occupant 14 will continue to be referred to as such even after having left the vehicle 10.

[0154] To overcome this difficulty of no longer being in the vehicle 10, and hence no longer being able to use the infotainment system 38, it is useful for the occupant 14 to carry a portable device 212 that executes a second touring application 214. An example of a portable device 212 is a smart phone, as shown in FIG. 16.

[0155] This second touring application 214 operates concurrently with the touring application 156 from the application suite 50. For convenience of exposition, the touring application 156 from the application suite 50 will now be referred to as the “first touring-application 156” to distinguish it from the “second touring-application 214” that executes on the portable device 212. Both the first and second touring applications 156, 214 are in communication with both the touring server 210 and the navigation server 202 and thus form, as a whole, a distributed touring system.

[0156] Since the navigation server 202 receives position information at known times, it is able to infer velocity of both the vehicle 10 and the portable device 212. Accordingly, the navigation server 202 is able to detect that the vehicle 10 has stopped. In addition, the navigation server 202 is also able to detect motion of the portable device 212 using the smart phone's own navigation system. As a result, it is possible for the navigation server 202 to detect zero velocity for the vehicle 10 and a non-zero velocity for the portable device 212. Based on having detected such a systematic difference in velocities, the navigation server 202 infers that whoever is carrying the portable device 212 is likely to be moving independently of the vehicle 10 using a different mode of transportation. Accordingly, the navigation server 202 provides this information to the touring server 210.

[0157] Other embodiments rely on different information to infer that the tour 166 is to be continued on the portable device 212. Examples of such different information include detecting that the vehicle 10 has been turned off and detecting that a wireless link between the portable device 212 and the head unit 24 has been interrupted.

[0158] Based on this information, the touring server 210 automatically adjusts the tour 166 so as to be consistent with the velocity of the portable device 212. For example, if the velocity of the portable device 212 indicates a walking pace, the touring server 210 switches to directions and narrative that are geared specifically for an occupant 14 who is on foot. Should the touring server 210 receive information from the navigation server 202 indicating that the average velocity of the portable device 212 has increased to that associated with a bicycle, the touring server 210 then switches to directions and narrative that are geared specifically for an occupant 14 who is on a bicycle.

[0159] Eventually, the occupant 14 will return to the vehicle 10. Should the touring server 210 receive information from the navigation server 202 indicating that the average velocity of the portable device 212 has increased to that consistent with a motor vehicle 10, the touring server 210 then switches to directions and narrative that are geared specifically for an occupant 14 who is in a motor vehicle 10. Concurrently, the first tour-guide 156 takes over control from the second tour-guide 214. As a result, the tour 166 continues, but using the first tour-guide 156.

[0160] Accordingly, the touring server 210 and the navigation server 202 cooperate to infer which of the first and second tour-guides 156, 214 is in control of the tour 166 and to switch seamlessly between the first and second tour-guides 156, 214.

[0161] It is to be understood that the foregoing description is intended to illustrate and not to limit the scope of the invention, which is defined by the scope of the appended claims. Other embodiments are within the scope of the following claims.