Automotive virtual personal assistant

[0001] As internet-connected devices are now ubiquitous, the development of technology focused at the mobile user has rapidly accelerated. Nowhere, in recent years, is this more evident than the automobile as manufacturers seek to provide users with an improved and connected experience. Currently, vehicles are capable of ascertaining the status of various sensors placed throughout the vehicle cabin and on the exterior of the vehicle, enabling generic identification of an open window, an unfastened seatbelt, the distance between vehicles, etc. In some vehicles, an advanced driver assistant system can aid the driver with dynamic cruise control, lane change assist, and blind spot detection, utilizing sensors on the exterior of the vehicle to enhance passenger safety while destressing the driving experience. Often existing in parts if at all, an improved and fully integrated automobile able to provide an automotive virtual personal assistant that is responsive and proactive to user and vehicle demands remains elusive. U.S. Publication No. 2015/0306969 entitled “Automotive Recharge Scheduling Systems and Methods” by Sabripour describes a system for receiving a user input from a microphone and, through cloud-based interactions, scheduling a vehicle event with the user.

[0002] The foregoing “Background” description is for the purpose of generally presenting the context of the disclosure. Work of the inventors, to the extent it is described in this background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present invention.

[0003] The present disclosure relates to a system configured to serve as an automotive virtual personal assistant. The system is responsive to a user's vocal, visual, or tactile commands. Additionally, the system proactively initiates interaction with the user when appropriate. When pertinent, the system is configured to ascertain the cognitive load environment of the user and control the flow of information to the user, accordingly. The system can also be configured to interpret customer preferences and interact with the user in the context of stored user data.

[0004] The present disclosure also relates to a system configured to evaluate the user environment inside and outside the vehicle in the context of learned experiences and to predict future user actions such that appropriate aid or advanced notification may be provided.

[0005] The present disclosure further relates to a system configured to authorize charges following confirmation of user identity.

[0006] The foregoing paragraphs have been provided by way of general introduction, and are not intended to limit the scope of the following claims. The described embodiments, together with further advantages, will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0018] The terms “a” or “an”, as used herein, are defined as one or more than one. The term “plurality”, as used herein, is defined as two or more than two. The term “another”, as used herein, is defined as at least a second or more. The terms “including” and/or “having”, as used herein, are defined as comprising (i.e., open language). Reference throughout this document to “one embodiment”, “certain embodiments”, “an embodiment”, “an implementation”, “an example” or similar terms means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of such phrases or in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments without limitation.

[0019] FIG. 1 illustrates an embodiment of a vehicle 101 with a driver 104 and a rear passenger 102. In other embodiments, a range of passengers are in the vehicle between the minimum number and the maximum number as defined by the vehicle manufacturer. In this embodiment, the driver 104 and/or the rear passenger 102 interact with an automotive virtual personal assistant (AVPA) 106 via vocal, visual, and tactile input 105. The AVPA 106 communicates with vehicle users via audio or visual output 103.

[0020] FIG. 2 provides an architectural diagram for an AVPA 210. The AVPA 210 is comprised of a combination of hardware control circuitry, firmware and high-level software, vehicle-based and cloud-based, configured to utilize artificial intelligence approaches including machine learning. The AVPA 210 is further configured to send and receive data with a combination of hardware, including vehicle-based sensors, via equipment typically used for wireless communication (e.g. encoders, transmitters, receivers). The interactions typical of the AVPA 210 are described by (1) user-directed vehicle interaction 210 and (2) vehicle-directed user interaction 220. The dashed lines encompassing both user-directed vehicle interaction 210 and vehicle-directed user interaction 220 indicate a listening embodiment 250 of the AVPA 210. The dashed and dotted lines encompassing only vehicle-directed user interaction 220 indicate a notification embodiment 260 of the AVPA 210. The listening 250 and notification 260 embodiments rely on data from one or more vehicle-based data sources 230 and/or cloud-based data sources 240. Further, the listening 250 and notification 260 embodiments described herein, and detailed in FIG. 3, can be deployed in isolation or interdependently.

[0021] At each level, the AVPA 210 is configured to deploy multiple artificial intelligence approaches, including natural language processing and machine learning, to understand, learn and predict user preferences. To communicate with vehicle-based data sources 230, the AVPA 210 is configured to obtain data by any of a variety of wireless communication means (e.g. cellular networks). The AVPA 210 is further configured to obtain data from cloud-based data sources 240, including a variety of servers containing relevant information (i.e., file servers, web servers), by implementations of client-server systems as typified by a request-response model.

[0022] In an exemplary listening embodiment 250 of user-directed vehicle interaction 210, the AVPA 210 is prompted by a specific user request and responds, accordingly, in context of data available from one or more vehicle-based data sources 230 and cloud-based data sources 240. In another exemplary listening embodiment 250 of vehicle-directed user interaction 220, the AVPA 210 observes audio and visual data from inside the cabin of the vehicle during unprompted sequences, constructing a knowledge graph of user preferences in the context of one or more vehicle-based 230 and cloud-based data sources 240, and interacting with the user, accordingly. In an exemplary notification embodiment 260 of vehicle-directed user interaction 220, the AVPA 210 is configured to evaluate vehicle state variables (from one or more vehicle-based data sources 230) in the context of the internal and external environment of the vehicle and notify the driver if pertinent to vehicle safety or function. In another exemplary notification embodiment 260 of vehicle-directed user interaction 220, the AVPA 210 is configured to receive user-specific information from one or more cloud-based data sources 240, evaluating information from one or more vehicle-based data sources 230, and determining an appropriate time to notify the user of the cloud-based, user-specific information.

[0023] FIG. 3A provides a process workflow of an embodiment of user-directed vehicle interaction 310. In an embodiment, a user may provide a verbal “wake up phrase” or expression to the vehicle such as, for example, ‘Is everyone buckled?’ 311. This “wake up phrase” is one of a plurality of “wake up phrases” designated by the user. Following reception of a query from a user 312, the AVPA queries and evaluates data 313 from one or more appropriate sources 314. In an exemplary embodiment, this data is generated by one or more sensors inside and outside the cabin of the vehicle, including but not limited to, seatbelt sensors, pressure sensors, tactile sensors, car seat sensors, carbon dioxide sensors, temperature sensors, radar, visible spectrum cameras, infrared cameras, and microphones. Following evaluation of data 313 from the one or more appropriate data sources 314, the AVPA generates an audio and/or visual response to the user 315. Following receipt of the response, the user addresses the query as appropriate 316 or continues engagement with the AVPA via vocal or tactile input.

[0024] In another embodiment, the “wake up phrase” 311 may be related to the inside of the cabin, for example, ‘Are the children buckled?’ The AVPA receives the query from the user 312 and queries data 313 from one or more appropriate sources 314, in this embodiment, both vehicle-based data sources 330 and cloud-based data sources 340. Using queried data, the AVPA determines the presence of children in the vehicle (from images acquired by vehicle-based cameras contextualized with an image database containing typical child features), recognizes that the children are in the rear of the vehicle, and evaluates 313 the current status of their seat belts, accordingly. Following determination of the status of the seat belts, the AVPA generates an audio and/or visual response 315 to the user such as, ‘The child in the rear right passenger seat is not buckled.’ The user, heretofore focused on safe operation of the vehicle, may then address this issue 316 with the child when safe or continue engagement with the AVPA via vocal or tactile input.

[0025] In another embodiment, the “wake up phrase” 311 may be ‘I need gas.’ The AVPA receives the query 312 from the user and queries data 313 from one or more appropriate sources 314, in this situation both vehicle-based and cloud-based data sources. Evaluating real-time and stored data 313 from vehicle-based data sources pertaining to fuel level and cloud-based data sources pertaining to fuel station location and preference, the AVPA provides a recommendation to the user in the form of an audio and/or visual response 315. The user then appropriately addresses the response 316 or continues engagement with the AVPA via vocal or tactile input. Exemplary of this embodiment, the AVPA is configured to recommend to a user ‘Based on your current fuel level and average fuel consumption rate, you will be able to travel 15 miles before empty. The nearest ENERGY fuel station is 3 miles away. Would you like directions?’ The “ENERGY” fuel station described above, and in subsequent paragraphs, is meant to represent one of a number of branded fuel stations, reflecting user preference for a specific fuel company, as determined by the AVPA from stored data in cloud-based data sources (e.g. user accounts, payment history). The user can then address the AVPA recommendation as appropriate 316, either via vocal or tactile input. In another example of this embodiment, in response to a “wake up phrase”, ‘Get me directions to Charleston.’ 311, the AVPA queries one or more cloud-based data sources 313 related to navigation (i.e. web searchable content), evaluates queried data 314, and provides route options to the user visually, audibly, or as a combination thereof 315. After selecting a preferred route 316, the user becomes aware of the need to refuel the vehicle. Querying the vehicle again, ‘I need gas.’ 311, the AVPA queries one or more data sources 313 related to fueling stations and customer preference, evaluates the queried data 314, and provides a recommendation to the user audibly, visually, or as a combination thereof 315. Cognizant that the user is traveling on a specific route, the AVPA recommends 315 ‘Based on your current fuel level and average fuel consumption rate, you will be able to travel 15 miles before empty. In 5 miles, there is an ENERGY fuel station 0.5 miles from your current route. Would you like directions?’ Combining data from a plurality of sources 314, the AVPA has provided a recommendation 315 to the user, mindful of navigational routes, mappings, and customer preferences regarding fuel station. The user then responds, as appropriate 316, or continues engagement with the AVPA via vocal or tactile input.

[0026] FIG. 3B provides a process workflow of an embodiment of vehicle-directed user interaction 320. In an embodiment, the AVPA actively listens to and visually monitors the cabin environment 321, locating and identifying users 322 when present. As a user performs ordinary tasks within a vehicle, the AVPA monitors and records user actions 323. In developing a knowledge graph of user interactions with the vehicle, the AVPA interacts with one or more vehicle-based and cloud-based data sources 324 to contextualize user actions and determine user preferences 325. This knowledge graph, and the resulting user preferences, reflects implementation of machine learning algorithms, including pattern recognition, to predict future events in the context of historical events. To improve predictive value, including for events when minimal personal user data is available, the AVPA can leverage universal user data (e.g. aggregate data from a group of users), in addition to personal user data, as context for future events. After determining user preference, the AVPA predicts user actions and recommends to the user a specific action 326. The user will address the recommendation of the AVPA 327 as appropriate, or continue engagement with the AVPA via vocal or tactile input. Further, the user response is appended to a cloud-based history of user-vehicle interactions to be used as context for subsequent events.

[0027] In one embodiment, the AVPA locates and identifies a user in the cabin 322. The user begins a conversation with another passenger or via mobile phone. In communication with one or more vehicle-based data sources, the AVPA listens to and parses the conversation 323. Combining this vehicle-based data with one or more cloud-based data sources 324, the AVPA begins to contextualize user preferences 325. In an exemplary embodiment, a user, in conversation with another passenger says, ‘I should give Gunnar a call next week.’ The AVPA, having recognized the user and the user's intention, will evaluate one or more cloud-based data sources to determine if and on what days the user is available 325. Through audio and/or visual means, the AVPA can query the user ‘You are available on Tuesday to call Gunnar. Would you like me to schedule the call?’ 326. The user will address this request in accordance with their interest 327 or continue engagement with the AVPA via vocal or tactile input.

[0028] In another embodiment, the AVPA locates and identifies a user in the cabin 322. The user is engaged in normal tasks related to operating a vehicle, including music selection. The AVPA, utilizing one or more vehicle-based data sources and cloud-based data sources 324, determines and records the user's mood state from visible spectrum cameras 323. Further, utilizing one or more vehicle-based data sources 324, the AVPA identifies a genre of music 323 of interest to the user at a specific time based on, for example, current music selection. Contextualizing these two data streams, the AVPA predicts the musical preference of the user while in a specific mood state 325. In an exemplary embodiment, a user enters their vehicle after an exhausting day of work. The AVPA locates and identifies the user 322, noting the exhausted mood state of the user and the user's choice of music 323. Having developed an understanding of the user's preference in specific mood states 325 from the knowledge graph of their personal, cloud-based user history, the AVPA will subsequently predict and recommend 326 the next song to be played, for example, ‘Would you like to listen to “Tupelo Honey” by Van Morrison?’ The user will address the AVPA's recommendation, as desired 327, or continue engagement via vocal or tactile input.

[0029] FIG. 3C provides a process workflow of an embodiment of vehicle-directed user interaction 320. In an embodiment, the AVPA recognizes a change in vehicle status 332, either the result of a user-independent system modification or the result of a user modifying the status of the vehicle 331 (e.g., placing the vehicle into drive). Upon recognition of the change 332, the AVPA simultaneously queries 333 one or more vehicle-based data sources and cloud-based data sources to evaluate the current state of users and the vehicle 335 and anticipate users' actions in the context of the current state of the vehicle 334. If the AVPA identifies a potential situation 336 that the user should be made aware of in the context of current state variables 333 and predicted user actions 334, the AVPA will provide an audio and/or visual notification to the user 338. If no notification is necessary, the AVPA will return to a monitoring state 337. In an exemplary embodiment, a user has placed a vehicle in park and unbuckled their seatbelt 331. A child, in the rear of the vehicle, is still buckled. Simultaneously, the AVPA recognizes this change in vehicle state 332 and queries 333 state variables of the vehicle 335, anticipating the next action of the user 334. In this embodiment, the AVPA recognizes that the next most likely action by the user is to exit the vehicle. However, upon exiting the vehicle, the user neglects to remove the child from the rear of the vehicle. Having queried seat belt sensors and determined that a child is still in the rear of the vehicle 333 via vehicle-based data sources 335, the AVPA determines that the user should be notified 336 of the error and an alert is given 338.

[0030] In an additional embodiment, the AVPA recognizes 332 a modification 331 to the vehicle that is independent of user action. In this exemplary embodiment, the vehicle, querying 333 both vehicle-based and cloud-based data sources 335, recognizes that pressure in the back left tire has dropped to a level below the recommended pressure range for the tire. Having determined the user should be notified 336, the AVPA provides a verbal notification to the user of the need to address the tire pressure.

[0031] FIG. 3D provides a process workflow of an embodiment of vehicle-directed user interaction 320. In an embodiment, a user receives an external input 341 as routed through the AVPA. The AVPA delays the message while determining user availability 342. To determine user availability, one or more vehicle-based and cloud-based data sources 344 are queried to determine the current cognitive load of the user 343. If the user is under increased cognitive load 345, the external input is delayed 347 until the cognitive load of the user is reduced below a threshold level. If the cognitive load of the user is below a threshold level, the external input is transmitted to the user 346. In an exemplary embodiment, a user is navigating traffic during a rush hour commute. The user receives a text message 341 from a friend. The AVPA delays the text message while determining if the user is under increased cognitive load 342. Querying data 343 from one or more vehicle-based and cloud-based data sources 344, such as current speed, traffic density, facial expressions, and anticipated traffic conditions, the APVA determines the user's current cognitive load is elevated, due to the increased traffic 345. Because the user is under increased cognitive loads, the message is delayed to the user 347 until conditions improve. To this end, the AVPA actively monitors user cognitive load 345 until the load falls below a predetermined threshold 345, at which time the user is notified of the contents of the text message 346.

[0032] While discussed as related to a single user, the above FIGS. 3A, 3B, 3C, and 3D are not limited to generic notifications of vehicle or user status. Using data gathered from one or more vehicle-based and cloud-based data sources, such as voice and facial recognition or other biometrics of users with established user accounts, it is possible to provide user specific assistance for a plurality of users.

[0033] For example, in an exemplary embodiment, the driver of a vehicle, Valerie, asks her APVA 311 regarding her son, ‘Is Prince buckled?’ The APVA recognizes the speaker both audibly and visually as Valerie 312 and begins to query 313 one or more vehicle-based and cloud-based data sources 314 to determine (1) voice and facial signatures of her son, Prince, (2) if Prince is in the vehicle, (3) the location of Prince in the vehicle, and (4) if Prince's seat belt is buckled. The APVA utilizes visual spectrum camera and microphone data to locate and identify Prince in the right rear passenger seat of the vehicle. Further, the right rear passenger seat belt transducer is queried to determine if Prince's seat belt is engaged. Having determined that Prince's seat belt is fastened, the AVPA will provide a verbal response to Valerie's query ‘Yes, Valerie. Prince's seat belt is buckled.’

[0034] In another exemplary embodiment, where multiple account users are in the vehicle, the AVPA may be controlled by a user other than the driver. For example, Scott is driving a vehicle with his friend, Luke. Luke decides that he'd like to listen to music but hasn't decided on an artist. Luke asks the AVPA, ‘I'd like to listen to music. Can you play something mellow?’ 311. The AVPA locates and identifies the requesting passenger 312 utilizing one or more vehicle-based and cloud-based data sources 314. Further, the AVPA tailors a music playlist to Luke's personal preferences based on a knowledge graph of user data from his personal, cloud-based user history. The AVPA responds 315 ‘Luke, I've put together a playlist of jazz music for you based on music by John Coltrane. Would you like to hear it?’, and Luke responds 316, as appropriate, via vocal or tactile input.

[0035] In another exemplary embodiment, where a family is traveling on vacation in a vehicle, the APVA is actively monitoring the cabin environment. One of the children, Darius, says to his parents, ‘I'm hungry.’ Using one or more vehicle-based and cloud-based data sources, the APVA locates and identifies the voice, interprets the input in the context of user preferences (e.g. favorite restaurant), and recommends to the vehicle ‘In 10 minutes, I've located a McDonald's 1 mile off your route. Would you like directions?’ Darius's parents can then respond to the APVA, as appropriate, or continue engagement via vocal or tactile input.

[0036] FIG. 4 is an illustration of an embodiment of the automotive virtual personal assistant during a ride in which user interaction with the AVPA is initiated by both the user and the vehicle, thus demonstrating the interdependence of the above embodiments. In an exemplary embodiment, a vehicle 401 user, Tim 404, is riding with a friend. In conversation, Tim 404 suggests that they give another friend, Nishikant, a call and ask if he would like to join them for lunch. The AVPA identifies the speaker according to voice and facial biometric data from one or more vehicle-based data sources 402, contextualizes the conversation (integrating one or more cloud-based data sources 408) and asks the user, ‘Tim, would you like to call Nishikant and schedule lunch?’ Tim 404 agrees, the AVPA places the call, and Nishikant agrees to meet them, recommending Different Café. When the call ends, Tim 404 asks the AVPA, ‘Hey Car. Can you get us directions to Different Café?’ Alerted by the “wake-up phrase”, the AVPA determines the fastest route to Different Café and notifies the user, ‘Tim, I've located Different Café in Santa Monica, Calif. Would you like to proceed to the route?’ Tim 404 answers in the affirmative and the AVPA provides turn-by-turn directions to Different Café. Because of heavy traffic, Tim 404 is late for lunch. Nishikant sends Tim 404 a text message to let him know that he is already at Different Caféand has a table for three. The AVPA, recognizing Tim's 404 increased cognitive load, as determined from one or more vehicle-based 402 and cloud-based data sources 408 (e.g., external visible spectrum cameras 407, radar, and knowledge of predicted roadway conditions), delays delivery of the message to Tim 404 until his cognitive load decreases. Having reached Different Café and parallel parked, the AVPA releases the text message to Tim 404. Having now read the message and in a rush to the café, Tim 404 unbuckles his seatbelt 403 and unlocks the door 406. External vehicle-based data sources 407 identify a bicycle rider 405 approaching the driver's side of the vehicle 401. Recognizing that Tim 404 has unbuckled his seatbelt 403 and unlocked the door 406, the AVPA predicts that Tim 404 will next open the door 406 and alerts him audibly ‘Tim, there's a bicycle approaching on the left side of the vehicle.’ Alerted, Tim 404 allows the bicycle rider 405 to pass by the vehicle before opening the driver side door 406. The above exemplary embodiment describes the integrated nature of the AVPA as it engages with the user in a user-directed and a vehicle-directed way, utilizing both vehicle-based and cloud-based data sources.

[0037] FIG. 5A provides an exemplary illustration of a passenger in the rear of a vehicle and the vehicle-based and cloud-based data sources that may be accessible to the AVPA in the cabin. The AVPA cross-references one or more vehicle-based data sources such as audio and video from visible spectrum cameras, infrared cameras, and microphones 503 with user biometric signatures confirmed in one or more cloud-based data sources 510 to perform voice 501 and facial recognition 502 to determine the identity of the user and provide a personalized experience. If user identity cannot be determined (e.g. no user account exist), the cameras and microphones 503, along with pressure sensors 507 within the seat cushion, provide generic, directionally-specific information (e.g. the child in the rear left passenger seat). Further, a user may query the AVPA regarding the status of windows 504 and door locks 505, the security of a car seat 506, and the use of seatbelts 508 in the rear of the vehicle. In the event that a child has been left in the vehicle and the user was not previously notified, a carbon dioxide sensor 509 communicates data via wireless communication technology to the AVPA. The AVPA is configured to evaluate the data and notify the user of the oversight.

[0038] FIG. 5B is an exemplary illustration of one embodiment of the location of one or more audio and video inputs and outputs available to the AVPA in the fore of the cabin. The AVPA utilizes data from one or more audio and video inputs 513 to make either user-directed or vehicle-directed determinations on the status of the vehicle. Following this determination, the AVPA notifies the user via audio 512 or visual 511 outputs.

[0039] With regards to the vehicle described in the foregoing discussion, it should not be implied that the vehicle in reference is limited to one with an internal combustion engine, of the kind typified by a gasoline engine. Vehicle type is not limiting in the application of the disclosure.

[0040] FIG. 6 is an illustration of a vehicle at a refueling or recharging station. In an exemplary embodiment, a user 604 of an electric vehicle 601 arrives at a charging station 607. To authorize the charge 608, the user 604 communicates 605 with the AVPA via one or more vehicle-based data sources 606. Incorporating one or more cloud-based data sources 609, the AVPA asks via audio output 603, ‘Today's charge will last 15 minutes and cost $21.67. Would you like to authorize the charge?’ The user 604 responds, as appropriate. For security purposes, voice and facial recognition of the user 604 are performed by the AVPA with data from one or more vehicle-based 606 and cloud-based data sources 609 to confirm user identity. The user 604 can then couple the vehicle to the charging station 607 and begin the charge.

[0041] It should further not be implied that the vehicle must be user-operated. The above-described embodiments are not limiting and thus include autonomous and semi-autonomous vehicles. In another exemplary embodiment, the vehicle is an autonomous electric vehicle 601 with multiple users. The AVPA, having recognized a depleted electric charge from one or more vehicle-based data sources 606 and having evaluated user preference from one or more cloud-based data sources 609, notifies a user 604 that the charge is nearly depleted and recommends a nearby charging station 607 based on user preference. The user confirms the proposed charging station 607 and the vehicle navigates to the proposed location. Upon arrival at the charging station 607, another passenger 602 that is not the main user 604 of the vehicle offers to pay for the charge. The AVPA locates and identifies the passenger 602 using voice and facial recognition from one or more vehicle-based 606 and cloud-based data sources 609. Further, the AVPA confirms the charge authorization 608 with the passenger 602 via voice and facial recognition. The user 604 can now begin charging the vehicle. Charge authorization may be performed in a variety of secure ways, including but not limited to bank account linking, vehicle manufacturer smartphone application, and vehicle manufacturer account databases. Authorization may be performed by any user in the vehicle with a linked account, or the charge may be authorized by a plurality of users and divided, accordingly.

[0042] Obviously, numerous modifications and variations are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.

[0043] Thus, the foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.