AUTOMATED TRANSCRIPTION AND DOCUMENTATION OF TELE-HEALTH ENCOUNTERS

[0001] This application claims priority to U.S. provisional application No. 62/489,380, filed Apr. 24, 2017, which is hereby incorporated by reference in its entirety.

[0002] The present technology pertains to tele-health systems, and more specifically to the automated production of medical documentation.

[0003] Studies have shown that as little as one-third of physician time is spent visiting with patients, and much of the remaining two-thirds of physician time is dedicated to documenting those patient encounters. These are often documented in the form of a SOAP (e.g., “Subjective, Objective, Assessment, and Plan”) note. A SOAP note may be entered into a medical record for the patient, typically an electronic medical record (“EMR”), and documents a patient statement of a reason for visiting a physician and the patient history of illness, observations of the patient made by the physician and other healthcare professionals (e.g., vital signs, weight, examination findings, and the like), medical diagnoses of the patient symptoms, and a determined treatment plan for the patient.

[0010] Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without departing from the spirit and scope of the disclosure.

[0011] It should be understood at the outset that although illustrative implementations of one or more embodiments are illustrated below, the disclosed apparatus and methods may be implemented using any number of techniques. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0012] The automated generation of a SOAP note from a live encounter can greatly increase the time a physician has available to be with patients. Medical providers, such as physicians for example, typically devote a significant portion of the day to administrative tasks such as generating documentation of patient consultations and the like. In particular, the manual production of SOAP notes is a time consuming and tedious process which often takes up a sizable portion of the workday.

[0013] The disclosed technology may provide additional benefits in the context of tele-health encounters. A typical tele-health encounter may involve a patient and one or more remotely located physicians or healthcare providers.—devices located in the vicinity of the patient and the providers allow the patients and providers to communicate with each other using, for example, two-way audio and/or video conferencing.

[0014] A tele-presence device may take the form of a desktop, laptop, tablet, smart phone, or any computing device equipped with hardware and software configured to capture, reproduce, transmit, and receive audio and/or video to or from another tele-presence device across a communication network. Tele-presence devices may also take the form of tele-presence robots, carts, and/or other devices such as those marketed by InTouch Technologies, Inc. of Goleta, Calif., under the names INTOUCH VITA, INTOUCH LITE, INTOUCH VANTAGE, INTOUCH VICI, INTOUCH VIEWPOINT, INTOUCH XPRESS, and INTOUCH XPRESS CART. The physician tele-presence device and the patient tele-presence device may mediate an encounter, thus providing high-quality audio capture on both the provider-side and the patient-side of the interaction.

[0015] Furthermore, unlike an in-person encounter where a smart phone may be placed on the table and an application started, a tele-health-based auto-scribe can intelligently tie into a much larger context around the live encounter. The tele-health system may include a server or cloud infrastructure that provides the remote provider with clinical documentation tools and/or access to the electronic medical record (“EMR”) and medical imaging systems (e.g., such as a “picture archiving and communication system,” or “PACS,” and the like) within any number of hospitals, hospital networks, other care facilities, or any other type of medical information system. In this environment, the software may have access to the name or identification of the patient being examined as well as access to their EMR. The software may also have access to, for example, notes from a nursing staff that may have just been entered. Increased input to the system may make system outputs more robust and complete. The outputs can be automatically incorporated into the appropriate electronic medical record (EMR).

[0016] There may also be other advantages and features, such as, without limitation:

• • [0017] Prior to a remote physician initiating a tele-health encounter or session, the patient-side tele-presence device may proactively prompt the patient with questions and the patient responses may be added to the note.
  • [0018] Utilize context of the physician interaction with a clinical documentation tool to add to completeness and robustness of a SOAP note generated by the physician or automatically by the system.
  • [0019] Utilize context of other physician user interface (“UI”) interactions. For instance, and without imputing limitation, the system can track whether the provider activated a camera zoom function to more closely examine the patient's eyes, a wound, and the like. The system can also track whether the provider activated a camera pan and/or tilt function to look at a chart or vitals monitor. Moreover, the system can track whether the provider accessed output from a medical peripheral device in the vicinity of the patient, such as a stethoscope, otoscope, sonogram, ultrasound, dermal camera, and the like.
  • [0020] The system can apply computer vision techniques to the video to perform scene recognition for the creation of additional context and content for SOAP note generation. This can include, for example and without imputing limitation, understanding what part of a patient is being looked at and automatically analyzing vital signs monitors.
  • [0021] The system may make use of a human or artificial intelligence (“AI”) language translator that has access to session audio and/or video. For example, the language translator can mediate by translating the physician audio from English into a native language of the patient and translate the patient responses back into English for both the physician and an AI scribe.

[0022] In one example, a physician uses a clinical documentation tool within a tele-health software application on a laptop to review a patient record. The physician can click a “connect” button in the tele-health software that connects the physician tele-presence device to a tele-presence device in the vicinity of the patient. In one example, the patient-side tele-presence device may be a mobile tele-presence robot with autonomous navigation capability located in a hospital, such as an INTOUCH VITA. The patient-side tele-presence may automatically navigate to the patient bedside, and the tele-health software can launch a live audio and/or video conferencing session between the physician laptop and the patient-side tele-presence device such as disclosed in U.S. Pub. No. 2005/02044381 and hereby incorporated by reference in its entirety.

[0023] In addition to the live video, the tele-health software can display a transcription box. Everything the physician or patient says can appear in the transcription box and may be converted to text. In some examples, the text may be presented as a scrolling marquee or an otherwise streaming text.

[0024] Transcription may begin immediately upon commencement of the session. The physician interface may display a clinical documentation tool, including a stroke workflow (e.g., with a NIHSS, or National Institutes of Health Stroke Scale, score, a tPA, or tissue plasminogen activator, calculator, and the like) such as disclosed in U.S. Pub. No. 2009/0259339 and hereby incorporated by reference in its entirety. Furthermore, the stroke workflow may be provided in the physician interface alongside a live SOAP note window.

[0025] The system can also monitor and process “sidebar” conversations. Such conversations can include discussions taking place between the physician and personnel at the patient site via, for example, a handset on the patient-side tele-presence device. Additionally, in a case in which there are multiple remote parties participating in the session via a multipoint conference, conversations between the remote participants can also be monitored and processed.

[0026] The system may distinguish among participants using voice recognition techniques. In one embodiment, the system may only populate the SOAP note with content from a specified participant such as, for example and without imputing limitation, the physician. In some examples, the audio can be processed by multiple neural networks or preprocessed by various services. For example, the audio may be first fed through a trained speech-to-text network such as Amazon® Transcribe® or Nuance® Dragon® and the like. The transcribed output text may then be used as input into a SOAP note generated by the physician. A network can be trained on a portion (e.g., 80%) of SOAP notes created in such a way and then tested against a remaining portion (e.g., 20%) of the SOAP notes.

[0027] As an encounter progresses, the system can automatically fill in the SOAP note. A deep learning neural network or other trained machine learning model analyzing the encounter can run concurrent to the encounter and update itself using automatic and/or physician-provided feedback. In some examples, early entries in the SOAP note may be inaccurate, but later entries will become increasingly correct as greater context becomes available throughout the encounter. While discussed in the context of a neural network, it is understood that various and multiple machine learning networks and methodologies can be used to train a model for use in automatically generating a SOAP note. For example, and without imputing limitation, logit, sequential logit, Hidden Markov Model, and other machine learning networks and models may be used as will be apparent to a person having ordinary skill in the art.

[0028] Further, the system may diarize audio and process speaker identity as further context and input for the deep learning neural network. In some examples, dedicated microphones on both the patient-side and physician-side of the system can inform the system which speaker is associated with what audio content through, for example, dedicated and predefined audio channels. In such a case, an audio channel associated with the physician-side of the system may be processed separately than, for example, an audio channel associated with the patient-side. Further diarization techniques can be applied to both audio channels to further distinguish, for example, a patient statement from that of an on-site attendant (e.g., nurse and the like) statement.

[0029] The SOAP note may be multimedia in that it includes text, pictures, clips or any other media relevant to the encounter. For example, a SOAP note may include an audio recording of either or both of the physician or patient. In some examples, the SOAP note can be linked to a PACS or similar in order to both directly and indirectly include imaging data and the like.

[0030] In one example, the physician may choose to add or change certain things in a live SOAP note as it is generated. The physician input can be integrated as another data source in the neural network. In some examples, the physician input can be used to update the neural network while the SOAP note is generated and thus increase the quality of the generated SOAP note as the encounter progresses.

[0031] In another example, the system may include meta-information derived from a patient speech in addition to performing patient speech transcription. For example and without imputing limitation, the system may track and make note of inflection, slurring, and pauses between a physician question and the start of the patient answer. This and other types of meta-information may be valuable to the SOAP note context.

[0032] The system may also track physician interactions to add further context for SOAP note generation. Physician interactions can include interactions with a clinical documentation tool (e.g., a NIHSS stage being viewed by the physician) and interactions with an endpoint UI (e.g., zooms, pans, tilts, switches between cameras, switches to a stethoscope, and the like). In some examples, the physician may toggle what input is tracked by, for example, holding space bar to pause tracking (e.g., where the physician is reacting to a matter unrelated to the patient interaction and the like).

[0033] The system may recognize references to content in the image and automatically capture the image and insert it in the documentation. For example, if the physician instructed the patient to “hold your hands up in front of you”, then the system may automatically capture an image or video clip of the subsequent activity. The system may also perform other visual recognition on video or images from the patient-side camera to further add context and make the note more complete and robust.

[0034] The system may also utilize other cloud-based AI systems to bring greater context to a given clinical situation. For example, if a CT scan is uploaded to a cloud service for analysis, the resulting analysis may be included in the SOAP note. In some examples, the system may directly interface with a PACS and the like to retrieve imaging data.

[0035] Upon completion of the live encounter with the patient, the physician can end the audio and/or video session. The video window closes and, in the case of a robotic patient-side endpoint, the patient-side tele-presence device may navigate back to its dock. The physician-side interface may display a patient record (e.g., within a clinical documentation tool). In some examples, the generated SOAP note may be displayed next to the patient record. The SOAP note may be editable so the physician can make changes to the SOAP note. When satisfied, the physician may sign the note and click a “Send” button to automatically insert the SOAP note into an EMR for that patient. Further, as discussed above, the physician changes to the generated SOAP note can be fed back into the neural network in order to further improve SOAP note generation. In some examples, the neural network can train a physician-specific model based on multiple SOAP note changes received from a particular physician.

[0036] The neural network can be one or more trained Deep Learning networks. The architecture of the neural network can be a single network or layers of networks through which data and outputs can be cascaded. The network may have been trained by data over several thousand encounters, using various input data, including, but not limited to, two-way audio recording from an encounter, interface data and/or visual data from the encounter, and meta-data from the encounter (e.g., pause durations, postures, UI interactions, and the like).

[0037] The neural network output data may include a SOAP note produced from the encounter. The SOAP note may be cleaned and curated by a third party or the responsible physician. In some examples, the SOAP note can be provided back to the neural network as, for example, further training data in order to improve the accuracy of the neural network for later encounter.

[0038] In one embodiment, the neural network can be a Recurrent Neural Network (RNN) built on the CaFE framework from UC Berkeley. The network may be embodied in a software module that executes on one or more servers coupled to the network in the tele-health system. Alternatively, the module may execute on a patient tele-presence device or a physician tele-presence device. The output of the module can include transcribed audio, a SOAP note, and the like. Further, in some examples, the module may transmit the output from a server to multiple and various tele-presence devices, from one tele-presence device to another, and/or to a medical records or documentation server where it can be stored in association with a patient medical record.

[0039] FIG. 1. Depicts a system 100 for automatically generating a SOAP note from a tele-encounter between a patient 108 in a patient environment 102 and a physician 118 in an operator environment 104. The physician 118 and patient 108 may be located in different locations and communicate with each other over a network 106 which may include one or more Internet linkages, Local Area Networks (“LAN”), mobile networks, and the like.

[0040] The patient 108 and the physician 118 can interact via a patient endpoint 110 in the patient environment 102 and a physician endpoint 124 in the operator environment 104. While depicted in FIG. 1 as computer terminals, it will be understood by a person having ordinary skill in the art that either or both of the endpoint 110 and the endpoint 124 can be a desktop computer, a mobile phone, a remotely operated robot, a laptop computer, and the like. In some examples, the endpoint 110 can be a remotely operated robot which is controlled by the physician 118 through the endpoint 124 which is a laptop computer.

[0041] Nevertheless, the endpoint 112 may include a patient-side audio receiver 112 and the endpoint 124 can include a physician-side audio receiver 126. The patient-side audio receiver 112 and the physician-side audio receiver 126 can provide audio data to a processing server 128 via respective endpoint 110 and endpoint 124 over the network 106. In some examples, the audio data is received as particular channels and may assist the processing server 128 in diarizing audio inputs to the system 100. The processing server 128 may be a remotely connected computer server 122. In some examples, the processing server 128 may include a virtual server and the like provided over a cloud-based service, as will be understood by a person having ordinary skill in the art.

[0042] The physician 118 may retrieve and review EMR and other medical data related to the patient 108 from a networked records server 116. The records server 116 can be a computer server 120 remotely connected to the physician endpoint 124 via the network 106 or may be onsite with the physician 118 or the patient 108.

[0043] In addition to patient audio and EMR, the physician 118 can receive diagnostic or other medical data from the patient 108 via a medical monitoring device 114 hooked up to the patient 108 and connected to the patient endpoint 110. For example, a heart-rate monitor may be providing cardiovascular measurements of the patient 108 to the patient endpoint 110 and on to the physician 118 via the network 106 and the physician endpoint 124. In some examples, multiple medical monitoring devices 114 can be connected to the patient endpoint 110 in order to provide a suite of data to the physician 118. Other devices such as, for example, a camera and the like may be connected to the patient endpoint 110 and/or the physician endpoint 124 (not depicted) and can further provide environmental and other contextual to the system 100. The processing server 128 can intercept or otherwise receive data transmitted between the operator environment 104 and the patient environment 102.

[0044] FIG. 2 and FIG. 3 depict in greater detail by which a system 200 and a method 300 respectively can automatically produce a SOAP note 202 from an encounter between a patient and a physician. A SOAP note 202 may include, for example, fields for subjective information, objective information, assessments, and treatment plans as will be understood by a person having ordinary skill in the art. In some examples, the SOAP note 202 can include multimedia information such as video or audio (not depicted).

[0045] A SOAP note generator 216 may be provided on a processing server 128 or otherwise connected to a patient endpoint 206 and a physician endpoint 220. In some examples, the SOAP note generator 216 can be located directly on the physician endpoint 220 or the patient endpoint 206. The physician endpoint 220 and the patient endpoint 206 may be similar to physician endpoint 124 and the patient endpoint 110 depicted in FIG. 1 and discussed above. In some other examples, the SOAP note generator 216 can be provided as a distributed system and may include components and/or processes run on the processing server 128, the patient endpoint 206, the physician endpoint 220, remotely provided services over, for example, the network 106, or some combination thereof.

[0046] The SOAP note generator 216 may include a deep learning neural network 224 for processing input data, a SOAP text generator 226 for converting outputs from the deep learning neural network 224 into text for the SOAP note, and a neural network feedback process 228 for updating the deep learning neural network 224 responsive to, for example, physician feedback. The SOAP note generator 216 can be communicatively connected to the patient endpoint 206 and the physician endpoint 220 and may further be communicatively connected to a records storage 204. The records storage 204 can receive the generated SOAP note 202 for storage in, for example, an EMR associated with a patient. In some examples, the records storage 204 can provide an EMR or other historical data to the SOAP note generator 216.

[0047] The SOAP note generator 216 can receive patient medical data as a medical record 208 and monitor data 210 (operation 302). The medical record 208 can be an EMR received from the records storage 204. In some examples the medical record 208 can include other SOAP notes, notes from nurses at the time of the current visit, and other data. The monitor data 210 can include data from any or all of multiple devices such as an EKG, blood pressure monitor, thermometer, and the like as will be apparent to a person having ordinary skill in the art.

[0048] The SOAP note generator 216 can also receive patient environment data (operation 304). Patient environment data can include a patient environment audio channel 212 as well as patient environment visual data 214. In some examples, either or both of the patient environment audio channel 212 and the patient environment visual data 214 can be preprocessed by, for example, text-to-speech software provided by a third party.

[0049] A physician audio channel 222 and control inputs 218 may be provided to the SOAP note generator 216 (operation 306). In some examples, the physician audio channel 222 can be limited by an attending physician through, for example, turning off recording and the like by pressing and/or depressing a space bar. The control inputs 218 can include, for example, the pressing and depressing the space bar above and other UI interactions on the physician endpoint 220. For example, the attending physician may be able to control a camera in the patient environment and camera control actions performed by the physician such as, for example, camera zoom, sweep, pan, focus, and the like as will be understood by a person having ordinary skill in the art.

[0050] The deep learning neural network 224 may generate SOAP note data using the physician data (e.g., the physician audio channel 222 and the control inputs 218) and the patient environment and medical data (e.g., the medical record 208, the monitor data 210, the patient environment audio channel 212, and the patient environment visual data 214) as inputs (operation 308). In some examples, specific physician audio data may be identified and only that data will be used to generate SOAP note data. FIG. 4 depicts one such method 400 for further processing the physician audio channel 222.

[0051] Spoken content uttered by the physician may be identified (operation 402). The physician audio channel 222 may include other audio noise such as other voices (e.g., other physicians performing tele-health consultations) or arbitrary environmental sounds the like. The identified content may be segments of spoken content provided by the physician amongst a larger volume of spoken content from the physician otherwise not intended to be included in the SOAP note data. In some examples, this can be performed on the physician endpoint 220 via UI interactions performed by the physician (e.g., pressing a record a key and the like) or through an automated process (e.g., voice command interaction and the like).

[0052] The spoken content may be processed to identify a portion for insertion into a SOAP note (operation 404). The identified portion may be provided to the SOAP note generator 216 as input into the deep learning neural network 224 or, in some examples, may be provided to the SOAP text generator 226. Nevertheless, the identified portion of spoken content may be converted into SOAP note data (operation 406). In some examples, the SOAP note data may be able to be directly inserted into the SOAP note 202 (e.g., as string variables and the like). In some other examples, the SOAP note generator 216 may further process the data through a SOAP text generator 226 for insertion into the SOAP note 202.

[0053] Returning to FIG. 3, the generated SOAP note 202 may be provided to the physician through the physician endpoint 220 (operation 310). In some examples, the SOAP note 202 may be provided as a final product after completion of the tele-health interaction with the patient. In other examples, the SOAP note 202 can be provided in real time to the physician endpoint 220 as a dynamic and on-the-fly generated UI element.

[0054] The physician may make physician corrections 230 to the SOAP note 202 and the corrections may be received by the neural network feedback process 228 of the SOAP note generator 216 (operation 312). In some examples, particularly where the SOAP note 202 is provided to the physician endpoint 220 in real time, the physician corrections 230 can be maintained in the UI while the at the same time being processed by the neural network feedback process 228.

[0055] The deep learning neural network 224 may be updated by the neural network feedback process 228 using the physician corrections 230 (operation 312). The neural network feedback process 228 may update the deep learning neural network 224 through, for example, a gradient descent algorithm and back propagation and the like as will be apparent to a person having ordinary skill in the art. In some examples, the deep learning neural network 224 may be updated in real time or near real time. In other examples, the neural network feedback process 228 may perform model updates as a background process on a mirror version of the deep learning neural network 224 and directly update the deep learning neural network 224 once the mirror version has converged on an updated model. In other examples, the neural network feedback process 228 may perform updates on a scheduled or through a batch process. The updates can be performed on a singular device or may be performed across parallelized threads and processes and the like.

[0056] Once the SOAP note 202 is reviewed by the physician on, for example, the physician endpoint 220, the SOAP note 202 can be transmitted to the records storage 204 (operation 316). The SOAP note 202 may be added to the medical record 208 of a patient, for example, to be used later as input to the SOAP note generator 216 during a future tele-health conference with the same patient.

[0057] FIG. 5 depicts an example computing system 500 that may implement various systems and methods discussed herein. The computer system 500 includes one or more computing components in communication via a bus 502. In one implementation, the computing system 500 includes one or more processors 514. The processor 514 can include one or more internal levels of cache 516 and a bus controller or bus interface unit to direct interaction with the bus 502. Memory 508 may include one or more memory cards and a control circuit (not depicted), or other forms of removable memory, and may store various software applications including computer executable instructions, that when run on the processor 514, implement the methods and systems set out herein. Other forms of memory, such as a mass storage device 512, may also be included and accessible, by the processor (or processors) 514 via the bus 502.

[0058] The computer system 500 can further include a communications interface 518 by way of which the computer system 500 can connect to networks and receive data useful in executing the methods and system set out herein as well as transmitting information to other devices. The computer system 500 may include an output device 504 by which information can be displayed. The computer system 500 can also include an input device 506 by which information is input. Input device 506 can be a scanner, keyboard, and/or other input devices as will be apparent to a person of ordinary skill in the art. The system set forth in FIG. 5 is but one possible example of a computer system that may employ or be configured in accordance with aspects of the present disclosure. It will be appreciated that other non-transitory tangible computer-readable storage media storing computer-executable instructions for implementing the presently disclosed technology on a computing system may be utilized.

[0059] In the present disclosure, the methods disclosed may be implemented as sets of instructions or software readable by a device. Further, it is understood that the specific order or hierarchy of steps in the methods disclosed are instances of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the methods can be rearranged while remaining within the disclosed subject matter. The accompanying method claims present elements of the various steps in a sample order, and are not necessarily meant to be limited to the specific order or hierarchy presented.

[0060] The described disclosure may be provided as a computer program product, or software, that may include a computer-readable storage medium having stored thereon instructions, which may be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure. A computer-readable storage medium includes any mechanism for storing information in a form (e.g., software, processing application) readable by a computer. The computer-readable storage medium may include, but is not limited to, optical storage medium (e.g., CD-ROM), magneto-optical storage medium, read only memory (ROM), random access memory (RAM), erasable programmable memory (e.g., EPROM and EEPROM), flash memory, or other types of medium suitable for storing electronic instructions.

[0061] The description above includes example systems, methods, techniques, instruction sequences, and/or computer program products that embody techniques of the present disclosure. However, it is understood that the described disclosure may be practiced without these specific details.

[0062] While the present disclosure has been described with references to various implementations, it will be understood that these implementations are illustrative and that the scope of the disclosure is not limited to them. Many variations, modifications, additions, and improvements are possible. More generally, implementations in accordance with the present disclosure have been described in the context of particular implementations. Functionality may be separated or combined in blocks differently in various embodiments of the disclosure or described with different terminology. These and other variations, modifications, additions, and improvements may fall within the scope of the disclosure as defined in the claims that follow.