Physiological feedback method and system

[0002] Pursuant to 35 U.S.C. Section 119, the benefit of priority from provisional application No. 60/118,772, with a filing date of Jan. 28, 1999, and provisional application No. 60/170,784, with a filing date of Dec. 15, 1999, is claimed for this non-provisional application.

[0004] 1. Field of the Invention

[0005] This invention relates to physiological feedback systems and methodologies. More specifically, the invention is a method and system for providing physiological feedback to a patient and/or physician in real time and in a way that allows the patient to relate his overall emotional and/or physical sensation with a monitored physiological effect.

[0006] 2. Description of the Related Art

[0007] Both biofeedback and physiological feedback systems and methods traditionally transform and display physiological information or effects in graph or chart form. The physiological information (e.g., temperature, blood pressure, etc.) can be obtained from skin surface sensors. The displayed graph or chart typically displays a value (e.g., temperature, blood pressure, etc.) measured by the sensor(s) as a function of time. While this type of data provides the medical professional with diagnostic information, it does not present information in a way that is easily interpreted by the patient. That is, the patient is not presented with an understandable representation of the ultimate physiological effect (e.g., blood flow, muscle contraction, sweat production, intestinal or stomach activity, etc.) being experienced by the patient. Accordingly, the patient is unable to relate internally-occurring physiological effects with his or her general physical and emotional sensation experience occurring at the same time.

[0008] U.S. Pat. Nos. 5,273,038 and 5,503,149 disclose the computer simulation of a live organ (e.g., the heart) using a variety of sensed electrical signals indicative of the beginning of activity of the particular organ. The electrical signals are processed/displayed to present an image of the organ performing its function. However, an image of the internal organ activity based on the initiating electrical event does not give the patient the feedback he or she needs to visualize and understand the ultimate physical effects that they are experiencing as a result of the organ activity.

[0009] Accordingly, it is an object of the present invention to provide a method and system that allows a patient to relate their overall general sensations with a specific physiological effect for physiologic education and/or biofeedback purposes.

[0010] Another object of the present invention is to provide a real-time physiological feedback system so that a patient can receive feedback regarding a physiological effect as it is being experienced.

[0011] Still another object of the present invention is to provide a physiological feedback method and system that presents a patient with a view of an internal physiological effect using non-invasive measurement techniques.

[0012] Yet another object of the present invention is to provide a physiological feedback method and system that immerses the patient in a feedback image environment to reduce or eliminate outside distractions.

[0013] Still another object of the present invention is to provide a physician a tool that satisfies the above objects for use in distinguishing various types of physiological dysfunction and for observing the progress of a disease process and the effects of a treatment intervention.

[0014] Other objects and advantages of the present invention will become more obvious hereinafter in the specification and drawings.

[0015] In accordance with the present invention, a method and system provide physiological feedback for a patient. At least one physiological effect experienced by a body part of a patient is measured non-invasively in terms of a time-series of discrete signals representing the physiological effect over a period of time. A three-dimensional graphics model is stored in a stereographic processor capable of generating a binocular image signal. The three-dimensional graphics model is an essentially analogous representation of the body part. Each discrete signal is processed using the stereographic processor to alter the three-dimensional graphics model. The binocular image signal output by the stereographic processor is representative of the three-dimensional graphics model so-altered. The three-dimensional graphics model so-altered is displayed for the patient and/or physician in a virtual reality environment.

[0016] FIG. 1 is a top-level block diagram of an embodiment of the physiological feedback system according to the present invention;

[0017] FIG. 2 is a two-dimensional graphic depiction of a model used in the present invention to depict the artery-to-vein interface when the present invention is used to provide blood flow feedback to a patient;

[0018] FIG. 3A is a graphic depiction of a perspective view of an embodiment of an “ideal” tapered cylinder which can be used in at least one embodiment of the present invention, for example, as depicted in the model shown in FIG. 2;

[0019] FIG. 3B is a graphic depiction of a perspective view of a simplified polygonalized cylinder which can be used in at least one embodiment of the present invention, for example, as depicted in the model shown in FIG. 2;

[0020] FIG. 4, is a box diagram showing one possible embodiment for a scene graph subtree for one arterial or venous branch;

[0021] FIG. 5A, a cross-sectional view of a tapered cylinder 201 without a cap, before and after scaling;

[0022] FIG. 5B shows a cross-sectional view of one possible embodiment of a cylinder 301 with a cap 302, before and after scaling; and

[0023] FIG. 5C represents a cross-sectional view of two tapered cylinders 401, 402 representing two adjacent vessels, before and after scaling.

[0024] Referring now to the drawings, and more particularly to FIG. 1, a top-level block diagram of an embodiment of the physiological feedback system according to the present invention is shown and referenced generally by numeral 10. System 10 and the method of operation thereof will be explained herein by way of illustrative example as it relates to blood movement or flow. However, as will be explained further below, the present invention can be used to present physiological feedback information for other physiological effects such as muscle deformation, perspiration production, digestive sounds, heart sounds, etc. In each case, the physiologic effect being experienced by the patient is presented to the patient and/or their physician, or other viewer, in real-time. In terms of the patient, he or she can thereby relate their own general physical and/or emotional state to a particular physiological effect. As a result, the present invention can be used to educate a patient about the severity, degradation or improvement of a physical condition. The present invention can also be used as part of biofeedback training. That is, the patient can visualize how their current emotional state affects a particular physiological effect.

[0025] System 10 has one or more non-invasive sensors 12 coupled to a patient's skin surface in an area of interest. In terms of measuring the physiological effect of blood flow, sensors 12 could be a thermistor for measuring temperature and a photoplethysmograph (PPG) for measuring blood volume pulse of the patient. The use of temperature and blood volume pulse individually as a means of determining blood flow is well known in the art. In the present invention, temperature can be used to monitor slow changes in blood flow and blood volume pulse can be used to monitor fast changes in blood flow. The combination of these two measurements enables a full characterization of blood flow. Further, other sensors or combinations thereof can be used to determine blood flow without departing from the scope of the present invention. Sensors 12 would typically be placed on the skin in an area of extensive superficial vascularization. e.g., a body extremity such as a fingertip or toe, etc., or placed on the skin at a point where an artery or vein passes close to the surface of the skin.

[0026] The physiological effects detected by sensors 12 over a period of time are transduced/converted to analog electrical signals which are passed on to a physiological data acquisition interface 14. At interface 14, the analog signals are digitized into discrete signals. If the discrete signals require manual amplitude adjustment during operation, a scaling control 16 can be coupled to interface 14. Interface 14 and scaling control 16 are configured to permit independent scaling/amplification of each sensed parameter.

[0027] The discrete (and, if necessary, amplified) signals from interface 14 are provided to a stereographics processor/computer 18 which typically includes a programmable digital computer, a monitor, a keyboard and a mouse. Some of the features preferably required of computer 18 by the present invention include a serial port interface for receiving discrete signals from interface 14, two video graphics adaptor (VGA) ports for outputting left and right (i.e., binocular) image signals, at least one ISA slot (for the interface board of head tracker 206 described below), a high-speed processor such as Intel's PENTIUM II processor, an operating system such as Microsoft's WINDOWS NT, and a sufficient amount of random access memory (RAM). It is further preferred that computer 18 include a computer graphics accelerator capable of rendering left and right three-dimensional views of an image. One commercially-available graphics accelerator is the REALIZM board from Intergraph Computer Systems, Huntsville, Ala. Each left and right view is preferably constructed at a rate of at least 50,000 polygons/second. Such computers are easily constructed using commercially available hardware. Computer 18 could also be realized with a UNIX workstation capable of generating enhanced three-dimensional graphics. One such workstation is the OCTANE available commercially from Silicon Graphics, Mountain View, Calif.

[0028] Operationally, computer 18 stores a three-dimensional graphics model that is an analogous representation of the patient's body part experiencing the physiological effect being monitored. In terms of blood flow, the stored three-dimensional graphics model is an analogous representation of a section of a vascular branch or network such as an artery-to-vein interface, the details of which will be explained further below. Computer 18 outputs left and right image signals 19A and 19B, respectively, of the model to a virtual reality display system 20. Further, computer 18 processes the discrete signals it receives from interface 14 to alter the model in accordance therewith. Computer 18 also processes inputs from a head tracker (explained further below) representing the current head orientation of the viewer immersed in display system 20 in order to alter the viewer's perspective of the model so-displayed. That is, virtual reality display system 20 receives head-tracked left and right image signals of the model undergoing change that is dictated by the physiological effects (e.g., blood flow) being sensed by sensors 12.

[0029] Virtual reality display system 20 is any three-dimensional display system that can be viewed or worn by the patient so that the patient is immersed in an interactive three-dimensional display environment. This encourages the patient to concentrate, isolate, visualize and explore his or her current overall physical and/or emotion sensation relative to the physiological effect being experienced/monitored. It is to be further understood that a physician or any viewer other than the patient could be monitoring display system 20 along with the patient or exclusive of the patient. For example, it may be desirable for the physician to be immersed in display system 20 so that he or she can change/explore various image perspectives along with or exclusive of the patient depending on the patient's physical and/or emotional state.

[0030] While a variety of such virtual reality display systems can be constructed/used in the present invention, one such system will be explained herein by way of example. Left and right image signals 19A and 19B, respectively, are input to a headset interface 202 before being supplied to a virtual reality headset 204. One such interface 202 and headset 204 combination is the “V8” helmet available from Virtual Research Systems, Santa Clara, Calif. A head tracker 206 is also included in system 20 to accurately compute the position and orientation of headset 204 worn by the patient and supply same to computer 18. An example of a commercially available head tracker 206 is the INSIDETRAK from Polhemus, Colchester, Vt.

[0031] System 10 can optionally include one or more monitors 22 coupled to receive one or both of left and right image signals 19A and 19B, respectively. Monitor 22 could provide an audience (e.g., doctors, nurses, students, etc.) a simultaneous display of the imaged physiological effect being experienced by the patient. The display could also be transmitted/viewed remotely via a network such as the internet. Such third-party monitoring could be used for patient evaluation, teaching purposes, etc.

[0032] System 10 could also include an input device 24, e.g., keyboard, mouse, joystick, voice control, etc., coupled to computer 18 to provide the equivalent of head tracker signals thereto. That is, input device 24 could be used to control viewing perspective of, for example, display(s) 22. Such control could be independent of head tracker signals received from head tracker 206 to suit the desires of a viewer other than the viewer immersed in display system 20.

[0033] As mentioned above, the three-dimensional graphics model altered by computer 18 is an analogous representation of the physiological effect being experienced by the patient. Accordingly, the model and resulting image generated therefrom in display 20 will vary depending on the physiological effect being monitored. For example, in terms of blood flow, the model represents a vascular network. A preferred embodiment of the vascular network at the artery-vein interface used in the present invention will now be explained with additional reference to FIG. 2 where the model is referenced generally by numeral 100. Model 100 will be discussed herein in terms of the image produced thereby. While the software construction of model 100 can be achieved using a variety of commercially available graphics programming environments, one way of constructing model 100 will be described later below.

[0034] Model 100 is essentially composed of a first three-dimensional structure representative of arteries and a second three-dimensional structure representative of veins. More specifically, the arteries are represented by a first tree-like structure having a trunk 102, primary branches 104A and 104B coupled to trunk 102, and secondary branches 106A-106D with branches 106A and 106B being coupled to branch 104A and branches 106C and 106D being coupled to branch 104B. The veins are represented by a second treelike structure having a trunk 112, primary branches 114A and 114B coupled to trunk 112, and secondary branches 116A-116D with branches 116A and 116B being coupled to branch 114A and branches 116C and 116D being coupled to branch 114B. Finally, secondary (artery) branches 106A-106D terminate at respective ones of secondary (vein) branches 116A-116D.

[0035] It is desirable to simulate the narrowing diameter of arteries as they branch out and ultimately reach their termination point as well as the expanding diameter of veins as a function of their distance from a coupling artery. To do this in a graphically simple manner, each branch of model 100 can be constructed as a cylinder that is linearly tapering (in the case of artery branches) or expanding (in the case of vein branches) in diameter. FIG. 3A shows an embodiment of a preferred ideal smooth shape of a tapered cylinder. FIG. 3B shows a more simplified version of the cylinder with polygonal sides, which, as known in the art, is generally required by a computer graphics processor. As the number of sides to the cylinder increases the smooth shape shown in FIG. 3A is approached.

[0036] The properties of each branch can be saved to and retrieved from a file stored in computer 18. By placing the branches end-to-end as illustrated in FIG. 2, a network of simulated arteries and veins is constructed. A curved or splined transition is formed along the outside edges of adjoining branches (see FIG. 5C). While this can be accomplished in a variety of ways using graphics tools, one way of creating such splined transitions is to use some or all of hemispherical “caps” which are standard filler tools/elements available in graphics programming environments. Another way to create such splined transitions is to use non-uniform rational B-splines (NURBS). The three-dimensional aspects of model 100 can be enhanced by applying specular lighting properties to each branch.

[0037] To render model 100 in real-time and simulate the expansion and contraction of the blood vessels from blood being pumped through them, the geometric shape, orientation and diametric scale of each branch is encoded in an efficient data structure known as a scene graph. A scene graph is a well-known construct in the field of computer graphics. One example of computer graphics software which makes use of scene graphs is the WorldToolKit from the Sense8 Product Line of Engineering Animation Inc. in Mill Valley, Calif. A scene graph allows three-dimensional geometry to be manipulated and rendered in real-time. It contains all the current geometric elements of a computer-generated scene (in the form of standard three-dimensional vertices, polygons, and, if used, colors) as well as position, orientation, and scaling information in the form of matrices. Matrix transforms are accumulated as the scene graph is processed from left to right. A SEPARATOR NODE is used in the scene graph to isolate matrix transforms to a single subtree.

[0038] FIG. 4 is a box diagram showing one possible embodiment for a scene graph subtree for one branch of model 100. Briefly, the baseline shape of each branch of model 100 is a tapered cylinder aligned with the z-axis (as in FIG. 5A) and is encoded in the scene graph of FIG. 4 under CYLINDER GEOMETRY. Similarly, the hemispherical cap (as in FIG. 5B) is encoded in the scene graph under HEMISPHERE GEOMETRY. The translation and rotation of this branch is encoded as a standard four-dimensional transformation matrix under ORIENTATION MATRIX. The current scaling matrix is divided into an X-Y SCALING MATRIX and a Z-SCALING MATRIX as shown. Since the scene graph is processed from left to right, the CYLINDER GEOMETRY will be scaled about only two axes, whereas the HEMISPHERE GEOMETRY will be scaled about all three axes. Every arterial branch is modeled with a similar scene graph subtree. Since all arterial branches are scaled the same, all subtrees can refer to the same X-Y SCALING MATRIX and Z SCALING MATRIX using constant POINTERs. Similar scaling matrices can be implemented for the venous branches.

[0039] Pulsating arteries and veins on display 20 can be created by the use of scaling matrices for arteries and veins within the scene graph which change over time. A scaling matrix in the present invention is of the form. $\left[\begin{array}{cccc}{S}_x& 0& 0& 0\\ 0& S_y& 0& 0\\ 0& 0& S_z& 0\\ 0& 0& 0& 1\end{array}\right]$

[0040] Every branch is initially aligned with the z-axis so that a uniform dilation/expansion of a branch by a scaling factor of D is created by equal scaling in the x and y-dimensions while keeping the z-dimension at unity (S_x=D, S_y=D, S_z=1). Note that the hemispherical “caps” used to create splined transitions between adjoining branches are uniformly scaled in all three dimensions about their centers (S_x=D, S_y=D, S_z=D). The equation for scaling factor D is explained below.

[0041] As shown in FIG. 5A, a cross-sectional view of a tapered cylinder 201 without a cap, the radius of the cylinder is scaled up or down, but the length, or height “H” remains the same.

[0042] FIG. 5B shows a cross-sectional view of one possible embodiment of a cylinder 301 with a cap 302, wherein the hemispherical cap is scaled in the x, y, and z axes, and the cylinder is scaled only in the x and y axes.

[0043] FIG. 5C represents a cross-sectional view of two tapered cylinders 401, 402 representing two adjacent vessels, wherein these cylinders are scaled in their respective x any y axes, while maintaining the respective “A” and “B” heights of the cylinders. A smooth transition T from one branch to the other can be preserved by scaling the hemispherical cap in its respective x, y and z axes.

[0044] All arterial branches in the scene can be scaled uniformly along their length (i.e., about their diameter) using the same scaling matrix. As a result, the same diameter scale factor is applied to all artery branches at the same time. Similarly, all venous branches can be scaled uniformly along their length but independently of the arterial branches. To provide the patient with an autokinetic sense of left-to-right movement, the venous side of model 100 is altered using discrete signals (received from interface 14) slightly after the same discrete signals were used to alter the arterial side of model 100. For example, if the display of model 100 is rendered at a resolution of 30 frames per second, the same scaling used on the arterial side in frame 1 is used on the venous side in frame 2. In other words, the venous side would lag the arterial side by one frame in order to create the illusion of flow from the arterial side to the venous side. Note that this visual effect could, for example, be an option turned on or off by the patient.

[0045] The pulsating autokinetic sense of movement methodology just described is simple to implement in a real-time fashion without much computational complexity. However, it is to be understood that the present invention is not so limited. For example, a left-to-right moving bulge or pulse in the arterial and venous sides of model 100 could be achieved by scaling at the individual polygonal level as opposed to the branch level just described. Obviously, such scaling comes with a computational cost and a possible loss of real-time feedback. However, it is to be understood that future advances in computer as well as physiological monitoring technologies could be used to improve the spatial and temporal fidelity of this model without departing from the scope of the present invention.

[0046] As mentioned above, color can be incorporated into each scene graph. For example, in terms of the illustrative blood flow embodiment, trunk 102 and branches 104A and 104B could be colored red. Branches 106A-106B could be colored red where they couple to branches 104A and 104B and then gradually transition to purple at the point where they couple to a respective one of branches 116A-116D. Branches 116A-116D could gradually transition (from left to right) from purple to blue. Finally, trunk 112 and 114A and 114B could be colored blue. Note that purple is the combination of red and blue and that the shade of purple used during transition will vary with contribution of red and blue. The shading from red-to-purple and then purple-to-blue at the arterial-venous interface branches provides the patient with a smooth visual transition indicative of the actual transition from arteries to veins.

[0047] The actual scaling values used to control dynamic changes in branch diameter can be changed as desired. In terms of measuring the patient's temperature and blood volume pulse with sensors 12 and in terms of uniform branch scaling described above, computer 18 can utilize the algorithm in equation below to develop the scaling factor D ( 0<D≦1) for each branch diameter. $D=0.5\left[\frac{M_1-M_{1-\mathrm{MIN}}}{M_{1-\mathrm{MAX}}-M_{1-\mathrm{MIN}}}*\frac{T-T_{\mathrm{MIN}}}{T_{\mathrm{MAX}}-T_{\mathrm{MIN}}}\right]+0.5\left[\frac{M_2-M_{2-\mathrm{MIN}}}{M_{2-\mathrm{MAX}}-M_{2-\mathrm{MIN}}}*\frac{P-P_{\mathrm{MIN}}}{P_{\mathrm{MAX}}-P_{\mathrm{MIN}}}\right]$

[0048] where

[0049] M₁is the amplification provided by scaling control 16 and applied to the temperature measured by one of sensors 12, M_1-MINis the minimum amplification value of M₁,

[0050] M_1-MAXis the maximum amplification value of M₁,

[0051] T is the temperature measured by one of sensors 12,

[0052] T_MINis the minimum temperature measured,

[0053] T_MAXis the maximum temperature measured,

[0054] M₂is the amplification provided by scaling control 16 and applied to the blood volume pulse measured by one of sensors 12,

[0055] M_2-MINis the minimum amplification value of M₂,

[0056] M_2-MAXis the maximum amplification value of M₂,

[0057] P is the blood volume pulse measured by one of sensors 12,

[0058] P_MINis the minimum blood volume pulse measured, and

[0059] P_MAXis the maximum blood volume pulse measured.

[0060] Minimum and maximum values are determined by the electrical characteristics of the amplification and measurement equipment. It is to be understood that similar equations could be used for scaling factor D.

[0061] The advantages of the present invention are numerous. Physiological effects experienced by a patient are fed back to the patient and/or physician, other viewer, etc., in an immersive display environment in real-time. In this way, the patient can relate their general physical and/or emotional state to a specific physiological effect for educational and/or therapeutic purposes. The sensors used to collect the data are preferably non-invasive thereby minimizing the amount of associated stress on the patient. With the present invention, expensive and time-consuming anatomical imaging methods/systems (e.g., MRI, CAT scan, etc.), as well as the preprocessing of data derived therefrom, are not necessary.

[0062] As noted above, the present invention is not limited to blood flow. Another embodiment of the present invention could examine the physiological effect of muscle contraction. In such an embodiment, the length and belly diameter of a graphically-depicted muscle displayed in an immersive display environment could be based on the measurement of the deformation of the skin surface over the muscle with a piezoelectric sensor (i.e., sensor 12), and/or measurement of the stretching of the skin with a strain gauge sensor (i.e., sensor 12). As muscles contract, the skin surface rounds, the muscle shortens and the muscle belly expands thereby stretching the skin over the muscle. Muscle contraction feedback could be useful for the treatment of muscle tension accompanying anxiety and headaches, and for neuromuscular rehabilitation.

[0063] Another embodiment involves displaying the action of a graphically depicted sweat gland or sudomotor pump (in an immersive display environment) based on the measurement of the saline saturation of the skin with an electrical conductance sensor (i.e., sensor 12). As sweat glands pump more perspiration onto the skin surface, the conductance of the skin increases. Skin conductance biofeedback could be employed in the treatment of anxiety and hyperhidrosis.

[0064] Still another embodiment involves displaying the action of a graphically-depicted stomach, intestine or bowel, or analogous representation thereof, based on acoustic measurements made at the skin surface. Stomach or intestinal activity generates noises that could be easily detected with sensitive microphones (i.e., sensors 12) properly placed on the patient's skin.

[0065] Yet another embodiment of the present invention involves displaying the action of graphically-depicted heart valves and/or heart structures, or analogous representations thereof, based on acoustic measurements made at the surface of the patient's torso. The heart action could be modeled to mimic a physician's mental visualization associated with turbulence in heart sounds normally heard with a stethoscope. Heart dynamic anomalies could thus be visually rendered. A related embodiment would involve displaying restricted blood flow due to vessel narrowing. Blood flow in a monitored blood vessel, or analogous representation thereof, could be graphically-depicted based on acoustic measurements made at the surface of the skin over the vessel. The blood flow dynamical display could be modeled to mimic a physician's mental visualization associated with turbulence in blood flow normally heard with a stethoscope. Other mental visualizations employed by the physician from signs sensed at the patient's skin surface during the physical examination could be supplemented by electronic display using the method of the present invention.

[0066] Temperature and blood volume pulse biofeedback can, for example, be used for general relaxation training, as well as treatment for a number of vascular problems, such as Raynaud's disease, migraine headache, vasoconstriction secondary to diabetes and connective tissue disease, and hypertension.

[0067] A pseudo-code for one possible embodiment of the software is as follows:

[0068] PHYSIO_DEVICE_INITIALIZE;

[0069] PHYSIO_DEVICE_REQUEST_PACKET;

[0070] HEAD_TRACKER_INITIALIZE;

[0071] HEAD_TRACKER_REQUEST_PACKET;

[0072] SCENE_GRAPH_INITLALIZE;

[0073] DO {

[0074] PHYSIO_DEVICE_READ_PACKET (TEMP, BVP, DIAL1, DIAL2);

[0075] HEAD_TRACKER_READ_PACKET (HEAD_POS, HEAD_ORIENT);

[0076] DIAMETER=0.5 * (DIAL1_DIAL1_MIN) / (DIAL1_MAX_DIAL1_MIN) * (TEMP_TEMP_MIN) / (TEMP_MAX_TEMP_MIN) +0.5 * (DIAL2_DIAL2_MIN) / (DIAL2_MAX_DIAL2_MIN) * (BVP_BVP_MIN) / (BVP_MAX_BVP_MIN);

[0077] SCENE_GRAPH_MODULATE_SCALE_MATRICES (DIAMETER);

[0078] PHYSIO_DEVICE_REQUEST_PACKET;

[0079] HEAD_TRACKER_REQUEST_PACKET;

[0080] UPDATE_PICTURE (SCENE_GRAPH, HEAD_POS, HEAD_ORIENT)

[0081] }

[0082] UNTIL (FINISHED)

[0083] Thus, although the invention has been described relative to a specific embodiment thereof, there are numerous variations and modifications that will be readily apparent to those skilled in the art in light of the above teachings. It is therefore to be understood that, within the scope of the appended claims, the invention may be practiced other than as specifically described.