Method and apparatus for encouraging physiological self-regulation through modulation of an operator's control input to a video game or training simulator

[0002] Pursuant to 35 U.S.C. §119, the benefit of priority from provisional application No. 60/143,028, with filing date of Jul. 9, 1999, is claimed for this non-provisional application.

[0004] 1. Field of the Invention

[0005] The present invention relates generally to the use of biofeedback to modify a subject's behavior, mental state, and/or physiological functioning. More specifically, the invention relates to apparatus and methods for modulating an operator's control input to an electronic game or simulator in response to measured physiological activity, such as autonomically-mediated and/or EEG physiological activity, wherein the player thereby learns to control the physiological and/or EEG activity.

[0006] 2. Discussion of Background Art

[0007] Biofeedback systems can be used for a variety of purposes, such as to address behavioral disorders, such as Attention Deficit Disorder, and for training job-related physiological activity.

[0008] Attention Deficit Disorder (ADD) is a behavioral disorder which is characterized by the inability to sustain attention long enough to perform some activities, such as school work or organized play. Current treatments for ADD include medication, behavior therapy and brainwave biofeedback training. In biofeedback training, the trainee, usually a child, is typically provided information in the form of a conventionally produced electroencephalograph (EEG) display which shows him how much he is producing the brainwave pattern(s) indicative of attention and/or inattention. This display is typically in bland, minimally motivating formats, and trains individuals by focusing their attention directly on the status of targeted physiological signals. For example, this feedback frequently consists of a video representation of the EEG graph. The procedure, though providing useful information, is often very limited in variability and predictable, and accordingly can induce boredom. This can lead to frustration when progress is slow, and makes it hard to encourage simultaneous desirable changes in multiple physiological parameters due to limitations in ability to attend to multiple signals. Positive reinforcement of attention states can accordingly be difficult to obtain, especially in children, and more especially in children exhibiting ADD.

[0009] Additionally, with increased sophistication in technology, human performance has increasingly become an important, frequently limiting, factor in the proper performance of many advanced technology job-related tasks. For example, both inattention and stress overload can play a substantial role in impairing pilot performance and producing flight hazards. Current biofeedback methods are hard to apply to job-related physiological training because the necessary focus on physiological feedback signals can distract trainees from challenging professional tasks. The ability to control physiological activity, such as to control stress or to remain aware of fluctuating attentional states, the ability to maintain effective physiological states, and the ability to recover efficiently from attention lapses or other ineffective physiological states are valuable in task settings requiring recognition and response.

[0010] U.S. Pat. No. 4,008,714 to Silva et al. Describes a system which generates an audible indication and starts a prerecorded educational program upon determination that the subject has a sufficient concentration level.

[0011] U.S. Pat. No. 4,461,301 to Ochs discloses a biofeedback system which provides a visual display of a metric indication of a physiological function and a visual display of a target metric which is readjusted to prevent attainment of the target.

[0012] U.S. Pat. No. 5,377,100 entitled “Method of Encouraging Attention by Correlating Video Game Difficulty with Attention Level” demonstrates the concept of improving attention skill by rewarding specific brain signal patterns with success at playing an action video game. The game is virtually impossible to win until the player exhibits the required brain signal patterns that accompany normal vs. attention-deficit behavior. Once the player exhibits the required “normal” brain signal patterns the game becomes manageable. A measurement system senses EEG signals and routes them to the computer where the game difficulty control signal is derived. This invention has the disadvantage of requiring extensive reprogramming of a video game, or the complete construction of a new video game, in order to implement the method. Because much video game software is proprietary and/or not available in source code, this software would be unusable for implementing the method.

[0013] A commercial product, entitled “The Mind Drive,” senses physiological signals from a finger sensor and uses the signals alone to drive a video game. Because “The Mind Drive” does not deliver biofeedback training while the trainee is playing a game or performing a task in a conventional way, its method does not reinforce desirable physiological changes in the realistic context of task performance. The Mind Drive, too, has the disadvantage of requiring extensive video game programming in order to implement the method.

[0014] It is accordingly an object of the present invention to provide a method and apparatus that provides positive reinforcement of desirable attention states and/or physiological changes or states.

[0015] It is another object of the invention to maintain the desirable attention states and/or physiological states once they are obtained in a subject.

[0016] It is an object of the present invention to provide a method and apparatus that improves a subject's subsequent ability to both achieve and maintain desirable attention states and/or physiological states, and improves the subject's performance or behavior when conducting his or her daily activities.

[0017] It is an object of the invention to provide a video or computer game or training simulation that responds to physiological activity as well as joystick, game pad input and/or other user input.

[0018] It is another object of the invention to provide a video or computer game or training simulation system that does not require extensive reprogramming, or complete construction of, the game's or training simulation's software.

[0019] Still another object of the present invention is to provide a method and apparatus that delivers biofeedback training while the trainee is playing a game or performing a task in a conventional way, and therefore its method reinforces desirable physiological changes in the realistic context of task performance, with successful performance of the task as intrinsic reinforcement.

[0020] It is an object of the present invention to provide apparatus and methods which can be used with off-the-shelf video or personal computer games or existing simulators.

[0021] Still another object of the present invention is to provide a method and apparatus that delivers biofeedback training which can help reduce the occurrence of ineffective physiological states, such as inattention and/or dysfunctional stress.

[0022] It is another object of the invention to provide a video game or training simulation system that provides engaging entertainment and/or challenges to the user.

[0023] Yet another object of the invention is to teach the trainee to incorporate autonomic or EEG (brainwave) physiological self-regulation into professional task performance through repeated associations (classical conditioning) without need for conscious attention to such regulation.

[0024] Other objects and advantages of the present invention will become more obvious from the specification and drawings.

[0025] The objects referenced above, and others made obvious herein, can be accomplished by the present invention which provides a video game, simulator, or another task challenge format, which can motivate trainees to participate in and adhere to the training process through the rewards inherent in mastery of popular video games, computer game, or job task simulators, and preferably without the demand, monotony or frustration potential of direct concentration on physiological signals. The present invention includes apparatus and methods modulating the control function of a computer simulation or game input device (e.g., joystick, button control) using physiological information so as to affect the user's ability to impact or control the simulation or game with the input device (control authority). One aspect of the invention is to use the present invention, along with a computer simulation or game, to affect physiological state or physiological self-regulation according to some programmed criterion (e.g., increase, decrease, or maintain) in order to perform better at the game task. When the affected physiological state or physiological self-regulation is the target of self-regulation or biofeedback training, the game play reinforces therapeutic changes in the physiological signal(s). The feedback preferably is, however, implicit in the task and not explicit in the form of direct feedback (e.g., bar graphs, tracings, etc.), and therefore offers more subtle conditioning of the desired physiological response(s) than conventional biofeedback methods and apparatuses. For example, when the present invention is used for training job-related physiological activity it can allow biofeedback training to occur without distracting the trainee from challenging professional tasks, and can teach the trainee to incorporate autonomic or EEG (brainwave) physiological self-regulation into professional task performance through repeated associations (classical conditioning) without need for conscious attention to such regulation.

[0026] Embodiments of the present invention will be described in more detail below, with reference to the following figures, in which:

[0027] FIG. 1 is a block diagram of an embodiment for a system for the modulation of an operator's control input to a video game or training simulator according to the present invention;

[0028] FIG. 2A is a flow chart of a possible embodiment of terminate and stay resident (TSR) software which may be used in accordance with at least one embodiment of the present invention;

[0029] FIG. 2B is a flow chart of another possible embodiment of TSR software which may be used in accordance with at least one embodiment of the present invention;

[0030] FIG. 2C shows one embodiment of source code for a TSR software module that can be used to implement the flow chart in FIG. 2A,

[0031] FIG. 3 is a block diagram of another embodiment for a system for the modulation of an operator's control input to a video game or training simulator according to the present invention;

[0032] FIG. 4 is a graph relating game port circuitry count to joystick output voltage;

[0033] FIG. 5 is a block diagram of another embodiment for a system for the modulation of an operator's control input to a video game or training simulator according to the present invention;

[0034] FIG. 6 is a graph showing a piecewise linear fit to the curve that represents count versus voltage (FIG. 4);

[0035] FIG. 7 is a graph relating joystick position versus game port circuitry count;

[0036] FIG. 8 is a block diagram of another embodiment for a system for the modulation of an operator's control input to a video game or training simulator according to the present invention; and

[0037] FIG. 9 is a block diagram of still another possible embodiment for a system for the modulation of an operator's control input to a video game or training simulator according to the present invention.

[0038] The present invention includes a method of transforming physiological information obtained from biomedical instruments in order to use that information to modify the functioning of computer simulation or game controllers or joysticks. The invention involves modulation that transforms the controller signals received at the computer's game port prior to their being used by the computer simulation or game software. The result can be that the magnitude of the effect of the game or simulation's input device (e.g., joystick, game pad, steering wheel) is modulated by the strength of the physiological signal(s). By making the joystick's “control authority” proportional to the physiological signal(s), the player is encouraged to change the physiological signal(s) according to a programmed criterion (e.g., increase, decrease, or maintain) in order to perform better at the game task. When the physiological signal(s) are the target of physiological self-regulation or biofeedback training, the game play reinforces therapeutic changes in related physiological processes. However, the reinforcing feedback is preferably implicit in the task, and not explicit in the form of direct feedback (bar graphs, tracings), as in conventional biofeedback training. In this way, contingencies for subtle conditioning of the desirable physiological response(s) are set up.

[0039] Different embodiments of the present invention are possible, and the components of the invention can vary depending upon implementation. Further, the invention is intended to be used with a variety of systems, such as standard video game systems (e.g., Sony Playstation, Nintendo, etc.), with standard personal computer systems, and with computer simulators and professional job training systems. Additionally, one or more of a wide variety of different measured physiological signals can be used in accordance with the present invention, for example, EEGs, skin temperature, skin conductance, heart rate, and/or event-related potentials (ERPs). U.S. Pat. No. 5,377,100, issued on Dec. 27, 1994 to Pope et al, and which is incorporated herein by reference as if set forth in its entirety, at column 3, line 8 to column 5, line 60, details a method for determining an individual's EEG index of attention, which index can be used to assess his or her mental engagement in a task.

[0040] FIG. 1 depicts one preferred embodiment, a “software/hardware combination” implementation, wherein the user's joystick 105 is connected to the primary joystick port 105aand the physiological signals are passed to the secondary joystick port 103aof a multi-port joystick interface. Joystick “counts” can be acquired by both interfaces 103a,105aby existing computer software. Additional software is added to modulate the primary joystick counts by the secondary physiological signal counts according to some predetermined algorithm.

[0041] As shown in FIG. 1, the physiological signal(s) 100, 100aare connected to the computer game port 102 through the terminals 103adesigned for the input(s) of a second joystick, but used for the modulating signal(s) 100, 100ain this embodiment of the invention. The game port charging circuit for each joystick axis is designed to cause the joystick-reading software to register a count proportional to the resistance connected to the circuit and inversely proportional to the voltage applied to the circuit. In conventional operation, this resistance is proportional to the position of the joystick in each axis and the applied voltage is supplied as a constant value from the game port. In accordance with at least one embodiment of the present invention, this situation is reversed: the resistance is set to a constant (for example, 20 kilohms) and the applied voltage is made to vary inversely with the physiological signal within the range of approximately 3.2 volts to approximately 5 volts. The count(s) for the input(s) ordinarily connected to joystick 2, registered by the joystick-reading software, is then proportional to the magnitude of the physiological signal(s).

[0042] The value of the physiological signal(s) obtained in this way can then be used to modify the count registered from joystick 1105 in the following manner.

[0043] The following operations can be performed by a terminate-and-stay-resident (TSR) software module that processes the joystick input prior to its being used by the game program. FIGS. 2A and 2B show example flow charts of TSR software modules. FIG. 2A shows a TSR software module that can modulate joystick input with the physiological signal(s) 100, 100aprior to the input being used by the game or simulation program. FIG. 2B shows another possible TSR module wherein the module time averages the joystick input, based on the physiological signal, or signals, prior to input being used by the game or simulation program. FIG. 2C shows one embodiment of source code for a TSR software module that can be used to implement the flow chart in FIG. 2A.

[0044] The definitions of terms in FIGS. 2A and 2B include the following. The functions, f(X_i,φ_i,θ_i), and g(Y_i,φ_i,θ_i), can take several forms, depending upon which embodiment of the present invention is being implemented. The two axes of joystick 2 (corresponding to the x and y inputs 103a) allow for two independent physiological signal inputs (φ,θ) to be applied to modulate joystick 1105 position variables (X,Y; X_c,Y_c=center position) singly (see equation sets A and B immediately below) or in some combinations of the signals (equation set C below) or to both axes simultaneously (example: equation set A below) or to each axis independently (example: equation set A1 below) or each signal to a different axis (example: equation set A2 below).

[0045] Equation set A (modulating with φ_i):

[0046] f(X_i,φ_i,θ_i)=(X_i−X_C)*(φ_i/φ_max)+X_C

[0047] g(Y_i,φ_i,θ_i)=(Y_i−Y_C)*(φ_i/φ_max)+Y_C

[0048] Equation set A1 (modulating the X axis independently with φ_i):

[0049] f(X_i,φ_i,θ_i)=(X_i−X_C)*(φ_i/φ_max)+X_C

[0050] g(Y_i,φ_i,θ_i)=Y_i

[0051] Equation set A2 (modulating with each physiological signal to a different axis):

[0052] f(X_i,φ_i,θ_i)=(X_i−X_C)*(φ_i/φ_max)+X_C

[0053] g(Y_i,φ_i,θ_i)=(Y_i−Y_C)*(θ_i/θ_max)+Y_C

[0054] Equation set B (modulating with θ_i):

[0055] f(X_i,φ_i,θ_i)=(X_i−X_C)*(θ_i/θ_max)+X_C

[0056] g(Y_i,φ_i,θ_i)=(Y_i−Y_C)*(θ_i/θ_max)+Y_C

[0057] Equation set C (modulating with a combination function of φ_iand θ_i, h(φ_i,θ_i)):

[0058] f(X_i,φ_i,θ_i)=(X_i−X_C)*h(φ_i,θ_i)/h(φ_max,θ_max)+X_C

[0059] g(Y_i,φ_i,θ_i)=(Y_i−Y_C)*h(φ_i,θ_i)/h(φ_max,θ_max)+Y_C

[0060] In another possible embodiment of the present invention, instead of using the current count of the physiological signal to multiply the stick position by, the current count may be used as an index of how many stick position readings to average to get a filtered count number (e.g., a joystick response “sluggishness” modulation):

[0061] N₁=integer(k₁*φ_i),

[0062] where k₁is a constant that scales the influence of the physiological signal on sluggishness. $f\left(X_j\right)=\left(1/N_1\right)\sum _{j=1}^{N_1}{X}_j$ $g\left(Y_j\right)=\left(1/N_1\right)\sum _{j=1}^{N_1}{Y}_j$

[0063] The signal that modulates the functionality of the joystick 1105 does not have to be a monotonic function of the physiological signal, but can, for example, represent the deviation of the subject's physiological signal from a baseline level. This procedure differs from those described above essentially only in that the magnitude of the effect of the game controller or joystick 1 on the game or simulation is modulated by the magnitude of the difference between the physiological signal(s) being input at the joystick 2 input(s) and a baseline value (φ_B) provided to the software by the operator:

[0064] Equation set D:

[0065] f(X_i,φ_i,θ_i)=(X_i−X_C)*absolute value [(φ_i−φ_B)/(φ_max−φ_B)]

[0066] g(Y_i,φ_i,θ_i)=(Y_i−Y_C)*absolute value [(φ_i−φ_B)/(φ_max−φ_B)]

[0067] The count corresponding to the midpoint of each axis excursion of joystick 1105 is determined when the game starts (X_C,Y_C).

[0068] In at least one embodiment, during game play, the counts corresponding to the current x and y position (X_i,Y_i) of joystick 1105 are then captured and the midpoint counts of each axis are subtracted from this value. The resulting signed difference in each axis is then multiplied by a value equal to the current count value of the physiological signal(s) divided by the number of counts in the full range of the physiological signal(s). This signed multiplication result is then added back to the midpoint count in the respective axis and this count value in each axis is provided to the game program as the current value of the joystick 1 input.

[0069] The result is that the magnitude of the effect of the game controller or joystick 1105 on the game or simulation is modulated by the strength of the physiological signal(s) 100, 100abeing input at the joystick 2 input(s) 103a.

[0070] A software patch can be applied to the game program to make the game compatible with the TSR program. In at least one embodiment, the patch can be made by disassembling the game program and locating the code that reads the joystick position count from the game port. The code can then be changed in a known manner, to have the game program execute the TSR code which scales the joystick values based upon the physiological signals.

[0071] The embodiment just described can be implemented with off-the-shelf games and simulations, such as, with a DOS-based off-the-shelf flight simulator game. Alternative embodiments are possible, for example, one such embodiment could include accomplishing the same functionality by modifying the WINMN.DLL software module in a Windows environment, or by using a separate application to trap all Application Program Interface (API) calls to joystick reading functions.

[0072] In another alternative embodiment of the “software/hardware combination” implementation of the present invention (see FIG. 1), instead of using the current count of the physiological signal 100 to multiply the stick position by, the current count is used as an index of how many stick position readings to average to get a filtered count number (e.g., a joystick response “sluggishness” modulation).

[0073] A second physiological signal 100acan be used to enable or disable the joystick buttons 106 which can be connected to the button input port 107. These joystick buttons can be momentary contact normally-open button switches, for example.

[0074] The two axes of joystick 2 allow for two independent physiological signal inputs to be applied to modulate the joystick position variables singly or in some combinations of the signals or to both axes simultaneously, or to each axis independently, or each signal to a different axis (equation sets A-C above).

[0075] The physiological signal(s) can be inverted, or not, with analog circuitry external to the computer game port, depending upon whether the object of training is to increase (as in the case of hand temperature in biofeedback-assisted relaxation training or brainwave band power ratios in neurofeedback training for attention-deficit disorder) or decrease (as is usually the case for heart rate or muscle tension in biofeedback-assisted relaxation training) the magnitude of the physiological signal(s).

[0076] In another possible embodiment of the present invention, the signal that modulates the functionality of the joystick does not have to be a monotonic function of the physiological signal, but can, for example, represent the deviation of the subject's physiological signal from a baseline level (equation set D above). The translation of deviations from baseline into degrees of joystick functionality may be done entirely in software, with the physiological signal voltage applied to the input(s) 103afor joystick 2, as described above. This procedure differs from that described above only in that the magnitude of the effect of the game controller or joystick 1105 on the game or simulation is modulated by the maginitude of the difference between the physiological signal(s) 100, 100abeing input at the joystick 2 input(s) 103aand a baseline value provided to the software by the operator.

[0077] FIG. 3 shows another, “hardware” embodiment of the present invention. In this embodiment the modulation operation can be performed entirely with hardware external to the computer game port, not requiring any software changes, thus allowing the joystick inputs to any off-the-shelf computer game or simulation to be modulated by a physiological signal. For example, this embodiment can be used with joysticks on a standard game system (e.g., Sony, Nintendo, etc.), or a standard computer or simulator arrangement. As described earlier, the charging circuit for each axis of a joystick is designed to cause the joystick-reading software to register a count proportional to the resistance of the circuit and inversely proportional to the applied voltage to the circuit. In conventional operation, this resistance is proportional to the position of the joystick in each axis and the applied voltage is supplied as a constant value from the game port. For the present method, this situation is reversed: the resistance is set to a constant (for example, 20 kilohms) and the applied voltage is made to vary with an analog voltage that is proportional to joystick position and also modulated by the physiological signal, or signals. As shown in the graph depicted in FIG. 4, the game port circuitry produces a count that varies nonlinearly with the input voltage in each axis within the range of approximately 5 volts to approximately 3.2 volts. To produce this input voltage in each axis, the joystick variable resistors in each axis can be reconfigured as voltage dividers (potentiometers) instead of rheostats (see FIG. 5). As shown in FIG. 5, the physiological signal 500 is connected to gone end of both of the voltage dividers 501, 502 and the inverted physiological signal 500ais connected to the other ends of the voltage dividers 501, 502. As either of the potentiometer wipers is turned from one end to the other, the voltage on the wiper changes from, for example, the current positive value of the physiological signal 500 through 0 volts to the current negative value of the physiological signal 500a.This signal is adjusted, by an amplitude control, which is part of the Physiological Signal Conditioning System 104, to have a maximum positive value of about 0.9 volts and maximum negative value of about 0.9 volts. This arrangement produces a joystick variable resistor wiper voltage that is approximately linear with respect to joystick position. As shown in FIG. 5, the resulting signal is offset by adding about 4.1 volts to each wiper signal with the offset circuitry 505, 506 to range from about 5 volts to about 3.2 volts, and then is applied through the fixed 20K resistors 503, 504 to the respective game port axis input (X and Y). With linear potentiometers 501, 502, this arrangement produces the nonlinear relationship between joystick axis position and the count read by the computer that is depicted in the graph of FIG. 4. Game software expects a linear relationship between these two variables. Therefore, the count versus voltage nonlinearity must be counteracted.

[0078] To produce the necessary linear relationship between joystick axis position and the count read by the computer, an operation that counteracts the inherent nonlinear relationship between applied voltage (from the joystick modified to be a voltage divider) and the count read by the computer must be performed. This operation needs to produce a characteristic curve that matches the count vs. voltage curve, but with joystick axis position replacing count on the y-axis. This curve representing joystick output voltage vs. joystick axis position resembles the characteristic curve of an audio taper potentiometer.

[0079] Therefore, in at least one embodiment, potentiometers with an exponential nonlinearity (audio taper) may be used to counteract the inverted logarithmic nonlinearity characteristic of the game port charging circuitry. The segment of the nonlinear potentiometer characteristic that produces the about +0.9 volt to about −0.9 volt signal sweep can be adjusted by selection of the potentiometer value, adjustment of the maximum physiological signal values, and adjustment of the 0 volt point by means of a linear trim potentiometer in series with the axis potentiometer.

[0080] To take advantage of the full nonlinear characteristic curve of an audio potentiometer requires more rotation of the potentiometer than is produced by the typical joystick (usually approximately 60 degrees), so a gearing arrangement preferably must be used to change the limited range of rotation of the joystick to a greater range (approximately 300 degrees) before the rotation is applied to the potentiometer.

[0081] In another possible embodiment, the mechanical manipulation can be replaced by an electrical manipulation by approximating the nonlinear curve representing joystick axis output voltage vs. joystick axis position with two linear segments of different slopes intersecting at the centered point of the joystick's movement, as shown in FIG. 6. This manipulation may be accomplished by amplifying the joystick output voltage by a constant when the voltage is negative and amplifying the voltage by a different constant when it is positive. This can be accomplished electrically by applying the joystick output voltage through an adjustable gain amplifier (gain=k1) to one input leg of a linear summing amplifier and, in parallel, through another adjustable gain amplifier (gain=k2) to a second input leg of the summing amplifier. A rectifying diode in series with the input to the second adjustable gain amplifier allows only the positive voltage to be amplified by k2. The output of the summing amplifier is equal to k1 times the joystick output voltage when the voltage is negative and to k1+k2 times the joystick output voltage when the voltage is positive. This output voltage is offset before being applied to the game port circuitry to produce the range of approximately 3.2 to 5 volts for full joystick deflection from the extreme position in one direction to the extreme position in the other direction for each axis, X and Y, with maximum modulation voltage. Control of the three parameters—negative voltage amplification, positive voltage amplification, and offset, by scaling, linearization and offset circuitry 207, as shown in FIG. 3—allows a close piecewise linear approximation to the needed nonlinear curve representing joystick axis output voltage vs. joystick axis position to be achieved (see FIG. 6). Modulating the voltage applied to the joystick potentiometer end terminals symmetrically modulates the maximum count available from deflecting the joystick to one extreme or the other, and proportionately modulates all joystick positions along each axis. The center point of joystick deflection remains at the offset voltage and is not modulated.

[0082] This arrangement produces an approximately linear relationship between joystick axis position and the count read by the computer, depicted in the graph shown in FIG. 7. Some joystick calibration capabilities, such as, the Windows 95 or 98 joystick calibration capability, allow further linearization to be done by the software system, enhancing the hardware linearization described above. The maximum range of the voltage at the output of the potentiometer is determined by the constantly changing value of the physiological signal voltage, resulting in modulation of the joystick input to the game port by the physiological signal.

[0083] The signal that modulates the functionality of the joystick does not have to be a monotonic function of the physiological signal, but can, for example, represent the deviation of the subject's physiological signal from a baseline level.

[0084] This can be accomplished, for example as shown in the embodiment shown in FIG. 3, by using a baseline deviation deriving system 208 that allows a baseline level to be set. When the input physiological signal 500 level is equal to the baseline setting, an output voltage of zero is present at the output. Physiological signal levels that are higher than the baseline would normally produce a positive voltage output, and signal levels lower than baseline would normally produce a negative output. Since departures from baseline in either direction preferably must have the same effect, the module output signal is internally rectified, resulting in output signal 215 voltages that increase in the positive direction whether the input physiological signal deviates upward from baseline or downward from it. The Baseline Deviation Deriving System 208 also scales, inverts, and offsets so that the physiological range will produce a voltage range corresponding to the desired range of impairment of joystick control.

[0085] The baseline deviation voltage 215 and its inverted voltage are applied to opposite ends of the joystick wired as a voltage divider 212. The resulting joystick wiper voltage 216 is scaled, linearized, and offset as described above, to produce the range of approximately 3.2 to 5 volts for full joystick deflection from the extreme position in one direction to the extreme position in the other direction for each axis, X and Y, with maximum modulation voltage.

[0086] For example, if a temperature deviation from baseline of 3 degrees is intended to cause full impairment of control, the rectified module output signal is scaled, clipped, inverted and offset to produce a maximum positive value of 0.9 volts for 0 degrees temperature deviation from baseline, and a minimum value of 0 volts for 3 degrees deviation. The resulting output signal 215 is connected to one end of the joystick voltage divider and the inverted signal is connected to the other. The resulting voltage divider wiper signal 216 is scaled, linearized and offset to range from a maximum possible of 5 volts to a minimum possible of 3.2 volts as above, see FIG. 3.

[0087] As shown in FIG. 3, if the impairment in functionality is intended to be modulated by deviation of the physiological signal from a set window (dead zone) around the baseline level, rather than from the baseline level itself, the rectified module output signal can be gated off for all values within that window by use of a comparator, which is internal to Baseline Deviation Deriving System 208, set at the window limits and connected to an analog gate module following the rectifier module, which gate module and rectifier module can also be internal to Baseline Deviation Deriving System 208. The output of the analog gate module will have a value of zero until the rectified signal exceeds the window limit, at which point the gate output will jump to the level of the limit and increase from there. If, for example, a temperature deviation from baseline of 3 degrees is intended to cause full impairment of control with no impairment occurring until the temperature deviation from baseline exceeds 1 degree, the window limit is set to correspond to 1 degree. In this example, the gate output can then produces 0.9 volts for 0 to 1 degree temperature deviation from baseline, and a range of approximately 0.6 volts to approximately 0 volts for 1 to 3 degrees temperature deviation from baseline, representing ⅓ to complete impairment of joystick 1212 functionality.

[0088] The resulting signal 215 is connected to one end of the joystick voltage divider, which is internal to the Joystick Wired as Voltage Divider 212, and the inverted signal is connected to the other end. The resulting voltage divider wiper signal 216 is scaled, linearized and offset to range from a maximum possible of about 5 volts to a minimum possible of about 3.2 volts as above described in reference to FIG. 3. FIG. 5 shows additional details of one possible embodiment of the Joystick Wired as Voltage Divider 212.

[0089] As shown in FIG. 3, the ability to control joystick buttons 106 may be made contingent on producing criterion values of the physiological signal. Buttons are turned off by breaking the switch closure circuit for that button with a relay closure within the Threshold Detector & Relays 210, in series with the button circuit. The relay can be driven by the binary output of a voltage comparator whose threshold level can be adjusted as training progresses to shape the physiological response in the desired direction.

[0090] FIG. 8 is a block diagram of another possible embodiment of the present invention, namely, a “Video Game Pad” implementation of the present invention. Video “game pads” 311 use discrete switches operated by button presses instead of variable resistors in the game controller. Therefore, the modulation of an analog input will not work with this type of video game controller. Instead, a form of “frequency modulation” may be used whereby the switch closures that control the movement of the player's character across the screen are pulsed at a frequency proportional to the physiological signal being monitored. This can be accomplished by driving a voltage-controlled oscillator (VCO) 301 (which is part of the Frequency Modulation Circuitry 315) with the physiological signal, and passing the frequency modulated (FM) square wave output of the VCO 301 to reed relays or solid state relays (in the Frequency Modulation Circuitry 315) that take the place of the game controller button switch closures that control motion of the player's game character or object. The FM signal is gated (AND gate) to the relay controlling one direction of movement (say, right) motion by the switch closure operated by the game controller's original button 303 that controlled movement in that direction, and the same for movement in the other direction (say, left). To get appropriate movement of the character, it is necessary to shape the VCO signal with a retriggerable one-shot set at a duration determined by trial-and-error adjustment. Since the one-shot is retriggerable, when the VCO frequency is higher than 1/(duration of the one-shot), the signal to the relay is constant, the relay closes as long as the controller button is pressed, and the game character moves at its maximum speed. In accordance with at least one application of the present invention, the player's ability to control the movement of his character can be affected. For example, the player's character moves only in short bursts when the physiological signal is low, and progress is slow. The interval between bursts decreases as the physiological signal increases, and progress speeds up. (Some characters can appear to tiptoe or sputter at these speeds.) At a high enough value of the physiological signal, the player has the full original control of the character's movement, and the character breaks into a run or full speed if the controller button is held down. Since the character's speed is modulated by the physiological signal in this way, the player is reinforced for producing high values of the physiological signal.

[0091] In at least one embodiment, a corresponding modulation of speed by physiological input can be achieved for a joystick with throttle control, such as that provided for racing and flight simulation games, by modulating the throttle control input (usually applied to the game port x-axis terminal for a second joystick) with a signal derived from the physiological signal in a manner similar to that described for modulation of the range of joystick x- and y-axis control described above. For example, at a high enough value of the physiological signal, the player has the full original control of the game object's (racecar, aircraft, game character, etc) forward movement by means of the throttle control. Since the object's horizontal speed can be modulated by the physiological signal in this way, the player is reinforced for producing high values of the physiological signal.

[0092] In the Sony Playstation game, “Metal Gear Solid,” the game character can take specific measures to steady a wavering of his aiming; the wavering returns as the effect wears off. In the Star Wars-based Nintendo game, “Rogue Squadron,” the player's spacecraft becomes harder to control when R2D2 is damaged. In accordance with the present invention, this unsteadiness in controllability can be modulated by physiological signals with the current invention, making game controllability a true reflection of player state.

[0093] As an alternative to modulation of range of control by a physiological signal, the reinforcement criterion can be based upon modulation of an uncontrolled disturbance by the physiological signal. As shown in FIG. 9, with a joystick implementation, this modulation can be accomplished by mixing a randomly fluctuating signal 401awith a DC signal (5 volts in the shown embodiment) 401band applying it as the supply voltage to the unmodified joystick 312 rheostats in place of the constant 5 volts from the game port. The amplitude of the randomly varying signal, is modulated by the physiological signal prior to mixing, which modulating and mixing is performed by the Voltage Controlled Amplifier or Filter and Mixer 402. A change in the physiological signal in the desired direction diminishes the amplitude of the random signal and, consequently, the fluctuations superimposed on the DC supply voltage to the joystick. The resulting effect upon the game or simulation is to create a wavering of the controlled object that is experienced as a loss of precision controllability as the physiological signal departs from criterion; with an aircraft simulation, for example, this is experienced as “buffet.” This control variability ranges from a maximum of operating as if the joystick is only able to apply a directional bias to a randomly moving object to a minimum of no wavering and full joystick control when the physiological signal is maximum.

[0094] In at least one embodiment of the gamepad implementation, as shown in FIG. 8, this modulation can be accomplished by subtracting a randomly varying monopolar signal 401afrom a DC signal 401band applying the result to the voltage controlled oscillator (VCO) 301. The amplitude of the random signal 401ais modulated by the physiological signal prior to substracting, which subtracting is performed by the Amplitude Modulator & Mixer 302, with the amplitude of the DC signal equal to the amplitude that corresponds to the “constantly on” frequency of the VCO range. [When the VCO frequency is higher than 1/(duration of the one-shot), the signal to the relay is constant, the relay closes as long as the controller button is pressed, and the game character moves, for instance, at its maximum speed.] A change in the physiological signal in the desired direction diminishes the amplitude of the random signal and, consequently, the effect of its subtraction from the DC signal. The resulting effect upon the game is to create a wavering of the controlled object that is experienced as a loss of precision controllability as the physiological signal departs from criterion. This control variability ranges from a maximum of operating as if the directional control button is only able to apply a directional bias to an alternately accelerating and decelerating object, through stages of the object accelerating and decelerating less while making directional progress, to a minimum of full, unwavering directional control when the physiological signal is maximum.

[0095] The amplitude modulation of the random signal may be accomplished with a voltage-controlled amplifier or by passing the random signal through a voltage-controlled filter, the filter's control voltage being the properly conditioned physiological signal. The amplitude of the random signal to the VCO can be either attenuated or amplified in response to changes in the physiological signal. For example, the random signal amplitude could be increased as the physiological signal increases, or decreased as the physiological signal increases. Likewise, it is also possible that the random signal amplitude could be increased or decreased as the physiological signal decreases..

[0096] An alternative implementation of the random modulation function for the gamepad, shown in FIG. 8, involves modulating the duty cycle of the pulses to the directional control relays located in the Frequency Modulated Gamepad Buttons 303 with complementary logic signals from the Duty Cycle Modulator 307. A periodic waveform 308 (e.g., triangle wave) is input to a voltage comparator and the physiological signal, properly amplified and level shifted, is applied as the comparison threshold signal. The physiological signal 100 is adjusted to range from 0 volts to beyond the maximum half-wave amplitude of the periodic input signal. The duty cycle of the resulting logic output ranges from about 50% when the physiological signal 100 is zero to 0% when the physiological signal is above the periodic signal maximum amplitude. The duty cycle of the complementary logic output ranges from about 50% when the physiological signal is zero to about 100% when the physiological signal is above the periodic signal maximum amplitude. These logic signals are each ANDed with the directional command logic signals from the switch buffers. When a directional control button is pushed, the duty cycle of the comparator “straight” AND logic output ranges from about 50% when the physiological signal is zero to 0% when the physiological signal is above the periodic signal maximum amplitude. The duty cycle of the comparator complementary AND logic output ranges from about 50% when the physiological signal is zero to 100% when the physiological signal is above the periodic signal maximum amplitude. The logic output of each of the “straight” and the complement AND gates corresponding to one directional control is connected to a different OR gate, and the outputs of the two other AND gates are connected, but in reverse order, to the same OR gates. The logic outputs of the comparator complement AND gates are connected to the OR gate commanding the directional control relay corresponding to the directional control button attached to those AND gates. The logic outputs of the comparator “straight” AND gates are connected to the OR gate commanding the opposite directional control relay. The resulting effect upon the game is to create a wavering of the controlled object that is experienced as a loss of precision controllability. This control variability ranges from a maximum of operating as if the directional control button is applying only a small directional bias to a alternately reversing object, through stages of the object occasionally reversing direction from the commanded direction, to a minimum of no reversing and full, uninterrupted directional control.

[0097] Other functions which can be accomplished by the Enabled/Disabled Gampad Buttons 309 of a video game pad controller (see FIG. 8) may be made contingent on producing high values of the physiological signal 100. Binary functions (either active or not), such as fighting action or availability of weapons or ducking action, for example, can be turned off by breaking the switch closure circuit for that function with a relay closure in series with the circuit. The relay is driven by the binary output of a voltage comparator, internal to the Threshold Detectors & Relays 304, whose threshold level can be manually (or automatically) adjusted as training progresses to shape the physiological response in the desired direction.

[0098] Some game pad controller functions are capable of being operated to several discrete levels. For example, the jumping function may have several discrete levels. In some systems or games, the duration of the game controller jump switch closure (operated by button press) can determine the height of the character's jump. For example, in this instance, three distinct heights can be achieved by pressing the controller button either briefly or longer. This button press action can be duplicated by three reed relays separately controlled by one-shot timers set at short, medium, and long durations (in FIG. 8 these relays and timers are part of the Threshold Detectors, Timers & Relays 305). As shown in FIG. 8, the relays can replace the jump switch closure which is part of the Duration Modulated Gamepad Buttons 306 in the gamepad controller 311, and the selected timer is operated by the game controller jump switch closure. The selected timer is determined by ANDing the game controller jump switch closure with each of the binary outputs of a three-level voltage comparator (or threshold detector), which comparator is also a part of the Threshold Detectors, Timers & Relay 305 in FIG. 8. When the physiological signal exceeds one of the comparator levels, the AND gate to the corresponding timer is opened and the next jump switch closure is passed to that timer, closing its connected relay for the timer's duration. The timer durations are set in ascending order to correspond to ascending levels of the comparator settings. In this way, the player is reinforced for more desirable values of the physiological signal (for example, either higher or lower values, depending on what physiological signal is being monitored) by having the character's jump height capability increased. As in the case of binary functions above, the voltage comparator's threshold level is adjusted as training progresses to shape the physiological response in the desired direction.

[0099] More recent video “game pads” incorporate analog joysticks as well as discrete switches in the game controller. These controllers can be modulated by employing the methods for both the joystick and the gamepad implementations described above. The modulation, enabling, and button press duration modulation operations can also be performed entirely with the software proprietary to the video game manufacturer, requiring no hardware modifications. As described for the joystick implementation above, different physiological signals and their training criteria can be applied to different game functions concurrently, enabling training to be directed at multiple simultaneous objectives.

[0100] In at least one embodiment, a data acquisition and display system 310, such as an analog to digital conversion system, can be used to provide an operator (usually someone other than the player of the game or the trainee, for example, a clinician or researcher) real-time data feedback and an ability to set thresholds and gains of the modulation according to the various physiological signals. In at least one embodiment, this display can be used to provide the operator with a scrolling display of the relative levels of the physiological signal(s), the adjustable comparator thresholds that define controller button enabling and disabling, and the empirically set voltage thresholds that define the various movement speeds and styles (e.g., “tiptoeing”, “limping”, “full run”, etc.), for example. The specific presentation and location of the data display 310, if any, can vary depending upon the embodiment and/or components of the system being used, and/or upon who is monitoring the display (for example, the clinician or the player). However, in the preferred embodiment this data feedback is not displayed on the same screen viewed by the player or trainee, for the reasons explained hereinabove, but rather is displayed separately, for example, on a different display terminal. In at least one embodiment, for example as shown in FIG. 8 as a “Signal Display System” 310 this tracking function can be provided by the J&J I-330 system, running the “USE” software. The operator is thus guided by the display of the relative levels of the signal(s) and the criteria in performing the shaping procedure described above. Further, in at least one embodiment of the invention this data acquisition and display system can be designed to provide for the transmission of the information over the internet to a distant location, such as to a doctor's office, for example. It is also within the scope of the present invention that software could be designed such that the shaping procedure, including the analyzing of the data feedback and the setting of thresholds and gains of the modulation, can be automatically calculated and adjusted by the system.

[0101] In accordance with at least one embodiment of the present invention, the disclosed apparatus and methods can be used specifically for entertainment purposes. For example, a player might simply use the system for the enjoyment and challenge of mastering a game or simulation, as well as the satisfaction of controlling his own physiological states. It is also within the scope of the present invention that a player could entertain himself by personally using data feedback to set challenges, thresholds and goals for himself to achieve.

[0102] Further, it is within the scope of the present invention to utilize a variety of input devices or controllers, as well as to utilize not just joystick or gamepad input ports, but to utilize a variety of other computer input ports as well, and thus varying the connection apparatus, as appropriate, to accommodate the type of port and/or system being utilized. The present invention could possibly utilize interface devices such as touch pads, light pens, power gloves, keyboards and weapons (e.g., hand-held guns), for example. Additionally, it should be noted that when the term “counts” has been used herein the term potentially refferred to any one of a variety of types or forms of information that a computer, video game system, and/or training simulator might receive at a computer or game port. For example, it is within the scope of the present invention that the information that the computer software reads from a port could be in the form of a digital value which can be expressed as a number, counts, etc.

[0103] Additionally it is expressly within the scope of the present invention that substantially all of the aforementioned and/or described hardware used in the above described embodiments of the present invention could be replaced with software. That is to say, for example, many functions, such as comparators, timers, threshold detectors, relays, voltage controlled amplifier, voltage controlled modulator, offset circuitry, AND gates, VCO, baseline deviation deriving system, etc., can be accomplished with software. For example, these functions could possibly be programmed in what is commonly known as a “virtual instrument” environment, such as National Instruments' LabView. In at least one embodiment, this type of environment would allow hardware used in the present invention to be replaced by functional modules represented as icons on a screen, which can be set-up and “wired” together. The resulting software system can then function just like its hardware counterpart, for example, interacting with external signals through analog and digital input and output boards installed in the computer.

[0104] The present invention fully integrates biofeedback training into a true dynamic video game, or realistic simulation, that turns the popular home pastime of playing video games into an opportunity for therapy or self-improvement. It is the prototype of a new generation of computer game environments that train valuable mental skills beyond eye-hand coordination.

[0105] Current systems typically deliver biofeedback in bland, minimally motivating task formats with direct feedback. The present invention's video game or task challenge format motivates trainees to participate in and adhere to the training process through the rewards inherent in mastery of popular video games or job simulators, and without the demand, monotony or frustration potential of direct concentration on physiological signals.

[0106] It should be understood by those skilled in the art that the descriptions and illustrations herein are by way of examples and the invention, or inventions, are not limited to the exact details shown and described. It is understood that the invention(s) is in no way limited to the above embodiments, and is capable of numerous modifications within the scope of the specification and following claims.