Method and Apparatus for Automatically Determining Somatic State

This invention relates to a method and apparatus for automatically determining somatic state of a human subject, and more particularly to quantifying the somatic state of the human subject to which the apparatus is connected.

There is a continuing demand in the field of human psychology for means whereby the somatic state of subjects can be quantified. A subject's somatic state is the mental state due to such things as the subject's emotions (e.g., joy, anger, happiness, sadness, elation, surprise, disgust or fear), level of mental activity (e.g., as a result of doing mental arithmetic or writing a poem) or motor activity (e.g., moving a hand or foot). Heretofore, psychologists have had to rely on interviews with the subject to elicit information about the subject's somatic state. Questions posed to the subject result in answers that require interpretation based on subjective criteria, and the results achieved by such methods are at best relative.

Somatic states tend to be associated with responses in a number of visceral organs (e.g., the heart, stomach and intestines). However, the particular combination of visceral responses that makes a particular stimulus or state "somatic" is not precisely defined. Similar combinations of visceral responses can be indicative of widely different somatic states. In addition to interviewing the subject, conventionally, psychologists also use electroencephalographic ("EEG") waveforms to interpret the subject's mental state. This too introduces subjective criteria into the analysis because a human expert is required to interpret the EEG waveforms.

Recently, neural networks have come to be used as tools to monitor drug-specific EEG power spectra patterns during the administration of drugs, as shown, for example, by Veselis et al., "Analytical Methods to Differentiate Similar Electrographic Spectra: Neural Network and Discriminant Analysis," 9 Journal of Clinical Monitoring 257 (September 1993).

In view of the foregoing, it would be desirable to be able to provide a method and apparatus for automatically determining somatic state, and more particularly to quantifying the somatic state of a human subject to which the apparatus is connected.

It is an object of this invention to provide a method and apparatus for automatically determining somatic state, and more particularly for quantifying the somatic state of a human subject to which the apparatus is connected. Possible applications of such an apparatus and method might be, for example, determining how much concentration it takes for a subject to do mental arithmetic, quantifying the amount and type of emotion that results from the application of a particular stimulus to the subject, determining emotional stability of a subject over a period of time by exposing the subject to the same stimulus at periodic intervals and measuring his/her response, monitoring the effects of having subjects carry out repetitive tasks in order to develop better methods of reducing repetitive stress disorders, and classifying subjects according to their natural abilities.

These and other objects of the invention are accomplished in accordance with the principles of the invention by providing an apparatus and a method for automatically determining the present somatic state of a subject. Characteristic values of the subject (e.g., scalp potential, muscle potential, heart-rate, eye-movement and frequency of eye blinks, or any combination thereof) are detected and corresponding output signals are produced. The output signals are then amplified and digitized. Fourier transformations are performed on the digitized output signals. A set of state variables is determined for each selected frequency sub-band of a selected frequency band for each of the output signals. Each of the sets of state variables is applied to a trained neural network to determine present somatic state of the subject. The present somatic state of the subject is then displayed. In order to train the neural network, sets of reference weights and sets of reference biases are formed for a neural network using sets of state reference variables corresponding to known somatic states.

The above and other objects and advantages of the invention will be apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings, in which the reference numbers refer to like parts throughout and in which:

FIG. 1 is a block diagram of a hardware implementation of a preferred embodiment of the apparatus for automatically determining the somatic state of a subject,

FIG. 2 is a flowchart showing the steps involved in automatically determining the somatic state of a subject,

FIG. 3 is an example of a power spectral density that resulted from the Fourier transform of a typical output signal of one of the electrodes attached to the subject, as shown in FIG. 1,

FIG. 4 shows a typical neural network, and

FIG. 5 shows graphically an output vector of the neural network, indicating that the present somatic state of the subject is a weighted combination of the reference somatic states.

The present invention is an apparatus and a method for automatically determining the present somatic state of a subject. Characteristic values of the subject which are indicative of somatic state are detected using sensors. Most typically, brain waves (which can be measured by measuring scalp potential using electrodes) are used as characteristic values. In addition, muscle potential, heart-rate, eye-movement and frequency of eye blinks, or any combination thereof, can be used as characteristic values. The sensors produce output signals corresponding to the detected characteristic values. The output signals are then amplified and digitized. In order to determine the power spectral density of the output signals, Fourier transformations are performed on the digitized output signals. Since some frequency bands are more important sources of somatic state information, only the most important frequency bands are selected. Each selected frequency band is further divided into several sub-bands. A set of state variables is determined for each sub-band of each selected frequency band for each of the output signal power distribution. Each of the sets of state variables is applied to a trained neural network to determine present somatic state of the subject. The present somatic state of the subject is then displayed.

The neural network is trained by forming sets of reference weights and sets of reference biases from sets of state reference variables determined from output signals detected during known somatic states. For example, consider the process of training the neural network to recognize two somatic states, the two emotional somatic states corresponding to happiness and sadness. The subject either imagines a happy thought or is made to be happy. Output signals corresponding to measured characteristic values are produced, amplified and digitized. A power spectral density of the output signal from each sensor is determined by performing the Fourier transformation on the digitized output signals. A set of state reference variables for happiness is obtained from the values of power spectral density from each sub-band of the selected frequency bands. The process is repeated, when the subject is feeling sad, in order to generate a set of state reference variables for sadness.

The purpose of the training phase is to generate a set of reference weights and a set of reference biases that can be used to determine unknown emotional somatic states as either happiness, sadness or neither. A method such as the back-propagation method with supervised learning is often employed. Initially the reference weights and the reference biases are set to random numbers. Then the set of state reference variables for happiness is applied to the neural network and the output of the neural network is compared to the desired output (i.e., happiness) and the values of the reference weights and the reference biases are adjusted accordingly. The process is repeated for sadness. In practice, the process may be repeated for many sets of reference variables corresponding to the emotional somatic states of happiness and sadness in order to further refine the reference weights and reference biases and improve the neural network's ability to recognize happiness and sadness. Ultimately, when the detection error rate has reached an acceptable level, a single set of reference weights and a single set of reference biases that can be used to determine unknown emotional somatic states as either happiness, sadness or neither is generated.

The invention is not limited to distinguishing between two somatic states (the above described emotional states of happiness and sadness). The neural network can be trained to recognize other somatic states (such as the emotional states of joy, anger, happiness, sadness, elation, surprise, disgust or fear, level of mental activity or motor activity).

FIG. 1 is a block diagram of a hardware implementation of a preferred embodiment of the apparatus for automatically determining the somatic state of a subject 110. The stimulus 100, which is applied to the subject 110, may be an externally applied signal such as noise, music or a television picture, or the stimulus 100 can be mental images generated internally by the subject 110 himself/herself. Affixed to the subject 120 is a plurality of sensors 120.

FIG. 2 shows the steps involved in the process 280 of automatically determining the somatic state of the subject 110. In step 200, the sensors 120 detect characteristic values of the subject 110. Characteristic values refer to those traits, such as scalp potential, muscle potential, heart-rate, eye-movement and frequency of eye blinks of the subject 110, which are indicative of the subject's 110 present somatic state. Indeed, any combination of such characteristic values can be used.

In step 210 of the process 280, the output signals from the sensors 120 are connected to a preamplifier 130 which amplifies the output signals by a predetermined amount. The output signals are then passed through to the main amplifier 140 which amplifies them yet again.

In step 220 of the process 280, the amplified output signals are fed into a computer 150 which digitizes the output signals.

In step 230 of the process 280, the computer 150 performs Fourier transformations on the output signals. The Fourier transform is a well known operation whereby the power spectral density of a signal can be obtained. The Fourier transforms of most signals have both real and imaginary components. Power spectral density is computed by taking the sum of the squares of the real and imaginary components of the Fourier transform for each frequency value of the transformed output signal.

FIG. 3 shows an example of a power spectral density that resulted from the Fourier transform of a typical output signal of one of the sensors 120. In FIG. 3, the duration of the output signal was T seconds and the frequency samples are separated by 1/T Hz. The highest frequency is 1/2.tau. Hz, where .tau. is the sampling rate at which the output signals are digitized in step 220. Thus, there are (1/2.tau.)/(1/T)=T/2.tau. values for power spectral density.

In step 240, the computer 150 determines a set of state variables for each selected frequency sub-band of a selected frequency band for each of the output signals. During the training phase, in step 250, the set of state variables will be used to train the neural network 280. During normal operation of the neural network, this set of state variables will be applied to a neural network 494 to determine the present somatic state of the subject 110, in step 260.

Brain waves are typically divided into several frequency bands. FIG. 3 shows a power spectral density of only the .alpha. band, a band of frequencies between about 8 Hz and about 13 Hz. The .alpha. band is an important source of information about somatic state. In addition to the .alpha. band, there is the .delta. band (from about 1 to about 3 Hz), the .theta. band (from about 4 to about 7 Hz), the .beta. band (from about 13 Hz to about 30 Hz) and the .gamma. band (above about 30 Hz). The following description employs only the .alpha. band, but the invention can be applied to any of the other bands. Characteristic values other than brain waves (for example, muscle potential, heart-rate, eye-movement and frequency of eye blinks, or any combination thereof) can also be divided into multiple frequency bands, so the following description applies.

In FIG. 3, the .alpha. band is sub-divided into three sub-bands, .alpha..sub.1, .alpha..sub.2 and .alpha..sub.3. Such sub-division can be accomplished by band-pass filtering. In this example the .alpha..sub.1 sub-band extends from about 8 to about 9.5 Hz, the .alpha..sub.2 sub-band from about 9.5 to about 11 Hz and the .alpha..sub.3 sub-band from about 11 to about 13 Hz. For purposes of illustration, assume that there are six sensors 120 attached to the subject 110 for each frequency band. A set of state variables can be created from cross-correlating pairs of the spectral power values of the sensor output signals. The cross-correlation operation for EEG potential V.sub.j measured at the jth electrode and the EEG potential V.sub.k measured at the kth electrode site where j=1, 2, . . . , 6 and k=1,2, . . . ,6 is defined as: ##EQU1## where the <> operator means average over a 5 second interval. Similar expressions are obtained for each of the .alpha..sub.1, .alpha..sub.2 and .alpha..sub.3 sub-bands. There are then .sub.6 C.sub.2 =(6.times.5)/2=15 combinations of pairs of sensors 120 for each frequency band and a total of 3.times.15=45 cross-correlation coefficients which can be used as a set of state variables. Another set of 18 state variables can be generated by adding powers (mean squares) of the electric potential of each of the 6 electrodes for each of the 3 bands. In fact, a set of 63 state variables can be generated by combining the set of 45 state variables with the set of 18 state variables.

Alternatively a set of 306 state variables can be generated from the 51 values of the Fourier transformation from 5 to 15 Hz for each of the 6 sensors 120 where the Fourier transformation is performed every 5 seconds.

In step 250, the computer 150 forms sets of reference weights and sets of reference biases for a neural network. Step 250 is also referred to as the neural network training phase. Each of the sets of reference weights and each of the sets of reference biases is formed from applying to the neural network a set of state reference variables corresponding to a known somatic state.

FIG. 4 shows a typical neural network 494 composed of two layers, an input layer 490 and an output layer 492. Neural networks with more layers and more neurons can also be used. However, the output layer 492 will always comprise a neuron for each somatic state to be determined. In this case, there are three somatic states to be detected, so output layer 492 has three neurons. The advantages to using neural networks are manifold. In order to build a system which can determine somatic information with high accuracy, the ability of a neural network to use multi-sensor inputs (scalp potential, muscle potential, heart-rate, eye-movement and frequency of eye blinks of the subject, etc) is a big advantage over having a human expert manually review the data. Conventional methods require the human expert to interpret an EEG graph. Indeed, it is possible to train a neural network using incomplete, noisy, redundant or even contradictory data--a big advantage over so-called "expert-systems" which require a complex set of formal rules.

Non-linear transfer functions are sometimes used for in the transfer function stage 430 to compress their inputs (which may have any value between plus and minus infinity) into the range of 0 to 1. A sigmoid transfer function can be used as the non-linear transfer function because it is differentiable. However, since the sigmoid function saturates (produces a value close to one) for most positive inputs, use of the sigmoid function results in a loss of information about the intensity of the somatic state. Therefore in order to preserve maximum information about the intensity of the somatic states, and to avoid the problems associated with a sigmoid function, the function preferably used in the transfer function stage 430 during normal operation is such that for inputs greater than or equal to zero its output is slowly saturating (for example, logarithmic or linear), while for inputs less than zero its output is zero. During the training phase, the function preferably used in the transfer function stage 430 is slowly saturating (for example, logarithmic or linear) for all inputs.

Consider an input layer 490 of neural network 494 composed of 63 inputs 400 x.sub.1,x.sub.2, . . . ,x.sub.63, a weighting stage 410, a biasing and summing stage 420, a transfer function stage 430 and outputs 440 z.sub.1, z.sub.2, z.sub.3. Output layer 492 consists of outputs 440 z.sub.1, z.sub.2, z.sub.3. In the training phase, transfer function 430 is a linear transfer function and the weights and biases are adjusted such that outputs corresponding to somatic states 0, 1 and 2 approach z=[1,0,0], [0,1,0] and [0,0,1] respectively. During operation of the trained neural network, the function 430 is such that for inputs greater than or equal to zero its output is linear, while for inputs less than zero its output is zero.

Biases help the neural network 494 to represent relationships between input 400 and output 440 more easily because a neural network layer without a bias has a net output to the following layer of zero when all of its inputs are zero.

At the start of the training phase the weights, represented by the matrix C and the biases, represented by the matrix B are set to random values. When the first set of state reference variables corresponding to a known somatic state is applied to the input 400 and propagated through the neural network 494, the output 440 will likely not indicate the correct somatic state. The output 440 is compared with the desired output 440 and an error is calculated. Training is completed when the difference between the actual output 440 and the desired output 440 is less than a selected value (calculated as the sum of the squares of the difference between the desired and actual output 440). This process is repeated for the sets of state reference variables corresponding to the known somatic states which are to be recognized.

In step 260, the computer 150 applies each of the sets of state variables to the neural network 494 which is formed with the sets of reference weights and the sets of reference biases. Using the neural network 494 to interpret the state variables, the computer 150 is able to determine the present somatic state of the subject 110.

The operation of a trained neural network 494 can be illustrated in reference to FIG. 4 and the following equations. The result of the weighting stage 410 and the biasing and summing stage 420 is expressed as a vector Y=[y.sub.1, y.sub.2, y.sub.3 ], whereby ##EQU2## The vector Y is then operated on by the linear transfer function stage 430 which creates vector Z: ##EQU3##

In step 270, the computer 150 displays the present somatic state of the subject 110 on a display 60, as shown in FIG. 5.

The neural network 494 of FIG. 4 can recognize combinations of three different reference somatic states, represented by the set of output 440 vectors Z={[1, 0, 0], [0, 1, 0] and [0, 0, 1]}. The appearance of an output 440 vector, for example [0.8, 0.6, 0.1] as shown graphically in FIG. 5, indicates that the present somatic state of the subject 110 is a weighted combination of the three reference somatic states. Analysis of the somatic state can be performed in nearly real time (for example, every 5 seconds) and monitoring the time change of the somatic state is useful for bio-feedback.

When a sigmoid function is used in the transfer function stage 430 output values tend to cluster, or saturate, about 1 or 0. This is an undesirable effect because information about the relative intensity of a particular somatic state is lost. In particular, a somatic state of strong intensity will always appear as a one.

To avoid the problems associated with a sigmoid function, during normal operation, the function used in the transfer function stage 430 is such that for inputs greater than or equal to zero its output is slowly saturating (for example, logarithmic or linear), while for inputs less than zero its output is zero. During the training phase, the function used in the transfer function stage 430 is slowly saturating (for example, logarithmic or linear) for all inputs.

A neural network of the present invention that has been trained at a particular time to recognize certain somatic states of a given subject will be able to recognize any such somatic state for that subject at any other time. Further, when the reference somatic states are specific enough (for example "joy coming from achievement"), a neural network trained for one subject can be used for any other subject--a universal database.

Consider the application of the present invention to reading the state of mind of a subject who has lost the ability to express his/her wishes verbally or by bodily motion. Using the present invention, such a patient can still communicate with his/her family. Indeed the display 160 could consist of a battery of lamps, each lamp corresponding to a different somatic state. Somatic state intensity is indicated by the brightness of the lamp.

Thus it can be seen that a method and an apparatus for automatically determining the somatic state of a subject, and more particularly for quantifying the somatic state of a human subject, is provided. One skilled in the art will appreciate that the present invention can be practiced by other than the described embodiments, which are presented for purposes of illustration and not of limitation, and the present invention is limited only by the claims which follow.