Management of physiological and psychological state of an individual using images biometric analyzer

[0002] This application claims is a continuation of U.S. patent application Ser. No. 09/031,245, filed Feb. 26, 1998, inventors Patton et al., U.S. Pat. No. 6,102,846 for A SYSTEM AND METHOD OF MANAGING A PSYCHOLOGICAL STATE OF AN INDIVIDUAL USING IMAGES.

[0003] This invention relates in general to the management of a physiological and/or psychological state of an individual and more particularly to the management of the physiological and/or psychological state of an individual through the use of images which have been customized for use by the individual and which can be part of a self-help process.

[0004] The physical, emotional and mental well-being of an individual can contribute greatly to the quality of life of that individual. In our hyperactive, hyperkinetic world, stress results in numerous physical reactions, such as, headache, muscle tension, dizziness or sleeplessness, weight gain, chronic coughing, nervous ticks, stomach upset and shortness of breath. Job stress alone is estimated to cost American business $300,000,000,000 annually. Stress is the response of the body and/or mind to a demand placed upon it. Stress can be caused by major events in one's life, such as, death of a loved one, marital breakup, personal injury or sickness, and job loss. Stress can also result from our day-to-day hectic style of living, where one attempts to excel simultaneously at being a super employee, a super parent, a super spouse, and a super citizen. Unless chronic stress is controlled, one puts oneself at risk for a host of serious problems, such as, heart disease, stroke, migraines, muscle and nerve disorders.

[0005] The typical path to obtain relief from stress is to visit one's doctor. Stress conditions result in up to 70% of all doctor's visits. Typically, drugs are prescribed to relieve stress. One stress reducing medication alone accounts for $6,000,000 per day in sales. Thus, alternative approaches to traditional medicine have become increasingly popular. Resort to Eastern religions, transcendental meditation, and biofeedback techniques have been proposed to empower the individual to reduce stress without the potential deleterious effects of powerful and expensive prescription drugs or invasive surgery.

[0006] It has been proposed to use images for the purpose of optimizing one's physiological and psychological state. There are several reasons for this.

[0007] (1) It has been shown that one can measure a reliable physiological response for images that differ in valence and arousal. It has been demonstrated that images rated differently with respect to perceived activation and pleasantness elicited physiological responses of different magnitude. Thus, magnitude of the skin conductance response correlated with perceived arousal level produced by pictorial stimuli. At the same time heart rate acceleration during first 4 to 5 seconds of image presentation reflected “valence” or degree of perceived pleasantness of an image. Other physiological parameters that reflect an individual's physiological reactions to images have also been demonstrated. These results imply that, for an individual viewer, images can potentially be classified based on one's physiological reactions in terms of emotional arousal.

[0008] (2) Imagery is known to be able to change a person's state. Paintings, movies, pictures are constantly affecting our mood and performance level. Power of visualization and affective content determine effective use of imagery in therapeutic sessions. Experimental research has also shown that presentation of images of similar content may cause significant shifts in physiological reactions.

[0009] (3) Digital imaging technology provides an almost instant access to image databases through the internet. Moreover, the potentially unlimited degree of digital manipulation makes images very attractive means of interaction and communication. Images can be easily transformed to alter or enhance people's preferences, i. e., for hue, saturation, depth, aesthetic feelings, etc. Image transformation by itself can provide biofeedback information to the user to facilitate learning how to control one's physiological and emotional state, e. g., stress.

[0010] Following are several proposals to use images as a means of changing one's state that have not proven to be entirely successful.

[0011] U.S. Pat. No. 5,465,729, issued Nov. 14, 1995, inventors Bittman et al. and U.S. Pat. No. 5,343,871, issued Sep.6, 1994, inventors Bittman et al., disclose the use of measurements of electrophysiological quantities to control a presentation to a subject of a series of prestored audio-visual sequences.

[0012] U.S. Pat. No. 3,855,998, issued Dec. 24, 1974, inventor Hidalgo-Briceno discloses an entertainment device that includes sensing means connected to the user for sensing galvanic skin response and brain theta waves. According to a given measured state of a user the device provides a given type of predetermined audio-visual stimulation to the user for a timed interval to hold one in or move one toward a desired state. At the end of the interval, the user's state is again measured and a further timed audio-visual response according to the measured state is presented to the user.

[0013] U.S. Pat. No. 5,596,994, issued Jan. 28, 1997, inventor Bro, discloses an automated and interactive positive motivation system that allows a health care professional to produce and send a series of motivational messages to a client to change or reinforce a specific behavioral pattern.

[0014] U.S. Pat. No. 5,304,112, issued Apr. 19, 1994, inventors Mrklas et al., discloses an integrated stress reduction system which detects the stress level of a subject and displays a light pattern reflecting the relationship between the subject's stress level and a target level. The system also provides relaxing visual, audio, tactile, environmental, and other effects to aid the subject in reducing one's stress level to the target level.

[0015] U.S. Pat. No. 4,632,126, issued Dec. 30, 1986, inventor Aguilar, discloses a biofeedback technique which permits simultaneous, preferably redundant, visual and auditory presentation on a color TV of any intrinsically motivating stimuli together with continuous information pertaining to the physiological parameter to be controlled. As the subject changes a certain physiological parameter, the image and sound become clearer if the change occurs in the desired direction.

[0016] U.S. Pat. No. 5,253,168, issued Oct. 12, 1993, inventor Berg, discloses a system for allowing an individual to express one's self in a creative manner by using biofeedback signals to direct imaging and audio devices.

[0017] U.S. Pat. No. 5,676,138, issued Oct. 14, 1997, inventor Zawalinski, discloses a multimedia computerized system for detecting emotional responses of human beings and changes thereof over time.

[0018] U.S. Patent 5,047,930, issued Sep. 10, 1991, inventors Marten, et al., discloses methods of analyzing physiological signals from a subject and analyzing them using pattern recognition techniques to determine a particular sleep state of the subject. Use of any associated feedbacks is not disclosed.

[0019] The following papers discuss various emotional responses and physiological responses of subjects to viewing images.

[0020] Affective judgement and psychophysiological response: dimensional covariation in the evaluation of pictorial stimuli; by: Greenwald, Cook and Lang; Journal of Pyschophysiology 3 (1989), pages 51-64.

[0021] Remembering Pictures: Pleasure and Arousal in Memory, by: Bradley, Greenwald, Petry and Lang; Journal of Experimental Psychology, Learning Memory and Cognition; 1992, Vol. 18, No. 2, pages 379-390.

[0022] Looking at Pictures: Affective, facial, visceral, and behavioral reactions; by: Lang, Greenwald, Bradley, and Hamm, Psychophysiology, 30 (1993), pages 261-273.

[0023] Picture media and emotion: Effects of a sustained affective context; by: Bradley, Cuthbert, and Lang, Psychophysiology, 33 (1996), pages 662-670.

[0024] Emotional arousal and activation of the visual cortex: An fMRI analysis; by: Lang, Bradley, Fitzsimmons, Cuthbert, Scott, Bradley, Moulder, and Nangia; Psychophysiology, 25 (1998), pages 199-210.

[0025] The techniques disclosed in the above references have the following disadvantages.

[0026] 1. There is no development of a personal image profile of an individual so as to provide for customized images which are specifically tailored for the individual so as to move the individual to a desired physiological and/or psychological state. This is important since an image which is restful for some may be stressful for others.

[0027] 2. The images or other stimuli for inducing change in state in an individual are preselected by someone other than the user. The selection is often based on the effect of the images on a large number of subjects rather than being personalized for the individual.

[0028] 3. Where measurement of physiological parameters are used as part of the state change technique, the measurement devices are often large and not very portable and therefore not conducive for use at work, at home or during travel.

[0029] According to the present invention there is provided a solution to the problems referred to above.

[0030] According to a feature of the present invention there is provided a method of determining the physiological reactivity of an individual with respect to images, comprising: measuring at least one physiological parameter of an individual during a baseline calm period of time; measuring the said at least one physiological parameter of said individual during a stress period of time and a succeeding rest period of time; dividing said measured data for each of said periods of time into a plurality of predetermined time segments; computing a histogram or Fourier analysis as appropriate on said predetermined time segments for each said period of time; standardizing said computed data; conducting a principal component analysis, a canonical discriminant analysis, or a neural network, on said data to establish a baseline calm or rest period set of scores and a stress period set of scores; measuring the said at least one physiological parameter of said individual during an image presentation period of time; repeating said time segmenting, said histogram or Fourier analysis, and said standardizing steps on said measured physiological data for said image presentation period of time; applying said vectors from said conducting step to said image data to produce an image set of scores which are compared with said baseline and stress sets of scores; and determining whether said presented image is activating, deactivating, or neutral based on said comparison.

[0031] The present invention has the following advantages.

[0032] 1. An individual is profiled to provide customized images which are specifically tailored for the individual to move the individual to a desired physiological and/or psychological state.

[0033] 2. The images or other stimuli for inducing change in the state of an individual are not preselected by someone other than the user, but rather by the user.

[0034] 3. A portable device is used to measure physiological parameters to predict an individual's state. The portable device is conducive for use at work, at home, during travel, or during exercise.

[0035] FIGS. 1 and 2 are flow diagrams useful in explaining the present invention.

[0036] FIGS. 3-5 are diagrammatic views illustrating several embodiments of a portable physiological sensor monitor.

[0037] FIGS. 6-11 are graphical views useful in explaining certain aspects of the present invention.

[0038] FIGS. 12 and 13 are diagrammatic views useful in explaining other aspects of the present invention.

[0039] FIG. 14 is a block diagram of the system of the present invention.

[0040] General Discussion

[0041] In general, as shown in FIG. 14, the system 100 of the present invention includes several interrelated components that can be used to help one to manage one's physiological and or psychological state. These components will be described in greater detail later but, in general, include one or more of the following:

[0042] 1. Portable Biosensor Device (102)

[0043] A portable biometric device that is worn or carried by a user and which senses and records physiological parameters on a continuous basis.

[0044] 2. Master Set of Images (104)/Therapeutic Image Classification System (106)

[0045] A set of images presented to a user to determine the user's physiological and cognitive image preferences. The images are classified according to a therapeutic image classification system.

[0046] 3. Biometric Analyzer (108)

[0047] A biometric analyzer which extracts the physiological activation state of user from one or more measured physiological parameters.

[0048] 4. Cognitive Analyzer (110)

[0049] A cognitive analyzer which extracts cognitive state from cognitive responses to images.

[0050] 5. Personal Image Profiler (112)

[0051] A personal profiler which combines the physiological and cognitive measures obtained from the biometric analyzer and cognitive analyzer to generate an individual's personal image profile for a given state response.

[0052] 6. Personal Image Classifier (114)

[0053] A personal image classifier which, based on an image bank having images which have been classified using a therapeutic image classification system, and on the personal image profile, selects activating and deactivating images to create a personal image set.

[0054] 7. Visualization System (116)

[0055] A visualization system which presents the personal image set to a person with the goal to help manage, modify or maintain current physiological and psychological state.

[0056] The components of system 100 can take different forms, depending on the application. For example, the portable biosensor device 102 measures one or more physiological parameters of an individual. The measurements can be recorded in the device and appropriate resident software used to analyze the state of the individual. Alternatively, the measured physiological parameters can be transmitted over a wireless channel to a server where they are recorded and analyzed. A warning signal can then be transmitted back to the portable device to warn the user of the need to manage one's state.

[0057] Components 104-112 can reside as software in a computer that is located with the individual, at a health care professional's office, or the like. The images selected by component 114 can reside in local or remote database(s) that can be communicated with over standard communication links (public telephone, cell phone, internet/world wide web, intranet, etc.). The visualization system 116, includes a display which can form part of a computer, a television, a handheld device, such as a PDA, game, or entertainment device, a slide projector, a cell phone. The visualization system can include devices such as a CD player, a DVD player, a VCR, etc. The system can include devices for other sensory feedback, such as, auditory, olfactory, tactile (heat, vibratory), etc. The applications of the present system are set out in greater detail below.

[0058] The interrelated use of these components is set forth in FIGS. 1 and 2. As shown in FIG. 1, the process 10 is started (bubble 12). It is determined (diamond 14) if this is a first time use. If the answer is no, the process of FIG. 2 is carried out (A). If the answer is yes, the process continues (box 16) where an appropriate visualization system presents a master set of images to the user. If the Portable Biosensor Device has been used, it is docked to the visualization device to give a record of physiological parameters of the user measured over a period of time (box 18).

[0059] The biometric analyzer(box 20) and cognitive analyzer measures (box 22) physiological and cognitive states from the user during presentation of the master set of images.

[0060] The personal profiler (box 24) generates the user's personal image profile based on the combined physiological and cognitive measures.

[0061] Based on the personal image profile and a therapeutic image classification system (box 26) for images in a therapeutic image data bank (databases) (box 28), activating and deactivating images are selected from the image data base(s) to create a personal image set (box 30).

[0062] The user then decides (diamond 32) if he or she wants to have a session. If no, the session ends (bubble 34). If yes, the process continues to A in FIG. 2.

[0063] Once a personal image set has been established, the user can start a session (A). The Biometric Analyzer (box 40) and Cognitive Analyzer (box 42) can be used to determine a user's desired direction/preference for a session (e.g., relaxation, optimal performance, excitation (box 44).

[0064] Based on the inputs, the Personal Profiler (box 46) decides if the current Personal Image set will work or if an updated, Personal Image profile is needed. The Personal Profiler can also receive inputs from a Portable Biosensor Device (box 48) and from a user's physiological, cognitive and image use history from a secured data base (box 50).

[0065] If the current image set is determined to be OK (diamond 52), the visualization device presents images to the user according to one's preferences. The duration and/or sequence of presentation, the type of transformation of the images are performed based on users physiology. (box 54) Input from a “Coach” (box 56) may also be provided. The “Coach” monitors physiological responses of the user and provides feedback in form of visual feedback, verbal reinforcement, verbal suggestions and new techniques.

[0066] The user then decides to continue or not (diamond 58). If yes, the process is returned to A. If no, the process is ended (bubble 60).

[0067] If the current image set is determined to be not OK (diamond 52), the process is operated in a learning mode (box 62) where other images from an image bank are shown on a trial and error basis. The user may wish to create an updated profile (diamond 64). If “yes”, the process continues to “B” in FIG. 1. If “no”, the process is ended (bubble 60).

[0068] Following are more detailed descriptions of each of the components described above.

[0069] In medical compliance (taking medicine regularly, exercising regularly etc), it may be beneficial for a user to have a system that tracks, reminds, and rewards the user. On the same token, for an excellent individualized biofeedback based wellness management program, The Portable Biosensor Device tracks and reminds the user to perform wellness management as needed.

[0070] The Portable Biosensor Device is a portable device having one or more sensors that record physiological parameters of an individual wearing the device. Different individuals react differently to different sensors under different situations. Through individual sensor response profile (as explained in personal profiler section) we will be able to produce a personalized device. The device contains multiple sensors to measure temperature, heart rate variability (HRV) (measured either from ECG, photoplethysmographic methods or continuous blood pressure), SCR (skin conductance response), EEG, EMG, eye saccades etc.

[0071] The device will accommodate different sensor sets based on the embodiment. For example as shown in FIG. 5, a wrist type device 70 with sensors 72 and computer 74 can record temperature, HRV through continuous blood pressure monitoring, and SCR. A head band type of device 80 with sensors, 82 connected to computer 84 (on waist band-not shown) shown in FIG. 3 can measure EEG and EMG. As shown in FIG. 4, an earphone type of device 90 with sensors 92 connected to computer 94 (on waistband not shown) could measure temperature, heart rate variability through photoplethysmographic methods, and SCR.

[0072] The portable biometric device is microprocessor based and records the user's physiology throughout the day, especially between sessions. Using digital signal processing on the sensor data, it will analyze (or analyze using the Profiler) and make predictions on the individual's state. Predictions will be made either using phasic physiological responses such as change in heart rate or SCR, or using sophisticated techniques such as Independent Component Analysis or pattern recognition. For example, increased heart rate and SCR could indicate activation or excitement, however, more sophisticated analysis could differentiate between excitement to startle and excitement in defense. According to Cacioppo et al (1996), though both the startle response and the defense response are associated with increased heart rate and SCR, they exhibit different patterns of activation. In the case of the startle response, the heart rate acceleration peaks and returns to near normal levels within two seconds, whereas in the case of the defense response, the heart acceleration does not begin to rise for several seconds and peaks much later.

[0073] Moreover, if the user chooses to know, the feedback to the individual user can be provided through either vibration (tactile or kinesthetic), auditory, or visual means. The data recorded in the device can either be stored on the device or transmitted to an individual server via wireless communication channel.

[0074] The Biometric Analyzer plots, on a two/multi dimensional plot, physiological reactivity of each individual for different situations such as

[0075] Baseline

[0076] Different type of stressors (active coping task such as mental arithmetic, passive task such as situation narration)

[0077] Calmed state

[0078] Energized state

[0079] It should be noted that

[0080] 1. The reactivity to specific images can also be plotted on this plot, and mapping is performed to cluster images in various groups.

[0081] 2. Various sensor measures, such as EEG, EMG, HRU, eye saccades, hand temperatures, etc., can be simultaneously used.

[0082] 3. Clustering of images into various groups can be done using techniques such as Euclidean distance, ratio of distances etc.

[0083] 4. Plotting can be done using different techniques such as principal component analysis, or independent component analysis, wavelet, neural networks, time series, and other signal processing techniques.

[0084] One such technique (CLMOD) using principal component analysis, mapping images between a baseline and arithmetic stressor, using eye saccades, heart rate and EMG measures, and using a ratio of distance of the image to the stress to the distance of the image to the baseline is explained in more detail below. In general, this technique determines which images are physiologically “activating” or “deactivating”. The technique can be implemented as follows.

[0085] A subject is seated in a comfortable chair before a display monitor. Sensors are attached to the subject to record biological information, such as, finger temperature, muscle tension, and heart rate. The physical responses are recorded while the subject views images presented on the monitor and while doing mildly stressful activities. The data is collected several (e.g., 256) times a second, while at rest, while viewing the images, and while cognitively rating them, as well as while talking about oneself and during a mental arithmetic task and during rest periods after each stress test.

[0086] A subset of the physiological measures from these time periods is selected for use. The data is prepared using Fourier analysis for some physiological measures and histograms for other physiological measures.

[0087] The data from the baseline, stress and rest time periods are broken into multiple, non-overlapping 15 second segments, and then a histogram or a spectrum computed from a Fourier analysis is used for each time segment. The histograms and/or spectra for each time segment are then fed into a Principal Component Analysis (PCA). In a preferred embodiment of this method, either Canonical Discriminant Analysis or Neural Networks might replace PCA. The result of the PCA analysis is that, (1) a set of weights called “loadings” is created, and (2) a set of “scores” or summary values, for each time segment is created. The data from the image periods are prepared using Fourier analysis and histograms, and the loadings are applied to these image period Fourier spectra and histograms. The result is a set of “scores” for each image period.

[0088] The image period scores are then compared to the scores for the baseline, stress and rest time segments. An image score that is “close” to the centroid of the baseline scores indicates an image that is “deactivating”. An image that is close to the centroid of the stress scores indicates an image that is “activating”. An image score that is not “close” to either the centroid of the baseline scores or the centroid of the stress scores indicates an image that is neutral. What is meant by “close” can be determined in several ways. One technique is to determine the Euclidian distance from each centroid and then create the ratio of the distance to baseline centroid divided by distance to stress centroid. The difference between the image score and the blank period score can also be used instead of the image score itself.

[0089] Following is a more detailed description of the CLMOD Analysis.

[0090] 1. Take physiology for baseline, discard first 2 minutes and last 2 minutes and chop remainder into non-overlapping 15 second segments. Call the data in these segments B₁through B₂₄. (An example of heart rate and EMG data for two consecutive 15 second segments are shown in FIGS. 6 and 7 respectively)

[0091] 2. Take physiology for Stress 1, chop into non-overlapping 15 second segments. Call the data in these segments S1₁through S1₁₂.

[0092] 3. Take physiology for Stress 2, chop into non-overlapping 15 second segments. Call the data in these segments S2₁through S2₁₂.

[0093] 4. Take physiology for Rest 1, chop into non-overlapping 15 second segments. Call the data in these segments R1₁through R1₁₂.

[0094] 5. Take physiology for Rest 2, chop into non-overlapping 15 second segments. Call the data in these segments R2₂through R2₁₂.

[0095] 6. For each data segment B₁-B₂₄, S1₁-S1₁₂, S2₁-S2₁₂, R1₁-R1₁₂, R2₁-R2₁₂, perform the following calculations:

[0096] (a) Take the heart rate data and compute the periodogram (Fast Fourier Transform). Interpolate this periodogram so that the height of the periodogram is available at pre-specified intervals. An example of two periodograms that corresponds to the data shown in FIG. 7 is shown in FIG. 8.

[0097] (b) Take the EMG data and compute the histogram, using pre-specified bin widths. Store the percent of data in each bin. An example of EMG histograms corresponding to the data shown in FIG. 6 is shown in FIG. 9.

[0098] 7. Combine the heart rate interpolated periodogram, EMG histogram percents and Eye saccade histogram percents into one data set, where the rows are the different data segments and the columns are the histogram bins and/or periodogram heights. The histograms need to be aligned (and padded with zeros if necessary) so that the data in each column represents the same bin.

[0099] 8. Scale this data set as follows: Subtract from each data point the mean of the column it is in. Each column then has a mean of zero. Each of the columns related to the heart rate interpolated periodogram has a variance (not standard deviation) of 1/n_H, where n_His the number of pre-specified frequencies to use in the heart rate FFT. Each of the columns related to the EMG histograms has a variance of 1/n_EMG, where n_EMGis the number of such columns. Each of the columns related to the Eye Saccades histograms has a variance of 1/n_EYE, where n_EYEis the number of such columns. This scaling ensures that heart rate, EMG and Eye saccades contribute equally to the next analysis. An example of the result of this step is shown in FIG. 10, where the scaling has been performed not just for the two 15 second intervals shown on the plots, but across the entire set of 15-second segments as explained above.

[0100] 9. Perform Principal Component Analysis (PCA) on this data, retaining the first 5 dimensions. (The number 5 was chosen arbitrarily, and it can vary from subject to subject.) Store the PCA scores in five dimensions.

[0101] 10. For each image period, perform the analyses described above in 6a, 6b, 6c, 7, and 8. In step 8, use the mean calculated in step 8, not a new mean calculated on the Image period data. Take care to align the columns of the histograms to match the way the columns are aligned for the baseline, stress and rest data. Call these data segments I₁-I₈₂. Apply the PCA vectors from step 9 to the I₁-I₈₂data segments to compute PCA scores in five dimensions. Append these scores with the PCA vectors computed in step 9.

[0102] 11. Plot the PCA scores in scatterplots, with different symbols for the different groups. An example of such a scatterplot is shown in FIG. 11.

[0103] 12. Compute the distance in n_ddimensions (where n_dis some pre-specified number) of each image location in PCA space from the centroid (mean PCA score) of each of the baseline, stress and rest period data. The metric for activation and/or de-activation is any one or more of the following. A threshold or cutoff needs to be set to pick which images are activating or de-activating or neutral.

[0104] a).Distance from Baseline (or calmed state) Centroid

[0105] b).Distance from Stress 1 (or activated state) Centroid

[0106] c).Parks Ratio, which is (distance from baseline centroid)/(distance from stress 1 centroid)

[0107] 10′. In addition to step 10 above, for each blank period perform steps 6a, 6b, 6c, 7, and 8. Call these segments BL₁-BL₈₂.

[0108] 11′. In addition to step 11 above, apply the PCA vectors to data segments BL₁-BL₈₂.

[0109] 11.5′ Subtract the PCA scores for each image segment from the PCA scores from each blank period. Call these data Δ₁-Δ₈₂.

[0110] 12′. Plot the PCA scores for Δ₁-Δ₈₂instead of the PCA scores for the image periods I₁-I₈₂. Also, as in the previous step 12, plot the PCA scores for the baseline, stress and rest periods. The subtracted PCA scores are interpreted as showing the direction and amount of movement due to the change from blank to image period. Thus, we are really plotting the end of a vector whose other end is at the origin. An image that has vector length close to zero shows little physiological movement and can be interpreted as neutral.

[0111] The following steps are used to determine activation and deactivation:

[0112] (a) Determine the angle for each image Δ₁-Δ₈₂. This can be done in nd dimensions (where n_dis some pre-specified number).

[0113] (b) Determine the set of angles for the baseline period data segments. If an angle for Δ₁-Δ₈₂is contained in the range of angles for the baseline period and the length of the vector for each of Δ₁-Δ₈₂is above some threshold, then we say that this image is de-activating. Vectors that point in the baseline direction but are less than this threshold value are considered neutral. (A modification would be to add±k to the range of angles to allow for some uncertainty is our ability to locate the baseline cluster; k might be 10 degrees, we need to experiment to find a good value for k.)

[0114] (c) Determine the set of angles for the stress 1 period data segments. If an angle for Δ₁-Δ₂is contained in the range of angles for the stress 1 period and the length of the vector for each of Δ₁-Δ₈₂is above some threshold, then we say that this image is activating. Vectors that point in the stress 1 direction but are less than this threshold value are considered neutral. (A modification would be to add±k to the range of angles to allow for some uncertainty is our ability to locate the baseline cluster; k might be 10 degrees, we need to experiment to find a good value for k.)

[0115] (d) Vectors that do not point towards either Stress 1 or Baseline are considered “other”. These might be pointing towards other stress modes, or other calming modes, or they may be neutral. We cannot decide from this analysis.

[0116] This scheme is a set of a scene and image related features or attributes (or characteristics) that are relevant to potential therapeutic effect in a broad sense which includes emotional, sensational, cognitive, arousing, esthetical and any other possible impacts registered psychologically or psychophysiologically, that an image may produce while a person viewing the picture. By therapeutic effect, hence, we understand the ability of an image or series of images, video clips, or other visual material alone or in combinations with other modalties purposely presented to improve a person's process, (quality, productivity or effectiveness) performance, state or attitude under consideration which otherwise would become a limiting or negative factor in the person's activities. These aspects are related to the person self, his/her interaction with the outside world (information, specific tasks, etc.) and inter-personal interaction and communication.

[0117] The above features are related to an appearance, content, composition, semantics, intentionally sought impression, uncertainty in interpretation, emotional load and probability of association with a particular emotion, etc. and ideally should represent all dimensions that may influence a holistic impression an image (or other type of visual and other stimulations mentioned above) produces.

[0118] The attributes can be rated in terms of importance and profoundness for each image.

[0119] Subject Matter

[0120] Anything that appears to be a primary subject or part of the primary subject is categorized.

[0121] Defined categories include:

[0122] Landscapes

[0123] Natural or imaginary scenery as seen in a broad view, consisting of one or more of the following elements which dominate a large percentage of and/or being central to the image.

[0124] Mountain

[0125] Water

[0126] Sun

[0127] Vegetation

[0128] Sand

[0129] Snow

[0130] Urban

[0131] Static

[0132] The subject either does not exhibit movement or intentionally poses.

[0133] Active

[0134] Captured at the moment of active motion.

[0135] No expression

[0136] Happy Faces

[0137] A happy facial expression of a person that is the subject matter.

[0138] Unhappy Faces

[0139] Unhappy facial expression of a person is a subject matter

[0140] Children

[0141] People who appear to be 18 years old or younger.

[0142] Family

[0143] The primary subject includes a group of two or more people, regardless of age, exhibiting strong bonds of familiarity and/or intimacy.

[0144] Pets

[0145] A domestic or tamed animal kept for pleasure or companionship (e.g., dogs, cats, horses, fish, birds, and hamsters).

[0146] Pleasant

[0147] A picture of a pet doesn't or is not intended to generate an unpleasant feeling or look.

[0148] Unpleasant

[0149] A picture of a pet does or is intended to generate an unpleasant feeling or look.

[0150] Wild

[0151] An undomesticated or untamed animal in its original natural state, or confined to a zoo or animal show (e.g., lions, elephants, seals, giraffes, zebras, and bears).

[0152] Pleasant

[0153] A picture of a wild animal doesn't or is not intended to generate an unpleasant feeling or look.

[0154] Unpleasant

[0155] A picture of a wild animal does or is intended to generate an unpleasant feeling or look.

[0156] An image which achieves its effect by grouping shapes and colors in satisfying patterns rather than by the recognizable representation of physical reality.

[0157] Other can be used when neither of the defined categories of the subject matter can be applied.

[0158] Sun

[0159] Predominantly distinct shadows are noted. This also includes indoor photos where the subject is directly illuminated by sun through a window. Shadows must be present. If the subject is shading itself, the primary type of light is SUN.

[0160] Sunset/Morning/Evening

[0161] This type of light is typified by long shadows.

[0162] Hazy/cloudy/Overcast

[0163] This type of light produces soft shadows (if any) and the light direction is often obscured, flat, and low contrast.

[0164] Shade

[0165] This light is relatively diminished or partial due to cover or shelter from the sun.

[0166] This also includes Indoor pictures where the subject is illuminated by diffuse daylight from a window.

[0167] Mix Sun and Shade

[0168] This type of light includes spotty sunlight or a mixture of sun and shade.

[0169] Flash

[0170] A brief, intense burst of light from a flashbulb or an electronic flash unit, usually used where the lighting on the scene is inadequate for picture-taking.

[0171] Determined when one or two colors are seeing to be the prominent and global descriptors for a particular picture. If three colors are seen than we define that the picture does not have a dominant hue.

[0172] Red

[0173] Yellow/Orange

[0174] Green

[0175] Blue

[0176] Purple/Magenta

[0177] Brown

[0178] White/Grey

[0179] No Dominant hue (if more than 2)

[0180] Front

[0181] Light shining on the side of the subject facing the camera. Hints: Sunlight conditions where the shadow falls behind the subject. Flash pictures from point and shoot cameras.

[0182] Side

[0183] Light striking the subject from the side relative to the position of the camera; produces shadows and highlights to create modeling on the subject.

[0184] Hints: Sunlight conditions where long shadow falls on the side of the subject.

[0185] Back

[0186] Light coming from behind the subject, toward the camera lens, so that the subject stands out vividly against the background. Sometimes produces a silhouette effect.

[0187] Hints: TV's, Sunsets, and “Lights are Subject” are backlit. In backlit scenes, the shadow may fall in front of the subject, and appears to come towards the photographer.

[0188] Zenith

[0189] Light coming from directly above the subject.

[0190] Hints: High noon lighting. Deep shadows in the eye sockets and below the chin.

[0191] Diffuse

[0192] Lighting that is low or moderate in contrast, such as an overcast day.

[0193] Hints: Diffuse produces no shadows or just barely visible shadows, such as found on a cloudy, hazy day. Some mixed sun and shade pictures are also diffuse when the direction of light can not be determined.

[0194] Multidirectional

[0195] This indicates lighting coming from different directions, such as some stage lighting conditions or in a home where a window is on one side of the subject and a lamp on the other. Multiple shadows should be present if the lighting is from different directions and a flash was used.

[0196] A subjective feeling (introspection) of moving one's eyes primarily along a particular trajectory or in a certain direction while viewing a picture.

[0197] Centered

[0198] Right to left/Left to right

[0199] Up to down/down to up

[0200] Multidirectional

[0201] Distance

[0202] Low 0-9 feet

[0203] Medium 9-20

[0204] High more than 20

[0205] Images from the master image set are presented to the subject to identify his or cognitive response profile in terms of which image attributes and images alter an individual emotional response and arousal level. The individual is asked to rank an image on one or more of three cognitive scales. Preferably, a measure of the cognitive preference computed from the scores along three cognitive scales is used, i. e., Valence, Arousal and Connectedness.

[0206] Definitions of Scales

[0207] Each scale is a two-ended scale with an anchor in the center marked 0. These three scales are described below:

[0208] Scale 1; Detached—Attached (Connectedness)

[0209] −4 −3 −2 −1 0 1 2 3 4

[0210] Detached_Attached

[0211] Detached is a feeling of not being able to personally connect or relate to the object or situation depicted in the image. Attached is a feeling of personnel connection to the object or situation depicted in the image.

[0212] Scale 2; Unhappy—Happy (Valence)

[0213] −4 −3 −2 −1 0 1 2 3 4

[0214] Unhappy_Happy

[0215] Unhappy is a feeling of sadness or disconnect that occurs when you view the object or situation depicted in the image. Happy is a feeling of contentment or satisfaction that occurs within you when you view the object or situation depicted in the image.

[0216] Scale 3; Calm—Excited (Arousal)

[0217] −4 −3 −2 −1 0 1 2 3 4

[0218] Calm_Excited

[0219] Calm is a feeling of tranquillity and silence that occurs within you when you view the object or situation depicted in the image. Excited is a physical state of being in which your feelings and emotions have become aroused that occurs when you view the object or situation being depicted in the image.

[0220] The user is given enough time to provide their reaction to each of the scales. He/she is instructed to follow their first impression. To facilitate the emergence of feelings that could be associated with an image, the users are encouraged to imagine themselves being “in an image” or part of an image that they are viewing. However, the users are requested not to force themselves to feel an emotion. Certain images may be neutral and elicit no emotions. Reactions will be different for each individual.

[0221] All three scales can be used individually to provide a measure of valence, arousal and connectedness. Each of them constitutes a valuable information source related to the person's reaction and can be independently used to assess them. The three scales can be combined to compute the measure of the cognitive preference.

[0222] Current implementation of the cognitive preference computation takes into account the absolute value of the total response, the variance of the ratings along each scale within an individual to normalize the response and logical rules that intend to increase the internal validity of the measure.

[0223] The method and procedure are as follows:

[0224] Step 1

[0225] Every image, I, is subjectively rated along each of the axes Attached/Detached (C), Calm/Excited (A) and Happy/Unhappy (V) such that it has three values C (I), A(I), and V (I) associated with it. R(I_i)=(C²(I_i)+A(I)+V²(I_i))

[0226] Step 2

[0227] Normalize scale values

[0228] C(I_i)=C(I_i)*R(I_i)/max_iR(I)

[0229] A(I_i)=A(I_i)*R(I_i)/max_iR(I)

[0230] CI_i)=V(I_i)*R(I_i)/max_iR(I)

[0231] Where i=1 . . . , 82

[0232] Step 3.

[0233] Compute the standard deviation per scale: $\begin{array}{c}S\left(V\right)=\sqrt{\frac{\sum _i{\left(V\left(I_i\right)-\mathrm{mean}\left(V\left(I_i\right)\right)\right)}^2}{n-1}}\\ S\left(A\right)=\sqrt{\frac{\sum _i{\left(A\left(I_i\right)-\mathrm{mean}\left(A\left(I_i\right)\right)\right)}^2}{n-1}}\\ S\left(C\right)=\sqrt{\frac{\sum _i{\left(C\left(I_i\right)-\mathrm{mean}\left(C\left(I_i\right)\right)\right)}^2}{n-1}}\end{array}$

[0234] Step 4.

[0235] Normalize every scale value from each image using appropriate standard deviations

[0236] C(I_l)=C(I_l)/S(C)

[0237] A(I_i)=A(I_i)/S C)

[0238] V(I_i)=V(I_i)/S(C)

[0239] Step 5.

[0240] If C(I_i)<=1, then C(I_i) is Neutral along the C scale

[0241] If A(I_i)<=1, then A(I_i) is Neutral along the A scale

[0242] If V(I_i)<=1, then V(I_i) is Neutral along the V scale

[0243] Step 6.

[0244] If image is Neutral along the V scale and is neutral along any other one scale it is overall neutral.

[0245] Step 7.

[0246] Paradoxical images

[0247] If image I is not neutral and (V(I)>0)&(C(I)<0)& (A(I)>0

[0248] or

[0249] (V(I)<0)&(C(I)>0)& (A(I)<0

[0250] then

[0251] I is considered to be a paradoxical one.

[0252] Step 8.

[0253] Cognitively preferred.

[0254] If image I is not neutral and not paradoxical I is cognitive preferred

[0255] ⇄

[0256] (V(I)>0)

[0257] with the score equals V(I)

[0258] Step 9

[0259] Cognitively not preferred

[0260] If image I is not neutral and not paradoxical I is cognitively not preferred

[0261] ⇄

[0262] (V(I)<0)

[0263] with the score equals V(I)

[0264] Each individual user has their own characteristics; preferences of images, music, coaching etc., in other words, each person has a unique personal profile. This profile is thought to allow us to be better able to select images or other stimuli for the user. To be able to select images requires sophisticated methods not only to record but to analyze and understand the trait and state data of a person. The personal profiler does exactly that.

[0265] 1. It gathers data from the portable biometric device (on going physiological data), biometric analyzer (Physiological data for different situations and images), and the cognitive analyzer (data on demographics, psychographics, cognitive preferences for images).

[0266] The preferred method is as follows:

[0267] Step 1. Selecting Activating/Deactivating Images:

[0268] a) Use BIOMETRIC analyzer method (e.g. CLMOD) to identify clearly activating or deactivating images. If the image is close to the baseline cluster then the image will be considered deactivating, or if the image is closer to the stressor cluster, then the image will be considered deactivating.

[0269] b) Only in situations, where we do not have enough images to fill the four categories 1-4, we will use CaIn/Exciting scale along with the BIOMETRIC analyzer method to pick activating/deactivating images.

[0270] c) Ranking rule: The images will be ranked based on following criteria a Ratio of distance of the image to stressor and baseline

[0271] Step 2. Dividing the Activating/Deactivating images into C+ and C− categories:

[0272] a) Use data from the scales in COGNITIVE ANALYZER to categorize images as preferred or not preferred. W e do not use Calm/Excited scale.

[0273] b) For PARADOXICAL images make decisions using rules specified in the COGNITIVE ANALYZER:

[0274] c) Ranking rule: The Euclidean distance on Unhappy/Happy and Detached/Attached scales will be used to rank the images in the C+ and C− categories. The top ranking images will be used if the total number of images in each category is more than 10.

[0275] Step 3. Augment images by reducing threshold if necessary

[0276] a) If the number of images in any of the four categories (picked based on Biometric analyzer method, and cognitive scales) is less than 4 then we will try to

[0277] Increase the number of images by lowering the threshold in BIOMETRIC model

[0278] Increase the number of images by lowering the threshold in the cognitive model to 0.55 SD (currently the threshold is 0.67 SD)

[0279] b) If the number of images in any category are more than 4 but less than 10, we will try to maximize the number of total images using upto 5 similar for each image. The minimum number of images in each session would be 15 and maximum will be 20.

[0280] c) If the number of images from any category are more than 10, we will pick the top 10. The ranking will be based on ranking rules specified in steps 1 and 2. The total number of images will be 30 and we will pick upto 2 similar for each image.

[0281] Step 4. Handle images that show very high physiology but neutral cognitive

[0282] a) We reduce the threshold of the two cognitive scales to to 0.55 SD to see if that puts the image in consideration into C+ or C−.

[0283] b) If not, we assign the image into both the C+ and C− category

[0284] Step 5. Handle images that show very high cognitive but neutral physiology

[0285] a) We reduce the threshold in BIOMETRIC ANALYZER model to see if that puts the image in consideration into activating or deactivating.

[0286] b) If not, we assign the image into both the activating and deactivating category

[0287] 2. Based on this data, it creates an individualized profile. The profiler uses generic models and population data to make predictions and personalization of coaching, stimuli, and even the user interface of the image presentation device.

[0288] 3. Using digital signal processing on the sensor data, it analyzes and makes predictions on the individual's state. Predictions will be made either using phasic physiological responses, such as change in heart rate or SCR, or using sophisticated techniques, such as individual component analysis or pattern recognition. For example, increased heart rate and SCR could indicate activation or excitement, however more sophisticated analysis could differentiate between startle and defense. According to Cacioppo et al (1996), although startle response and defense response are both associated with increased heart rate and increased SCR, they exhibit different patterns of activation. In the case of startle, the heart rate acceleration peaks and returns to near normal levels within two seconds, whereas in the case of defense response the heart acceleration does not begin to rise for several seconds and peaks much later.

[0289] 4. All the data is recorded in the profiler for future reference and use. The Personal Profiler also keeps records of data collected from subsequent biofeedback sessions.

[0290] 5. Using statistical methods, the profiler tries to understand what worked and what did not.

[0291] The Personal Profiler collects the data from the BIOMETRIC analyzer and COGNITIVE analyzer and classifies the images into

[0292] 1) Cognitively preferred/Physiological activating

[0293] 2) Cognitively preferred/Physiologically deactivating

[0294] 3) Cognitively not preferred/Physiological activating

[0295] 4) Cognitively not preferred/ Physiological deactivating

[0296] Images selected using cognitive analyzer and biometric analyzer are treated as a collection of images that describes an individual image profile. After classifying the master images into these four categories, the Personal Image Classifier, builds these image sets by picking images from the Therpaeutic Image Bank using similarity metrics method. Therpaeutic image bank uses the therapeutic image classification scheme to accurately mark each individual image with its inherent characteristics. The goal is to find images similar to each image in a profile to create sets of images that share similar characteristics with respect to individual's reactions. Therpeutic image bank may contain personal pictures as well as stock photographs. The ultimate goal is to be able to classify images automatically using this scheme. The procedure currently used is:

[0297] 1. All the images in the image bank are tagged with a 0 (for a particular feature not existent in the image) or 1 (for a particular feature not existent in the image), as shown in tables below. The colums represent the features from the classification scheme.

[0298] 2. To build a particular image cluster (say calming images), copy the classification record of all calming images from the master set for the particular individual into a buffer.

[0299] 3. Examine each image from this buffer and using similarity matrix techniques such as Minkowski method, find similar images from the image bank. A similarity metrics can be established as the sum of all agreements between image features established in the step 1 and weighted by the feature importance (for an individual). Thus the Minkowski metric with various exponents can be used to determine the similarity. We used the value of the exponent equal to 1. Weighting coefficients are determined experimentally using a screening or specifically designed testing procedure and are considered as the order of an individual's feature importance related to the therapeutic effect.

[0300] 4. Copy the new formed cluster into a new database for the individual with other metadata such as user identification, data and time of clustering, physiological reactivity to each image, cognitive reactions to each image.

[0301] This metadata will eventually be used in the personal profiler to evaluate the effectiveness of images in subsequent sessions.

[0302] This is the main component that the user works with images to relax, energize, or do biofeedback training. This could be implemented on a computer, TV with set top box, handheld device such as PDA's, CyberFrames, or gaming devices . The purpose of the Visualization System is to allow participants to maintain their mind-body wellness using proper personalized coaching based on trends in physiology and cognitive feeling. Uniqueness of Visualization System is:

[0303] Personalized image selection and training that understands the users trends in activation or deactivation.

[0304] Coaching that continues even after the session and allows one to do a retrospective analysis of physiology changes between sessions.

[0305] Intelligent image understanding and personal preference data allows the coach to guide the user to certain parts of image or to a totally different image as needed.

[0306] The overall mind-body wellness is achieved by presenting a series of stimuli (e.g. images) that are selected based on personal cognitive and physiological preferences in an order that is “natural” for the individual, along with personalized coaching and relevant feedback. The process includes the following steps:

[0307] (1) To initiate a session, the user docks the buddy into a docking station (if the image presenter is implemented on a TV or a desktop computer) or into a docking port if it is a handheld device.

[0308] (2) If this is the first session, the system needs some profiling data to understand what images are suitable for this individual. The profiling is done using a Master Image Presenter and Personal Profiler. The system records demographics and psychographics data for the user.

[0309] (3) A master set of images (A Kodak set designed based specifically for different cultures) are presented to the user and their cognitive feelings and physiological reactivity are recorded for each image.

[0310] (4) As described in the Personal Profiler (including biometric analyzer and cognitive analyzer), the cognitive preference is recorded using the three scales, whereas physiological reactivity is recorded for the most sensitive measures. The physiological sensitivity for each individual is recorded using different situations such as baseline, different stress activities, calming, and energizing activities.

[0311] (5) As described in the Personal Image Classifier, the cognitive and physiological feelings are combined using certain rules and used to categorize the master set images into preferred calming, preferred activating, and neutral images. The Personal Image Classifier builds a unique set of images for the individual, based on similar images selected from the therapeutic image classification scheme and the therapeutic image data bank. Each image is coded with metadata, such as the features of the image, its rank on physiological reactivity for the subject, its rank on the cognitive scaling, etc. This metadata is used eventually to decide when and how long this particular image will be presented. At this stage either cognitive, physiology or both can be used for categorization. Different product embodiments can have different implementations.

[0312] (6) In subsequent sessions, the Image Presentation Device uses the unique image set in presentation.

[0313] (7) Establish the identification of the participant before allowing access to the system either through password authentication or physiology measures signatures. Understand from the user what they would like to do today and try to assess the correlation between how they feel cognitively and what their physiology is suggesting.

[0314] (8) Provide general instructions on how to breathe as the user views different images. This coaching will be a combination of diaphragmatic breathing, autogenics, guided imagery, and meditation thoughts. The Visualization System incorporates appropriate coaching (male/female voice, autogenics/no autogenics, some mechanism of trust-building, diaphragmatic breathing etc), different types of feedback, personalized order of presentation, personalized schemes of fading, and appropriate timing.

[0315] (9) Feedback can be either direct feedback through either digital readouts of physiology and/or various graphical means such as abstract bars, trend charts, slider graphs, colored bar graphs, etc., or indirect feedback through changes in the image parameters such as hue, saturation, sharpness.

[0316] (10) The system will also provide continual reinforcement based on the trend and temporal changes in the user's physiology state.

[0317] (11) Through out the session, the system tracks the physiology trends on the sensors that are most sensitive to the user. The intelligent coaching agent has certain generic rules built in. It also has a learning system that understands and records the user's sensitivities to different physiology measures as well as their responsiveness, and according modifies the instructions. The coaching agent bases its instruction both on the physiological changes as well as the feelings that are recorded through cognitive scales.

[0318] (12) The user interacts with the coach through natural interactions such as speech, direct point and click, and physiology changes. The coaching agent has a “persona” that is customized for each individual. Different persona of the coach could be varied on the gender, ages, instruction styles, mannerisms, personality types that a particular user likes. Certain amount of anthropomorphism is also provided in the coaching agent to facilitate one-to-one connection between the coach and the user.

[0319] (13) The coach also has intelligent image understanding and provides certain cues on contents of the images. These cues are stressed if the coach has prior knowledge about the user's preference.

[0320] (14) Apart from the individually selected mix of images, the Visualization System also provides individual image categories (sunset, beaches, rain, landscapes, family, children etc).

[0321] (15) It also provides both individualized and generic transforming images. Transforming images can include images that transform existing content such as an image showing sunset, or a flower blooming as well as adding new content e.g. a waterfall scene with a rainbow added to the scene if the user achieves a certain stage in the calming process.

[0322] (16) Throughout the session the Personal Profiler records the efficiency of the images. The profiler keeps record of what worked and what did not. (This is thought at the current moment to be available in the advanced implementation).

[0323] (17) The influence of the Visualization System on the user's behavior does not end at the end of the session. At the end of the session, the coaching system records how the user feels and will tell the user that they should carry the feelings and learning from this session to the real world. The user physiology will be monitored by the portable biosensor device between the sessions. The coach can then query, understand and advise the user based on the physiology data that is collected between sessions.

[0324] The invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention.

[0325] 10 process

[0326] 12 60 process steps

[0327] 70 wrist type device

[0328] 72 sensors

[0329] 74 computer

[0330] 80 head band type device

[0331] 82 sensors

[0332] 84 computer

[0333] 90 earphone type device

[0334] 92 sensors

[0335] 94 computer

[0336] 100 system

[0337] 102 prtable biosensor device

[0338] 104 master set of images

[0339] 106 therapeutic image classification system

[0340] 108 biometric analyzer

[0341] 110 cognitive analyzer

[0342] 112 personal image profiler

[0343] 114 personal image classifier

[0344] 116 visualization system