System for correlating in a display stimuli and a test subject's response to the stimuli

[0002] 1. Field of the Invention

[0003] The present invention is directed to a system which subjects a test subject to certain stimuli, measures a reaction of the test subject to the stimuli, and subsequently overlappingly records and/or displays data of both the stimuli and the test subject's reaction to the stimuli in synchronization. The recorded data can also then be displayed or replayed to evaluate the test subject's reaction to the stimuli.

[0004] 2. Discussion of the Background

[0005] It is often desirable to monitor how a person reacts to certain stimuli, such as an audio presentation, a video presentation, etc. The different ways of monitoring how a patient reacts to such stimuli may be by taking an EKG of the patient, an EEG of the patient, heart rate, etc. As one concrete example, it is often desirable to take EEG measurements of a test subject to determine the brain activity of the test subject. There are many medical applications that require testing and measuring of EEG signals to determine a test subject's medical condition. There are also instances in which it is desirable to measure a test subject's brain activity for other purposes.

[0006] One such purpose is to monitor a test subject's brain activity to measure the test subject's reaction or interest to certain stimuli. For example, it may be desirable to be able to measure a test subject's attention level to a particular marketing idea, television commercial, presentation, etc.

[0007] One problem with measuring a subject's response to certain stimuli and effectively using that data to provide a meaningful graphical representation of the subject's responses to the certain stimuli occurs when the test giver attempts to accurately correlate the subject's response to the stimuli.

[0008] That is, after, as one example, an EEG measurement is taken of a test subject, this data must be evaluated relative to the stimuli to which the test subject was subjected. This is often a very difficult and time consuming operation since it is often very difficult to correspond the test subject's reaction as indicated in the EEG data obtained from monitoring the test subject, with the stimuli to which the test subject was subjected.

[0009] The Applicant's of the present invention have recognized that in the above-discussed type of analysis, it would be beneficial to synchronize the test subject's response to stimuli along with the stimuli in an integrated overlapping display and/or recording that would enable a person reviewing the test data to observe on a single display both the graphical representation of the test subject's response and the stimuli to which the test subject is responding.

[0010] It may also be desirable to integrate into the displayed results other data relating to the test conditions. This data could include, but is not limited to, test parameters such as time and date of the test, demographic data relating to the test subject, and other control data related to the test.

[0011] The Applicant's of the present invention have thus further recognized that there is a need for a system and method to accurately synchronize on a single display and/or recording a graphical representation of a test subject's response to stimuli, a display of the stimuli to which the subject is responding, and other data relating to test parameters.

[0012] Therefore, it is an object of this invention to provide a novel system and method to accurately overlap and synchronize on a single display and/or recording a graphical representation of a test subject's response to stimuli and the stimuli to which the subject is responding. The single display and/or recording may also include, if desired, other data relating to test parameters.

[0013] It is another object of this invention to provide a novel system and method to easily and efficiently allow for the display and/or recording with minimum input needed from an operator.

[0014] The present invention is directed to a system which can subject a test subject to certain stimuli, detect the test subject's response to the stimuli, and then display and/or record in synchronization the stimuli to which the test subject was subjected, and the test subject's detected response to that stimuli. The recorded data can then also be replayed, displayed, etc.

[0015] A further object of the present invention is to record as a data file the overlapped and synchronized graphical representation of a test subject's response to stimuli and the stimuli to which the test subject is responding. This recorded data can then be synthesized and processed with other data as desired.

[0016] A more complete appreciation of the present invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

[0017] FIG. 1 illustrates a preferred embodiment of the present invention showing hardware components of the system of the present invention;

[0018] FIG. 2 illustrates a flow chart showing processing of signals to obtain a synchronized and overlapping output display; and

[0019] FIG. 3 illustrates a preferred embodiment of on output display of the present invention.

[0020] FIG. 1 illustrates the components of a preferred embodiment of the system of the present invention used to provide precisely synchronized and overlapped output test data which represents a test subject's response to certain stimuli, the certain stimuli, and if desired other control data and parameters, and FIG. 2 shows the control steps executed in the system.

[0021] The present invention is directed to a system which can subject a test subject to certain stimuli, detect the test subject's response to the stimuli, and then display and/or overlappingly record in synchronization the stimuli to which the test subject was subjected, and the test subject's detected response to that stimuli. This recorded data can then also be replayed, further processed, displayed, etc.

[0022] The Applicants of the present invention anticipate widespread uses for such a system as in the present invention. One envisioned use of the system of the present invention is to measure a person's level of engagement to a video presentation, such as an advertisement, a speech, a sales pitch, etc. In these envisioned uses of the present invention, a test subject is shown a video presentation, and as the test subject is viewing the video presentation, EEG data of the test subject is detected. Then, the system of the present invention records and/or displays the video presentation which the test subject is viewing along with the detected EEG data in synchronization, for example on a video tape.

[0023] This operation in the present invention provides a video tape which shows both a video presentation which a test subject viewed along with the test subject's EEG response to that video presentation. The result of this is that someone evaluating the video presentation can easily and efficiently know which points of the video presentation elicited certain engagement activity from the test subject. For example, a person evaluating the video data can evaluate precisely at which portions of the video presentation the test subject was most engaged and at which portions of the video presentation the test subject was least engaged. This data is easily and quickly discemable to someone viewing the video data because the video presentation is displayed simultaneously and in synchronization with the test subject's EEG data.

[0024] The example noted above provides an example of a video presentation and measuring a test subject's level of engagement, i.e., generating EEG data. The present invention can also be applied to other stimuli, such as an audio presentation, in which case the recorded data will synchronize the audio presentation with the EEG data, which can still be displayed. Other stimuli and other measures of a test subject's response to the stimuli, such as EKG data, heart rate data, blood pressure, body temperature, galvanic skin response, pupil dilation, etc., can also clearly be instituted.

[0025] The following described preferred embodiment of the present invention is directed to the example noted above of a test subject viewing a video presentation and detecting EEG data from the test subject, although again as noted above other stimuli and other responses are clearly possible.

[0026] In the preferred embodiment of the present invention as noted above, EEG data of a test subject must be obtained. This can be done by monitoring a test subject in any known way. The Applicants of the present invention have devised a novel way of easily and efficiently monitoring a test subject to detect EEG data, which is disclosed in copending U.S. application Ser. No. 09/187,525, the entire contents of which are hereby incorporated by reference. The present invention is not limited to such a system of monitoring a subject to obtain EEG data, and the present invention can be applied to any system which monitors a subject to obtain such data or similar desired data.

[0027] The present invention will now be described in greater detail by reference to FIGS. 1 and 2. FIG. 1 is a hardware diagram of the present invention, and FIG. 2 is a chart indicating control processes executed by the hardware of FIG. 1. In the following discussion of the operation of the present invention, reference is made to both FIGS. 1 and 2.

[0028] An overview of the operation of the system as shown in FIGS. 1 and 2 will now be given prior to describing the specific controls executed. In the system as shown in FIGS. 1 and 2, a test subject views video data on a TV monitor 11 generated from a source VCR 9. A monitor 1 detects the test subjects EEG response to viewing the video data on the TV monitor 11. The EEG data monitored from the test subject is then fed to a computer 7. In synchronization with the computer 7 receiving the EEG data from the test subject, the computer 7 also receives the video data that the test subject is viewing from the source VCR 9. That is, the source VCR 9 outputs the video data both to the TV monitor 11 and to the computer 7. The computer 7 then combines the source video data which the test subject is viewing with the EEG data generated while the test subject views the video data. The video data and the test subject's EEG response to the video data is then overlapped in synchronization and provided to a destination VCR 14 to be recorded thereon.

[0029] The end result of this process of the present invention is that a recording is made at the destination VCR 14 in which the video tape includes video data which a test subject viewed, and a synchronized display of the test subject's EEG response to that video data. Someone then evaluating the combined video data can see precisely how a test subject's EEG response varied with the video data. Someone evaluating that video tape can easily detect at exactly what scenes of the video data the test subject was most engaged, less engaged, etc.

[0030] Now the operation of the present invention is described in further detail with reference to FIGS. 1 and 2.

[0031] In the present invention, and with reference to FIG. 1, a monitor 1 is placed on a test subject (not shown) in order to measure the EEG activity of the test subject (see also step S5 in FIG. 2). The monitor 1 can be the headset of Applicants' copending U.S. application Ser. No. 09/187,525, but is not limited to such. The test subject will be viewing video data on the TV monitor 11, and the monitor 1 provides a raw EEG signal based on the test subject's response to viewing the video data on the TV monitor 11. The raw EEG signal data obtained from the monitor 1 is then input through signal line 2 to an analog signal processor 3 that processes the raw EEG signal into a form that can be further processed into a meaningful representation of the subject's responses to certain stimuli (see also step S10 in FIG. 2). The analog signal processor 3 can perform processings such as amplification, filtering, etc., as needed. An analog EEG signal is output from the analog signal processor 3 signal to an analog to digital (A/D) converter 5 on a signal line 4. The analog to digital converter 5 converts the analog EEG signal into a digital form that can be used to provide a graphical representation of the test subject's responses to viewing the video data on the TV monitor 11 (see step S15 in FIG. 2).

[0032] The A/D converter 5 has a bidirectional signal line 6 connecting the A/D converter 5 with a computer 7. This bidirectional signal line 6 allows the computer 7 to accurately synchronize the collection of test data and the activation of the test stimuli as discussed below.

[0033] The accurate synchronization of the test stimuli and collection of data is enabled by the inclusion of a switch SW8 located within the A/D converter 5 and a switch SW23 in a source audio-video device 9. When the test operator (not shown) commences the test, the operator, by, e.g., depressing a button or clicking on a mouse, signals the beginning of the test. If the operator had to then manually activate a source audio-video device 9, such as a video cassette recorder that acts as the source of the stimuli, i.e. the video data, for the subject, there could be a discrepancy in the synchronization of the data collected and the stimuli to which the test subject is responding. The present invention eliminates this possibility because the switch SW8 in the A/D converter 5 receives a signal over bidirectional signal line 6 from the computer 7 (see step S50 in FIG. 2). Activation of the switch SW8 then further results in sending a signal from A/D converter 5 over signal line 10 to the switch SW23 of the source audio-video device 9 to activate the source audio-video device 9 and commence the presentation of the video data, i.e., the stimuli, to the test subject (see step S20 in FIG. 2).

[0034] At the same instant in time, the computer 7 begins collecting digital EEG data from A/D converter 5 over bidirectional signal line 6. Thus, the test operator with one simple motion simultaneously activates the collection of EEG data and activates the display of stimuli from the audio-video source 9. A precisely synchronized method of providing stimuli and collecting test subject data is thus achieved.

[0035] The source audio-video device 9 can be a conventional video cassette recorder (VCR) with one significant modification. The source audio-video device 9 includes the additional switch SW23 which receives a signal from the A/D converter 5. This switch SW23, as noted above, activates the source audio-video device 9 based on a signal received from A/D converter 5 to send the audio-video signal to the TV monitor 11.

[0036] That is, upon activation of the audio-video source 9 through switch SW23, the output of the audio-video source 9 is sent to the display monitor 11 over signal line 12; that output is the video data that is viewed by the test subject being monitored by monitor 1 (see step S30 in FIG. 2). The test subject observes the display monitor 11 and passively responds to the stimuli to provide the raw analog EEG signal that is further processed as noted above.

[0037] While the audio-video source 9 outputs a signal to the display monitor 11 over signal line 12, the audio-video source 9 simultaneously sends the video component of the output signal to the computer 7 over signal line 13 (see step S25 in FIG. 2). Also simultaneously, the audio-video source 9 outputs the audio component of the output signal to a destination ANV recording device 14, which also can be a standard video cassette recorder (VCR), over signal line 15 (see step S40 in FIG. 2). The audio component of the output signal can also be fed through the computer 7, if desired.

[0038] The computer 7 is now simultaneously receiving digital EEG data from A/D converter 5 from bidirectional signal line 6 (see step S45 in FIG. 2) and the video component of the A/V source 9 signal from signal line 13 (see step S55 in FIG. 2). The computer 7 is equipped with appropriate software that processes both signals simultaneously. The computer processes the digital EEG signal from the A/D converter 5 into a graphical representation of the test subject's responses to the source stimuli. This graphical representation of the test subject's responses can take various forms but a preferred form may be a graph showing peaks (higher brain activity levels) and valleys (lower brain activity levels) that represent the test subject's interest to the stimuli representing specific brain activity as a function of time. The computer 7 is also simultaneously processing the video component of the A/V source 9 signal from signal line 13. In addition to processing these signals, the computer 7 is also processing data relating to test parameters or control data such as test time and date, demographic data of the test subject, and other relevant control data.

[0039] The computer 7 then outputs a video signal that combines the three separate data sources (i.e., the graphical representation of the digital EEG data, the video component of the A/V source 9 signal, and the test parameter and control data) into a composite VGA signal over signal line 16 (see step S60 in FIG. 2). This composite signal is sent over signal line 16 to a VGA/video converter 17 that processes the composite signal and produces two output signals. This VGA/video converter 17 can also be an integral part of the computer 7. One output signal from VGA/video converter 17 is a VGA signal that is sent to the computer monitor 19 over signal line 18 (see step S65 in FIG. 2). The computer monitor 19 can then display, e.g. in three separate windows, 1) the graphical representation of the digital EEG signal, 2) the video component of the A/V source 9, and 3) test parameter and control data. The VGA/video converter 17 simultaneously sends a video signal having the same three window displays over signal line 20 to the destination A/V recording device 14 (see step S70 in FIG. 2). This computer monitor 19 is provided for a test giver to monitor the testing.

[0040] The destination A/V recording device 14 is now simultaneously receiving the composite video signal from VGA/video converter 17 over signal line 20 and the audio signal from A/V source 9 over signal line 15. The destination A/V recording device 14 simultaneously records both signals to produce an overlapping and synchronized recording (e.g. on a video cassette tape or other recorded media) that captures the processed graphical representation of the digital EEG signal, the A/V source stimuli, and test parameter and control data, on a single synchronized medium. This single synchronized representation of the three video signals and one audio signal can be recorded and viewed at a later date. If desired, the destination A/V recording device 14 can record the signals and simultaneously output the composite signal to a destination monitor 21 over signal line 22 (see step S75 in FIG. 2). The destination monitor 21 may be provided so that another person can view the testing, and the destination monitor 21 may be provided at a location remote from the test site.

[0041] Thus a system and method to accurately synchronize on a single display a graphical representation of a subject's response to stimuli, a display of the stimuli to which the subject is responding, and other data relating to test parameters, is achieved.

[0042] FIG. 2 is the flow chart essentially reiterating the paths that the various signals follow in order to produce a synchronized representation of the test data. The flow chart also shows the monitor 1 that obtains raw EEG data from the test subject (step S5). The raw EEG data is sent via signal line 2 to the analog processor 3 that sends the processed signal (step S10) via signal line 4 to the A/D converter 5 to convert the signal to a digital EEG signal (step S15). The bidirectional signal line 6 connects the computer 7 to the A/D converter 5. The computer 7 sends an activation signal to the switch SW8 in the A/D converter 5 (step S50). The activation of the switch SW8 by the computer 7 results in sending a signal from the switch SW8 to the switch SW23 of the A/V source 9 via signal line 10 to begin presentation of the stimuli to the test subject (step S20). The A/D converter 5 is simultaneously sending digital EEG data obtained from the test subject to the computer 7 via bidirectional signal line 6. Thus, the presentation of stimuli to the test subject and the collection of EEG data by the computer 7 are automatically synchronized.

[0043] Now that the A/V source 9 has been activated, it sends an A/V signal to the display monitor 11 via signal line 12 so that the test subject can view the A/V stimuli (step S30). At the same time, the A/V source 9 sends a video signal via signal line 13 to the computer 7 for further processing (step S25). Also at the same time, the A/V source 9 sends an audio signal to the destination A/V recording device 14 via signal line 15 (step S40).

[0044] At this time, the computer 7 is simultaneously receiving a video signal from the A/V source 9 (step S55) and digital EEG data from the A/D converter 5 (step S45). The computer 7 processes the digital EEG data, the video signal from the A/V source 9, and other test parameter and control data stored in the computer to produce a composite video signal having, e.g., three windows to display the various aspects of the composite video signal (step S60). The computer 7 sends this composite video signal via signal line 16 to the VGA/video converter 17.

[0045] The VGA/video converter processes the signal and sends a VGA signal via signal line 18 to computer monitor 19 so that the test operator can monitor the operation of the test (step S65). At the same time, the VGA/video converter 17 is sending a composite video signal via signal line 20 to the destination A/V recording device 14 (step S70). The destination A/V recording device 14 is now simultaneously receiving the audio signal from the ANV source 9 via signal line 15 and the composite video signal from the VGA/video converter via signal line 20 (step S75). The destination A/V recording device 14 can now record a synchronized composite audio-visual representation of the graphical representation of the processed EEG data, the stimuli observed by the subject, and test parameter and control data. If desired, the synchronized composite A/V representation of this data can be recorded and viewed at a later time or can be recorded and output to a destination monitor 21 via signal line 22.

[0046] Thus, a real time synchronized an overlapped audio-visual representation of the graphical representation of the processed EEG data, the stimuli observed by the test subject, and test parameter and control data is achieved.

[0047] FIG. 3 illustrates a preferred embodiment of an output display 30 of the present invention. The output display 30 can be observed on the computer monitor 19 or the destination monitor 21. The visual display on both monitors 19, 21 is the same. The display can include three windows 31-33. Window 33 is an area where the graphical representation of the processed EEG data obtained from the computer 7 is displayed. Window 32 is an area where the A/V source 9 stimuli signal, i.e. the video data, that the test subject observed is displayed. Window 31 is the area where the test parameter and control data from the computer 7 is displayed. The integration of windows 31-33 into a single, synchronized display provides an efficient and easily reviewable presentation so that a person reviewing the results of a test can readily ascertain the test subject's responses to various stimuli.

[0048] FIG. 3 only shows one example of a format that an output display can take. The display can clearly take other formats such as, but not being limited to, deleting the output and controls window 31, spreading out the contents of the output and controls window 31, changing the positioning and size of the source video window 32 and the EEG data chart window 33, adding a video feed of the test subject, as the test subject is monitored, in the output and controls window 31, etc.

[0049] Moreover, a result of utilizing the system of the present invention is that a final product is a composite data recording file which synchronizes stimuli which a test subject was subjected, the test subject's response to that stimuli, and any other necessary control data. The result is that a resulting composite data file is generated which may be utilized in many other types of useful processings. For example, someone evaluating test results can combine plural of such composite data files from many test subjects into one larger composite file, can compare such files with other data, etc. Thus, one further benefit of the present invention is providing a single file composite data with different information which can be further processed as desired.

[0050] Further, it is noted that the computer 7 includes computer codes which execute the controls as discussed above. The computer codes can be written in any language well known to those of ordinary skill in the art.

[0051] The present invention as discussed above has also focused on the example of monitoring a test subject to detect their EEG response in relation to the test subject viewing video data. The present invention can clearly be applied to a test subject being subjected to different stimuli, such as an audio presentation, or any other sensory stimuli. Also, the information monitored from the test subject is clearly not limited to EEG data but can include many different kinds of data such as EKG data, heart rate, blood pressure, body temperature, galvanic skin response, pupil dilation, etc. In certain of these instances, the present invention can also find many types of medicinal or therapeutic uses.

[0052] The present invention has also provided an example where video data and EEG data generated in response to a test subject viewing that video data are both recorded on a video tape in a destination VCR 14. Clearly the present invention can record any type of stimuli and the test subject response to the stimuli in any manner, including recording such data on an optical disk, a magnetic disk, or any other memory system.

[0053] A system and method has been shown to effectively and accurately synchronize on a single display a graphical representation of a subject's response to stimuli, a display of the stimuli to which the subject is responding and other data relating to test parameters. Furthermore, the system and method eliminate the potential for human error in obtaining the test data.

[0054] Obviously, numerous additional modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the present invention may be practiced otherwise than as specifically described herein.