Apparatus for Measuring Eye Gaze and Fixation Duration, and Method Therefor

(i) Technical Field of Invention

The invention is a form of oculometer and as such can be used to measure eye gaze and fixation duration as well as the dual eye binocular convergence point, and as such has potential applications in the medical, scientific, engineering, manufacturing, military, and entertainment fields. The apparatus may be used as a tool for medical diagnosis of ocular functions, as an aid to the paraplegic or handicapped, for the measurement of ocular functions and workload in human factors studies, as a measure of subject training, as a tool for fatigue monitoring, as part of an electronic safety net to detect performance degradation due to pilot incapacitation in piloted and teleoperated vehicles, as a component of an electronic intelligent pilot-vehicle interface used for situation awareness aiding in piloted and teleoperated vehicles, for task scan analysis including measuring situation awareness, for human operator control of machines and computer games, and for advertisement and usability analysis.

The invention is used with head-mounted video displays such as those developed for virtual reality, stereographic displays, monocular or binocular vision helmet mounted displays, and night vision goggles used in piloted helicopters, vehicles, and teleoperated robotics control stations. The invention can be used as an eyetracker to control computerized machines from an electronic video display by the ocular gaze point of regard and fixation duration.

Examples of machine control by ocular functions include updating computer generated information displays, selecting panel switches and instruments, controlling the fidelity of computer generated imagery scene inserts in simulations, controlling the viewing direction of remotely located cameras, controlling the movement of teleoperated robotics platforms or vehicles, selecting display subareas for automated scene analysis in aided target recognition, designating targets from direct sight or from a sensor display, and weapon system pointing.

The invention has particular applications to time shared concurrent tasks where the hands are involved in a continual time critical pilotage task and the eyes may be used intermittently to control a discrete task. The use of this invention enables both tasks to share a common visual working area with overlaid visual images. In this way, task interference is reduced by dedicating eye-movements and visual attention to the same working surface.

An example of such an application would be single pilot nap-of-earth low-level helicopter flight while updating onboard heads-up displays. A similar application is teleoperations of remote vehicles from video displays with camera control. Another such application is to the operation of completely enclosed armored vehicles with "transparent" or "see through" armor where the operator sees a video projection of the outside scene as recorded by externally mounted cameras and relayed to internal monitors; the operator would use the invention to control displays overlaid on the scene projection while concurrently performing the vehicle pilotage task. Similar comments apply to the piloting of "glass cockpit" designs for completely enclosed high performance aircraft.

(ii) Description of the Prior Art

The existing technology for oculometers is based upon the optical measurement of reflected infrared light from the human eye, and is more than thirty years old in concept. In its present form, an oculometer contains a single infrared light source which is directed at the eye and the reflected light is imaged onto a charge-injection (CID) or charge-coupled device (CCD) sensor array. The image of the eye is then electronically processed to determine the corneal reflection, the pupil centroid orientation, or both. These parameters are used to determine the angular location of the eye relative to the camera within a fair degree of accuracy. The technology is either head-mounted or mounted in a panel in front of the user.

The head mounted systems are awkward to wear. The accompanying optics are bulky and heavy, limited in field of view, and induce neck and shoulder muscle fatigue. The infrared light source is placed next to the eye; a filter shields the source, eye, and sensor from the ambient light. The arrangement limits and interfere with the visual field. The boresight established in calibration, is easily perturbed by head movement or vibration induced disturbances in the head support system. In addition, adjustment difficulties may occur with individuals having a large nose bridge, deep lying eyes, bulging eyes, jutting eye lids, or other extreme deviations in facial structure.

These limitations become apparent when integrating the present technology with helmet mounted displays. Recent advances in display technology have produced extremely light weight helmet mounted displays comparable in size to bulky sunglasses. The eye relief distance is specified in millimeters and there is little room between the display and the eye for placing the infrared sources and fiber optics probes used with the present technology.

The optics for the panel mounted system are mounted in front of the user and directed toward his face. The panel mounted system is limited to low ambient light levels and objects that the user may need to work with can not be placed between his face and the optics. A servomechanism is used to keep the optics aligned on the user's eye, and the servomechanism adjustment is noticeable to users following head movement. Excessive head movements and interference of the optical path by facial features such as the user's nose, are not tolerated.

The oculometer determines the angular location of the eye relative to the sensor array within a fair degree of accuracy. The measurement of head position and orientation for the head-mounted system allows the determination of eye position and orientation in the workspace, and therefore the computation of eye point of regard. Similarly, the determination of the range from the sensor to the eye by either ultrasonic or image focusing allows the computation of eye point of regard for the panel system. The accuracy of the technology is roughly about +/- one degree in practice; a lower limit appears to be about +/- 0.75 degree.

The oculometer is used to determine the location of the user's visual attention; it can be used to select display icons in machine control. The user may be looking over a display or scene to acquire task related information in preparation to initiating machine control. Some other means is usually required for the subject to indicate that a particular display element or scene feature has been selected by him for machine action. For example, the user may perform a motor function in conjunction with a forced gaze at a selected item, by pressing a switch or speaking a voice command to an automatic speech recognizer.

Exemplary of existing technology include U.S. Pat. No. 3,986,030 to Teltscher which discloses eye-motion operable keyboard-accessory includes light source means for directing a light ray at the eye of an operator, for obtaining a light reflection there from, and a plurality of light-responsive sensors selectively actuable by the operator reflected light ray. U.S. Pat. No. 4,109,145 to Graf discloses an apparatus that is controlled by movement of the eye which using a line of sight detecting apparatus such as an oculometer produces signals indicative of the direction along which an operator is looking and positions a display for viewing by the operator. U.S. Pat. No. 4,209,255 to Heynau et al. disclose single source aiming point locator using a single source light emitting diode (LED) mounted on the helmet of a pilot in an aircraft, which can be used to designate a point on a display from photodiodes located adjacent to the display. U.S. Pat. No. 4,623,230 to Weinblatt discloses media survey apparatus and method using thermal imagery to monitor visual observation of a transitionally viewed display by directing infrared radiation from behind the display into the path of travel of a moving observer and detecting the retinal eye reflections from the observer when he is viewing the display. U.S. Pat. No. 4,648,052 to Friedman et al. discloses an eye-tracker communication system based on a system for computer vision comprising a source of digitized video signals and a frame encoder circuit which encodes threshold crossing events in eye fixations on a two dimensional area within the frame and stores the results in a cache memory. U.S. Pat. No. 4,761,071 to Baron discloses an apparatus and method for determining the corneal and scleral topography by a plurality of triangulations each made to determine the elevation of a different location on the surface of the eye from a positive sensitive detector of known orientation and an incident narrow beam of light with known position and direction, which illuminates a fluorescent substance instilled on the surface of the eye. U.S. Pat. No. 4,973,149 to Hutchinson discloses an eye movement detector utilizing an infrared light emitting diode mounted coaxially in front of the lens of an infrared sensitive video camera for remotely making images of the eye of a computer operator. U.S. Pat. No. 5,035,500 to Rorabaugh et al. disclose an automated ocular perimetry, particularly kinetic perimetry, which using a single visual light source moved with a x-y plotter under computer control to discrete positions within the visual field of a subject, has the subject indicate detection of other light sources positioned in a regular array and momentarily illuminated at various times, while the subject is fixating the first light source. U.S. Pat. No. 5,170,153 to Migozzi et al. disclose an optical device for the display of light data collimated to infinity which uses an oculometer to determine the pupil center of the eye and an automatic control circuit to adjust the position of the display image so that it is centered over the pupil. U.S. Pat. No. 5,220,361 to Lehmer et al. disclose gaze tracking for field analyzer through automated video surveillance of the patient's eye from an infrared source mounted in the center of the test screen to the front of the patient, and the video measurement of corneal reflected light and the computation of the pupil chord to determine the gaze direction. Adams, C. (1990). "If looks could kill: the eyes have it". Military & Aerospace Electronics. (Mar) 35-37. Reviews infrared light source oculometers and the potential for use by pilots of military aircraft in designating targets by eye gaze. Anliker, J. (1976). Eye movements: on-line measurements, analysis, and control. In R. A. Monty & J. W. Senders (Eds.), Eye Movements and Psychological Processes. (185-199), Hillsdale, N.J.: Lawrence Erlbaum Associates. Describes automatic fixation and saccade detector classification schemes. Calhoun, G., Janson, W., & Arbak, C. (1986). "Use of eye control to select switches". Proceedings of the Human Factors Society 30th Annual Meeting, Dayton, Ohio. Describes the experimental results and limitations of pilots using a head mounted infrared light source oculometer for selection of panel switches in a fighter aircraft simulator. Goode, P. W. (1984). "Eye Directed View". 21st Space Conference, Kennedy Space Center. Describes experimental results of using an infrared light source oculometer for control of remote camera viewing direction. Jacob, R. J. K. (1989). "What you see is what you get: the use of eye movements in human-computer interaction techniques". Washington, D.C.: Naval Research Laboratory. Describes techniques and limitations of using a panel mounted infrared light source oculometer for control of computer driven displays by eye fixation and gaze duration. Smyth, C. C. & Dominessy, M. E. (1989). Comparison of oculometer and head-fixed reticle with voice or switch and touch panel for data entry on a generic tactical air combat display (Technical Memorandum 21-89). Aberdeen Proving Ground, Md.: U.S. Army Human Engineering Laboratory. Describes the experimental results and limitations of using a head mounted infrared light source oculometer for control of computer driven displays by eye fixation and switch or voice commands.

It is an objective of the invention to provide an apparatus that can be used as an eyetracker to control computerized machinery by ocular gaze point of regard and fixation duration, this parameter may be used to pre-select a display element causing it to be illuminated as feedback to the user. The user confirms the selection with a consent motor response or waits for the selection to time out. The ocular fixation dwell time tends to be longer for display element of interest.

It is an objective of the invention to overcome disadvantages of existing technological by incorporating oculometers into head mounted electronic video displays at a lower cost, increased accuracy, rapid response, slight bulk, light weight, relatively free field of view, and increased ease of application and calibration.

It is yet another objective of this invention to provide an apparatus that is auto-calibrating thus allowing the user to quickly and accurately go through a calibration procedure that correlates visual fixation position with line of sight.

It is yet another objective of this invention to provide an apparatus whose accuracy is independent of shifts of the helmet on the user's head; theses shifts are caused by changes in facial expression and head movement dynamics. This is due to the ability to compute an exact eye model from the locations of the light sources and the sensors which are fixed by the helmet-construction. Measurements of the helmet location and orientation are used to relate the visual axes to the user's three dimensional workspace.

It is another objective of the invention to provide an apparatus that uses an electronic video display array as an accurate and rapid sequence of point light sources. This reduces the components needed for eye-tracking since the light imaging system is used to illuminate the eye in a controlled manner.

It is yet another objective of the invention to utilize a video imaging system that may be active emitter (CRT, thin-film electroluminescence phosphor, plasma panel) or passive (Liquid crystal) including active matrix. In all of these cases, the oculometer makes use of the initial flare of illumination common all of these sources either due to emission or crystal realignment, that occurs as each display pixel is refreshed in an orderly sequence. In a similar manner, the apparatus of the invention may instead use an array of near-infrared light emitting diodes turned on and off in sequence as an illuminating source.

It is yet another objective of the invention to provide an apparatus that employs a realistic model of the eye, derived from physiologic optics, to accurately determine the optical axes and the corresponding visual line of sight. The computations needed for this derivation are based on the measurements of the multiple sequence of light reflections from the cornea and internal components of the eye.

It is yet another objective of the invention to replace the sensor image focusing and processing used in prior technology with a simple array of photo-transistor light sensors and amplifiers directed toward the cornea of the eye. The objective further includes using this opto-transistor array, a comparator array and an encoder and latch clocked by the raster-scan pulses of the display driver, to construct a pairing table of sequential source corneal reflections to sensor activations over the display field refresh cycle. The pairing table listings of reflections is used to compute an accurate three dimensional ocular model which for each display field refresh cycle, locates the corneal center and optical axis as well as the corneal orientation from the major and minor axes. The visual origin and axis is then computed from these parameters.

It is yet another objective of the invention to easily integrate the apparatus with a miniaturized CID/CCD video camera coupled with image processing to acquire an eye image. Ophthalmometric techniques are used to locate the pupil center as an additional parameter for the computation of the optical axis. In addition, the pupil diameter can be measured for workload measurements.

It is yet another objective of the invention to provide an apparatus that can easily extend to simultaneously track both eyes, thereby allowing measurement of the optical convergence point in the user's three dimensional workspace (real or virtual).

It is yet another objective of the invention to provide an apparatus and method that is particularly attractive for measuring ocular functions with virtual image displays such as helmet mounted displays and night vision goggles. However, the invention is readily incorporated into panel mounted displays and head-free sensor arrays.

FIG. 1 is a schematic drawing of the invention showing the hardware components.

FIG. 2 is a detailed drawing of the enhanced schematic showing the electronic hardware and software components.

FIG. 3 is a schematic drawing of the invention showing the processing of the light reflected from the user's eye.

FIG. 4 is a drawing showing headset design.

FIGS. 5a-c are drawings showing various configurations for the headset mounted optical device of the invention.

FIG. 6 is a flowchart for the digital computer routines.

FIGS. 7a-d are flowcharts for the computer routine used to compute visual line of sight.

FIG. 8 is a schematic showing a method for using the invention as a computerized machine controller.

The invention consists of a headset 1, a video device 2 mounted on the headset, an opto-electronic device 3, a digital processor 4, an image processor 5, a headset position and orientation sensor 6 and a digital computer 7. Video device 2 holds a video display 2a and an sensor system 2b. Video display 2a illuminates the user's eye while the sensor system 2b detects the reflections from the eye. The optical output of the sensor system 2b is the input to the opto-amplifier array 3a of opto-electronic device 3. The electrical output of the array 3a is the input to comparator array 3b which in turn provides electrical input to encoder and latch 3c. The electrical output of encoder and latch 3c is the electrical input to digital processor 4. Comparator array 3b isolates amplifier element 3a responding to the instantaneously illuminated element of video display 2a, and processor 4 writes the encoded address from latch 3c to a double buffered digital memory shared with digital computer 7, on the clocked raster-scan pulse 2c from the video display 2a. The electrical output of the array 3a is also input to the charge-coupled device (hereinafter referred to as CCD) 5a of image processor 5. The user's pupil centroid is isolated by processor 5b on clocked display-field refresh pulse 2d from video display 2a, and the electrical output is digital input to the digital computer 7. The electrical output of the headset position and orientation sensor 6 is digital input to the computer. The digital computer runs several processes which are released from hibernation upon the clocked display-field refresh cycle pulse 2d from the video display 2a; these are routines to compute: the visual line of sight 8, the workspace line of sight 9, fixations 10, gaze points 11 and finally the binocular convergence point 12 for the dual eye system.

The invention uses electronic video display 2a to illuminate the eye in a controlled manner. The sequence of momentary flares in display pixel brightness, generated during the electronic refresh sweep of display 2a, provides an accurate and rapid sequence of point light sources. This use of video imaging system reduces the components needed for eye-tracking. The video imaging system 2 may be active emitter such as a cathode ray tube, solid phosphor thin-film electroluminescence, gas discharge plasma panel or passive emitter such as a liquid crystal display including active matrix. The invention makes use of the initial flare of illumination common to all of these sources due either to emission or crystal realignment, that occurs as each display pixel is refreshed in an orderly sequence. Furthermore, the invention may be extended to light emitting diode displays where the display elements are re-drawn sequentially at a video rate. The only requirement is that an momentary increase in brightness occur during electronic refresh of the display pixel element. The momentary flare in brightness detected by electronic sensor systems is not noticed by the human eye, which averages out transients in brightness less than 0.10 seconds.

The display 2a refresh illuminance conditions depend upon the display medium such as the CRT display phosphors, LCDs or LEDs. The CRT display phosphors generally have a electron beam refresh excitation rise time on the order of several hundred nanoseconds, a fall time on the order of several microseconds, and a decay time on the order of a millisecond. However, more than 90 percent of the illuminance energy is emitted within the first few microseconds and each pixel is dark most of the time. The CRT display is point addressable to maintain pixel spatial compactness. Similar comments apply to thin-film electroluminescence phosphor displays. In contrast, the rise and fall times of liquid crystal displays to electrostatic refresh is on the order of several milliseconds. LCDs are generally interspaced matrix line addressable in which all display elements in a row are refreshed as the column lines are set at the appropriate voltages. Whereas most display designs set the column lines on the same refresh clock pulse, they with little difficulty could be set sequentially at a video sweep rate with no display degradation. Finally, light emitting diodes used in display matrixes have an on-rise time and off-fall time on the order of several hundred nanoseconds. Again LED displays could be refreshed on and off in a vertical and horizontal synchronized manner.

The video display 2a may be incorporated into a head mounted display system; FIG. 4 shows a headset 1 design for a virtual reality display. The user is wearing a head-mounted electronic video display 2a and a periocular array of light sensors 2b all packaged as a complete unit 25. In this case, the unit 25 is supported by a headset 1 which is designed for stability with headbands 26 and hatbands 27 each having contact pads 28 through 31 located on the load bearing portions of the face and head. The load bearing portions of the face and head include cheek bones, having contact pad 28, temporal regions having contact pad 29, the back of the head having contact pad 30, and the inion budge at the rear of the skull having contact pad 31. These skin surfaces are essentially devoid of flesh and muscle unlike the rest of the human face which is extremely pliable forming an unstable base. Contact pad 29 forces hat band 27 to stand off of the forehead. This in done to prevent eyebrow movement from disturbing the settings. Unit 25 houses display 2a optics and sensors 2b, which is supported on the nose bridge 32 by pivot joints located at the sides of hatbands 27. Unit 25 can be rotated up out of the user's field of view.

As shown in FIG. 5a, the video display 2a image light reflected from the user's eye is directed back to the periocular array of light sensors 2b of the optical device 2 by display optics D. The periocular array of light sensors 2b detects the display 2a generated light reflected from the cornea and inner structures of the user's eye. The particular optical design for the sensor array 2b is determined by the video display 2a design of which the sensor unit is a part. A virtual display 2a design shown in FIG. 5a employs an on-axis folded optical geometry. In this case, the light from video display 33 is focused through a dielectric polarizer 34 and folded at a semi-reflective mirror/splitter 35. The light then passes through a quarter-wave polarization rotator (wave plate) 36 to the front visor/combiner 37. The light reflected from the visor 37 passes back through the wave plate 36 and with the light's Electric-vector now rotated 90-degrees, passes back through the splitter 35, from which it passes through another wave plate 38 to the eye 39. The light reflected from the eye 39 passes back through the wave plate 38, and with the Electric-vector now rotated another 90-degrees, is folded by the mirror/splitter 35 onto the periocular array 40 of light sensors 2b. The array of light sensors 2b detects the display generated light reflected back from the cornea and internal components of the eye such as the crystalline lens and the iris surrounding the pupil. FIGS. 5b and 5c show several other designs for the optical device where similar comments apply to the optical geometry. In all designs, the light reflected from the eye 39 passes back through the wave plate 38, and is folded by the mirror/splitter 35 onto the periocular array 40 of light sensors 2b.

The light sensor array 2b uses a fiber-optics bundle probe to acquire an eye image. The output of the optics probe feeds a two-dimensional array of photo-transistors 3a. A spectral filter is used to ensure that only light of the appropriate wavelength from the display 2a is collected. The simple array of photo-transistors 3a instantaneously converts the reflected light to an electrical current. The photo-transistors 3a detect the corneal reflex (first Purkinje image) of the light source from the outer surface of the cornea. The incident light is specularly reflected from the spherical anterior surface of the cornea to create a virtual image of the source. The corneal reflex has a bell-shaped light distribution with the brightest part at the center of the image corresponding to the reflection point.

The sensor array 2b output is piped through the amplifier array 3a to an array of comparators 3b where the amplified output of each photo-detector 3a is compared to a threshold level. The threshold level is set to collect the brightest part of the image. The level is set above the system noise which is mainly due to optical reflections from the other sources, the photo-transistor dark noise, and the Schottky, Johnson, and flicker noises in the amplifiers. Comparator array 3b is interfaced to a matrix array static encoder 3c. A comparator voltage output above the threshold level forces the corresponding matrix address on the output lines of the encoder 3c. The encoder output code is latched for the remaining duration of the raster scan clock pulse 2c. In this way the electronic encoder 3c isolates the photo-transistor 3a with maximum response to the corneal reflex from the instantaneously excited light source 2a. The encoder latch is reset by the next raster scan clock pulse 2c; the reset is momentarily delayed to allow interaction with the digital processor 4.

The electrical output of the latched encoder 3c is the electrical input to the digital processor 4. On the raster scan clock pulse 2c from the video display 2a, the processor 4 writes the encoded address from the latch 3c for the light sensor 2b matrix address to a double buffered digital memory 4a shared with the digital computer 7. The vertical and horizontal matrix address of the corresponding video display 2a pixel element is determined by the display refresh order, which is indexed by the raster scan clock pulse 2c count. The process is repeated by processor 4 during the display refresh cycle, producing a digital memory table 4a of array matrix addresses for photo-transistor sensor 2b and light source 2a pairings, which is accessible by the digital computer 7 routines. The digital memory 4a is buffered and the raster scan clock pulse counter is reset to zero on the clocked field refresh pulse 2d.

The digital computer runs several processes which are released from hibernation upon receipt of the display field refresh clock pulse 2d from the video display 2a. These are routines to compute: the visual line of sight 8, the workspace line of sight 9, fixation status 10 and the gaze point 11 for each eye. The routine 8 computes the visual line of sight relative to the optical device 2. The routine 9 reads the headset position and orientation from sensor 6 and transforms the visual line of sight to that for the real or virtual workspace which the user views through visor 37 of optical device 2. Routine 10 maintains a running record of the line of sight and classifies the eye patterns as fixations, saccadic movements, or pursuit tracking. Finally, routine 11 groups the eye fixations into cognitive meaningful gaze points and movements. The binocular convergence point 12 is then computed for the dual eye system. The gaze points for both eyes determine the binocular point of convergence for the dual eye system.

Routine 8 is essential to the invention. Routine 8 is based on a three dimensional model of the eye derived from the corneal light reflections and pupil image data. The routine calls on a set of subroutines which computes the corneal center and surface reflection points 8a for the display refresh cycle 2d, the pupil center 8b, the optical origin and axis 8c, smoothing of the reflection points 8d, the median axes of the corneal surfaces 8e and the viewing origin and axis 8f. Subroutine 8a makes use of the memory sensor 2b-source 2a pairing table 4a and the optical device 2 geometry. The subroutine 8b uses the pupil centroid output of the processor 5b for the computations as well as the pairing table 4a. The optical origin and axis result from the corneal and pupil centers. Smoothing of the reflection points are required for the median axes computations. Finally, the visual line of sight is computed from the optical origin and the optical and corneal median axes.

The three dimensional model of the eye used in routine 8 is based on expert knowledge briefly described as follows. The adult human eye is about 25 millimeters in diameter and set in an orbital socket in the skull. It is connected to and held in place by external muscles, nerves, blood vessels, and tissues. The sclera an envelope of tough white tissue, holds the eye mechanisms. The cornea is a transparent shield over the front of the eye. Vision occurs when light entering the eye through the cornea, passes through the pupil defined by the iris, and is focused by the transparent lens onto the retina, a lining of light sensitive nerve elements at the back of the eye connected directly to the optic nerve.

The center of rotation of the eye is located approximately 13 millimeters behind the front surface of the cornea. For normal, unaided viewing, the comfort limit for eye rotation from straight ahead is approximately 40 degrees. The surfaces of the optical elements of the eye are not perfectly spherical but are somewhat flatter at the peripheral edges than at the optic center. However, the cornea is nearly spherical within the central 30 degrees. The radius of curvature of the anterior surface of the cornea is 7.8 millimeters for the normal human eye, and 6.8 millimeters for the posterior surface.

The cornea and crystalline lens have optical axes that do not quite coincide, and the optical axes of the eye does not coincide with the visual axis (the line connecting the fovea to the point of fixation) by 5 to 7 degrees. Despite this complexity, for practical purposes the eye can be regarded as having four major refracting surfaces: the anterior and posterior surfaces of the cornea, and the front and back surfaces of the transparent lens. These surfaces may be regarded as spherical, with a common axis, and indices of refraction may be taken as constant within any given optical element. The refractive index of the cornea is commonly about 1.377; that of the aqueous humor and vitreous body within the eye is normally about 1.337.

The digital computer routine 8 computes the center of the corneal anterior surface in the optical device 2 coordinate system from the sensor 2b-source 2a pairings listed in the memory table 4a. As part of this computation, the routine converts the pairing table to a mapping of the corneal surface locations of the reflection points and the corresponding corneal radii for the tabulated pairing 4a of sources and sensors. The routine has in permanent memory the locations and orientations of the sensors 2b and light sources 2a in the optical device 2 coordinate system referenced by the appropriate matrices addresses. For example, the data file for a virtual image head mounted video display system contains the directional cosines of the image light from the display elements in the coordinate system of the video device 2 housing. The directional cosines are determined by the display optics operating on the fixed matrix of display elements in the video display 2a. Furthermore, the location and directional cosines for the sensors 2b are fixed in the display coordinate system by the sensor optics. These locations and directional cosines are computed prior to usage and fixed in permanent memory for the routine.

A set of five equations describes the relation between the center of the anterior spherical corneal surface and the orientation of the display light source 2a and the location and orientation of the sensor 2b paired at a corneal reflection point. The equations depend upon the type of video display 2a; the source position is specified for the direct view display, while the orientation of the source is specified for the virtual image display. Three of the equations relate the reflection point 17 to the corneal center, the source 16 orientation, and the sensor orientation 18 and location 19 as shown in FIG. 3. The application of Snell's law gives two more equations, one for the equality of the incident and reflection angles at the reflection point 17, and the other for the coplanar orientation of the source 16 ray, sensor 19 reflection ray 18, and the corneal surface normal at the reflection point 17. Another equation is given by the fact that the corneal surface normal has unit magnitude.

Tabulated data from three pairings 4a of emitting sources to sensors are needed for a solution; regression analysis techniques are applicable with data from more than three pairings. The equations are solved using numerical analysis techniques The output of the subroutine is the location of the corneal center in the coordinate system of the video device 2 housing for each display refresh cycle and the corneal radius and reflection point coordinates for each pairing in the table 4a during that cycle. The processor first solves for the directional cosines of the corneal surface normal at each paired reflection point using Snell's laws and the unit magnitude property; a unique solution is specified by the fact that the viewing direction is toward the display. The three equations relating the reflection point coordinates to the emitting source and receiving sensor are then solved for the corneal center and the reflection points radii using the data from three or more pairings.

Given the corneal center, the optical axis may be estimated from the difference between the center and the theoretical rotational point for the eye in the optical orbit of the skull. The rotational point may be assumed fixed in the head coordinate system and may be determined in calibration. However, in actuality, the eye rotational point will shift slightly in the optical orbit of the skull with muscular tension and rotational movements. These translational shifts can produce severe displacements in viewing direction; for example, a 0.1 millimeter shift in rotational point can produce a 1.4 degree shift in viewing direction. Furthermore, the eye will roll in the optical orbit to compensate for slight tilts in head orientation. These changes cause inaccuracies in the computation of the viewing direction based on just the corneal center.

For this reason, the tabulated mapping of the corneal surface locations of the reflection points for the paired sources and sensors and the associated corneal radii are used to determine key corneal features. The tabulated corneal radii are first smoothed by averaging over adjacent reflection points determined by a clustering algorithm based upon the proximity of angular separations about the corneal center. The table of smoothed radii are then searched for clusters with minimum, maximum, and average values. While the central 30-degrees of the corneal anterior surface is spherical in shape, the basilar surface is slightly astigmatic with two principal meridians perpendicular to each other. The corneal horizontal radius of curvature (normally 7.98 mm) is roughly equal to that of the optical center; however, the vertical radius is less (7.60 mm). Therefore, routine 8 first isolates minimum radii points that are separated by more than 30-degrees to define the vertical meridian. The surface point midway between the two minimum radii points defines the optical surface center; the corneal radius is the distance between the corneal center and the optic surface center. Finally, routine 8 searches for radii points equal to corneal radius that are separated by more than 30-degrees to define the horizontal meridian.

The optical axis is defined by the corneal center and the optic surface center; routine 8 computes the center and directional cosines for the optical axis in the display coordinates from these values. The roll of the eye is then computed in routine 8 to complete the specification of the optical orientation. The vertical and horizontal meridians are used to compute an orthogonal set of coordinate axes perpendicular to the optical axis.

The result is an accurate description of the optical axes and center in the display coordinate system for each display refresh cycle. However, the visual axis extends from the first Gaussian nodal point of the eye (normally near the center of curvature of the corneal anterior surface) and is commonly perpendicular to the cornea; it is generally directed 5 degrees inwards to the optical axis and 2 to 3 degrees upward. The visual axis is defined by an origin point in the optical axes coordinate system and three directional cosines determined in an initial calibration process. These parameters defining the visual axis are transformed into coordinates for the display coordinate system in routine 8 from the relations for the optical axes. The result is an accurate specification of the visual axis.

In some video device 2 designs, the corneal mapping computations may inadvertently include the corneal peripheral when the eye is rotated to the side; in this case the computations may be in error due to incorrect centering on the cornea. Several checks on the mapping computations are made based on the expert knowledge of the cornea derived from the calibration process. First, the angular separations between the optic surface center and the reflection points for the maximum and minimum radii as measured from the corneal center should be fixed values normally greater than 15-degrees and less than 45-degrees. Further, the optic center corneal radius should agree with that derived in calibration.

However, a further embodiment of the invention uses the computation of the pupil center in the video device 2 coordinate system to center the corneal computations and as a check on the determination of the optical axis. In this design, the output of the opto-transistor array 3a is piped to a CCD video accumulator 5a during the display refresh cycle. The device accumulates an image of the eye with the reflected light from the excitation and decayed states of the refreshed light sources over the display refresh cycle. The accumulated image is downloaded to an image frame grabber at the end of the cycle output and then piped to an image processing board 5b where the centroid of the pupil image is computed using a binarization and image brightness threshold technique. The center 25 and major and minor axes of the centroid 23 in the accumulator's image plane 24 coordinates as shown in FIG. 3, are computed and stored in memory accessible by routine 8. Finally, the pupil diameter is computed from the pupil image centroid major and minor axes and the optical axes.

The pupil optical center is computed in routine 8 using a variation of an ophthalmometrical ray tracing technique for determining the position of internal surfaces of the eye. The tabulated listing 4a of sensor to light source pairings is searched for a sensor set 22 overlapping the apparent pupil image center 25. The light ray from the optical pupil center is transmitted through the eye and refracted to the image plane 24 of the accumulator 5a, from the cornea at the reflection point 21 for the display light sources 20 paired to the sensor set 22. The transmitted ray is coplanar to the incident and reflected light from the set of paired display sources to the corresponding sensors, and the refracted ray is parallel to the reflected light. The location of the reflection point 21 on the anterior surface of the cornea is computed from the tabulated source orientations 20 and sensor 22 positions and orientations. The directional cosines for the transmitted ray is computed from the corneal index of refraction and Snell's law relating the angle of incidence to the angle of refraction. In turn, the optical pupil center is computed from the prior computation of the corneal center and the distance between these centers. Finally, the optical axis is computed from the locations of these two centers. The result is the computed center and directional cosines for the optical axis of the eye in the video display 2a coordinate system as a check on the computations made from the corneal reflection point table.

The accuracy of these computations is dependent upon an initial calibration step in which the user gazes in turn at a series of cue markers displayed on the video display 2a. The data collected is used to solve for characteristics of the user's eye such as the corneal radius of curvature and the separation distance between the pupil center and corneal center. The relation of the viewing axis to the optical axes is determined by solving for the axis origin and the directional cosines in the optical axes coordinate system.

In some applications, the user may be viewing through the visor 37, an external real or virtual workspace which is independent of optical device 2; we may wish to know which elements of the visual workspace are being viewed. In this case, the visual line of sight computed by routine 8, must be transformed to that for the external workspace. This is accomplished by first locating the headset in the workspace. A tracking system 6 is used to measure the position and orientation of the headset 1 holding the video device 2. The tracking system consists of a source 6a, sensor 6b, and electronics processor 6c. The source is mounted in the user's workspace; the sensor is mounted on the headset. The system allows continual tracking of the six degrees of spatial freedom; the processing unit continually computes the position and orientation of the sensor relative to the source, and controls the transmission of the output data. Tracking systems are available employing different sources: infrared, ultrasonic, optical, and magnetic fields.

The computer routine 9 computes the user's viewing origin and direction in his visual workspace from the viewing point and axis given in the optical device 2 coordinates and the position and orientation of the headset as measured by the tracking system 6. The viewed point in the user's visual workspace is then computed using computer graphics concepts and a computer memory data base listing the objects and their locations in the user's workspace. The routine 9 computes the point of intersection of the viewing direction with each workspace surface facing the user from the surface normal and the location of a vertex. The routine checks to ensure that the intersection point is within the edges containing the surface. The viewed point is the contained intersection point closest to the user's eye.

The positions and orientations of the surfaces of the objects in the user's visual workspace are listed in a digital memory computer data file 9a by the directional cosines of their surfaces normals and the coordinates of the vertices. The surfaces of the objects are described by a piece-wise net of planar segments for computational purposes. Note that the objects being viewed for a virtual workspace are those generated by the drawing file for the display driver of video display 2a; in this case, the digital memory computer data file 9a is defined by the virtual workspace used to set up the display driver. Similarly, the position coordinates and directional cosines of the orientation of the tracking system source 6a in the workspace are listed in the computer memory data file.

The computer routine 10 performs as an expert system, classifying the immediate visual state defined by the workspace viewing origin and direction, as fixation, saccade, pursuit or blink from an embedded expert knowledge base of these ocular states. An understanding of the visual process is necessary to understand how the electronic expert system functions. Three sets of extraocular muscles control the movement and positioning of the eye. These muscle pairs include the horizontal and vertical rectus muscles, the lateral and medial rectus muscles, and the inferior and superior oblique muscles. Each pair of muscles exerts effort in a different axis, so that the eye is capable of movement to an extremely large number of positions. Some of the movements are involuntary and involve compensatory responses to movements of the head or to stimulation of certain facial nerves.

When studying a static visual display, the eye is continually rotating and fixating different portions of the scene; for this reason, the eye gaze consists of sudden saccadic jumps with interposed slightly jittery fixations. The saccadic jumps which move the eye between fixations are sudden and rapid ballistic movements lasting about 30 to 120 milliseconds and transversing 15 to 40 degrees. The peak velocity for a 10 degree saccade can reach 450 degrees per second; the peak velocity for a 20 degree saccade can range from 290 to 400 degrees per second. Vision is suppressed during a saccade. A 200 millisecond refractory period is required for recovery before another saccade can be executed.

The fixation following a saccade is a brief period of relative stability which allows that portion of the scene within the foveal vision to be studied. The fixation period is typically from 200 to 600 milliseconds. Small jittery motions occur during a fixation; these motions are composed of slight drifts with amplitudes less than one degree, followed by microsaccadic corrections. A low amplitude, high frequency tremor is superimposed. The purpose of a saccade is to rotate the eye so that the image of a cue in the visual scene falls within the foveal vision. The image is then studied during the fixation; the microsaccadic movements induce changes in the image required for seeing. However, the eye seldom remains in one fixation for long.

On the other hand, pursuit movements are slow involuntary eye tracking movements which smoothly follow a moving object of interest in the visual field. Pursuit movements have an initial reaction time of 0.12 to 0.25 seconds and are limited to a maximum velocity of 25 to 30 degrees per second. While the saccadic control system acts as a position servomechanism directing the fovea of the eye to a target, the pursuit system acts as a velocity servomechanism rotating the eye at the same angular rate as a target. Finally, blinks start slightly before or simultaneously with upward directed saccades, but never after the eye movement. In contrast, blinks are sometimes initiated after the onset of downward directed saccades. It has been reported that only in less than one percent of blinks is there apparent eye movement. Blinks usually last from 20 to 200 milliseconds with the longest observed terminating smoothly around 300 milliseconds.

The routine 10 detects saccades and predicts fixation end points, as well as separates saccades from pursuit eye tracking of moving targets, and eye blinks. The routine uses an automatic fixation and saccade classification scheme operated in conjunction with a saccadic end point predictor. The classification scheme is a combination of these two methods: the position variance method 10a and the velocity detection method 10b, operated as parallel processing channels. The end point predictor 10c is derived from a knowledge base on the relation between saccade peak velocity and amplitude of movement. The position variance method 10a is based on the expert knowledge that a fixation is characterized by relative immobility with low variance in eye movement, while a saccade is distinguished by rapid change (high variance) in position. In this method, the means and variances are computed for a sliding window of time samples in eye position. The variance of the windowed samples for a fixation is lower than an empirically determined threshold level. When a saccade occurs the variance rises, reaches a peak and then subsides toward the fixation level again. The initiation and termination of a saccade is automatically detected by comparing the position variance to the threshold level determined in an initial calibration process. Furthermore, curve fitting to the variance function is used to make predictions about the future variance values and therefore the time and location of the next fixation. The method has a time lag attributable to the width of the sampling window.

The velocity detection method 10b operates in parallel. In this method, the eye movement speed is computed from the windowed samples by first smoothing with a low-pass digital filter to remove noise, and then numerically differentiating the filter output. The computed speed is compared to an empirically determined threshold level separating saccadic movements from fixations. A speed above the threshold is classified as saccadic while that less than the threshold is classified as fixation. The position variance and velocity methods give similar results. These two methods are combined to obtain a more reliable saccade discriminator. The combined method can be based on either the agreement between the separate threshold outputs, or more elaborately, upon saccade state-probability estimates derived from the magnitudes of the position variance and speed measurements. The saccadic end point predictor 10c is activated once a saccade has been detected. This method predicting the end point of the saccadic movement from the peak velocity, discriminates between saccades and pursuit tracking. The predictor is based on the expert knowledge that the eye-movement velocity function during a saccade is nearly symmetrical about a peak velocity located approximately half-way to the next fixation. In detail, the saccadic velocity function rises above the threshold detection level, accelerates to a peak velocity about half-way to the next fixation, and then decelerates below the threshold at the fixation point. Although it has been reported in the literature that a sine wave is a good fit for the velocity function, the function is in fact, more or less symmetrical depending upon the magnitude of the peak velocity. In actuality, the saccadal amplitude is related to the maximum velocity by an exponential function using curve fitting constants. A look up table contains empirically determined correction factors needed to adjust for the asymmetry in the velocity function. The correction factors are derived in the calibration process. Therefore, the processor by detecting the peak of the saccadal velocity, uses the relation to the saccadal amplitude to predict the termination time and location of the next fixation. A classification as saccadal movement that does not satisfy the end point predictor is reclassified as pursuit movement; these movements are less than 30 degrees per second. Blinks correspond to cases where no reflection points are detected due to obscuration of the cornea by the eyelid.

The computer routine 11 groups the fixation data into task related visual gaze points. The user will tend to view a task related visual element with a sequence of fixations all in the same display area connected by small saccades. These fixations grouped together form a single gaze point of regard that more closely approximates the user's cognitive functioning. The routine uses a retrospective algorithm to determine the start, continuation, and termination of each gaze point as follows:

(1) A 100 millisecond sequence of eye fixations within 0.5 degrees of each other defines the start of a gaze point; the gaze point center is the mean of the eye fixations. The algorithm delay of 100 milliseconds is undetectable to the user.

(2) The gaze point is continued as long as eye fixations remain within one degree of the current gaze point center. The gaze dwell time is the sum of the 100 millisecond start time and the current continuation time.

(3) A 50 millisecond sequence of eye positions that are beyond one degree of the current gaze point defines the termination of the present gaze. A blink up to 200 milliseconds is allowed without termination.

The routine 11 computes the gaze point as the visual viewing point of the eye as computed by routine 9 for pursuit tracking; initially pursuit tracking will consist of a saccade to acquire the target on the fovea followed by steady tracking. A rate varying target will be tracked with saccadic updating in tracking rate.

The routine 12 computes the convergence points of the left and right eye gaze directions for a dual eyetracker configuration. The convergence points are the points of closest approach between the two gaze lines; the line segment connecting the two lines at these points is perpendicular to both lines.

Appendix A is a computer program written in C-language which implements the computations of routines 8, 9, 10, 11 and 12 for an optical device 2 as described above. The computer program uses the tabulated display source-to-sensor pairing table 4a produced from digital processor 4 and the pupil image processing from image processor 5b. The program computes the separate gaze points for each eye and the binocular convergence point for a dual eye system.

The computer program calls on the subroutine "Initial.sub.-- values()" to read in the calibration values and supporting data tables on the display source and sensor arrays, and the workspace panels. The program next enters a loop where it waits for a pulse trigger 2d from the display field refresh clock of the optical device 2. Following the interrupt, the program first calls on "Get.sub.-- time()" and then for each eye in turn loads the working files from "Load.sub.-- tables()" and calls for computation of the gaze point from "Update.sub.-- gaze()". The program then saves the computations for that eye in "Collect.sub.-- gaze()". Following computation of the gaze point for each eye, the program computes the dual eye convergence point with "Converge()", and outputs the results to the host computer from "Output.sub.-- gaze()". The program then returns to the start of the loop to wait for the next interrupt.

The computer program performs the gaze point computations of routine 8, called for from "Update.sub.-- gaze()", in accordance with the following mathematical relations described by equations represented in vector dot (.) and cross (x) product notation, and scalar multiplication (*) and division (/) notation. Vectors are in upper case and scalars in lower case. Subroutine "Corneal.sub.-- ctn()" acting as 8a, computes the location of the corneal center and the corneal radii for the reflection points from the source to sensor pairing table 4a. Let Pr[xr,yr,zr] represent the location in the display space 2a Cartesian coordinates of the corneal anterior surface reflection point 17 of the source 16 as detected by the sensor element 19 at the location Pe[xe,ye,ze]. Similarly, the directional cosines of the source 16 are represented by Ns[as,bs,cs], those of the ray 18 from the reflection point 17 to the sensor 19 are Ne[ae,be,ce], while Nr[ar,br,cr] are those of the corneal anterior surface normal for the radius, rc, originating from the anterior center Pc[xc,yc,zc] to the reflection point 17. The three equations relating the locations of the corneal center Pc and point 19 via point 17 are given in vector notation by:

where re is the distance between the points 17 and 19. Snell's law equating the ray's incidence and reflection angles about the surface normal Nr at point 17 gives the equation:

Similarly, the law requiring that the incident and reflected rays be coplanar with the surface normal gives the equation:

Finally, the unity requirement on the surface normal gives the equation: Nr.Nr=1. The unknowns (Pc[xc,yc,zc], Pr[xr,yr,zr], rc, re) are solved for from a set of 18 equations with 18 unknowns for three source to sensor pairings 4a assuming that the corneal center location Pc is the same for all pairings. These equations have a unique solution given the display direction and that the user is looking at the display 2a.

The routine "Chk.sub.-- pupil.sub.-- ctn()" acting as 8b, computes the pupil center from the pairing table 4a and the pupil image center. Consider the corneal surface reflection point 21 mapping to the pupil image center 25. The ray from the pupil center which is refracted at this point to the CCD accumulator 5a, coincides with the ray reflected from source 20 to the sensor 22. The reflection angle, ar, is equal to the inverse cosine of the dot product of the unit vectors for the surface normal at point 21, Nr, and the reflected ray, Ne,

In turn, the refraction angle, at, is computed from Snell's law by

where index is the effective optical index of refraction. The directional cosines of the refracted ray unit vector, Nt[at,bt,ct], from the pupil center to point 21 are next determined from three equations. The dot product of the unit vectors for the refracted ray, Nt, and the point 21 surface normal, Nr, is equal to the cosine of the refraction angle,

Furthermore, the unit vectors for the refracted ray, Nt, is coplanar with those for the incident ray, Ns, and reflected ray, Ne, at point 21,

Finally, the refracted ray unit vector has unit length and the corresponding dot product equals unity,

The distance, rt, along the refracted ray, Nt, from the point 21 to the pupil center is computed. The pupil center is the intersection point of the ray with a sphere centered on the corneal center, Pc, and with radius, rp, the distance between the corneal center and the pupil center, a parameter derived in calibration. The distance rt is given by--

where rc is the corneal radius at point 21. The pupil center, Pp[xp,yp,zp], is computed as--

The pupil center is used to compute the optical axis from the relation Pp=Pc+rp*No between the corneal center and the optical unit vector in accordance with 8c.

The subroutine "Cluster.sub.-- pts()" acting as 8d, smoothes the corneal radii by a clustering algorithm performed with the dot product of the corneal surface normals Nr[ar,br,cr] for all reflection point combinations. The dot product is equal to the cosine of the angle between the unit vectors of the surface normals, and the corneal radii for product values larger than a critical value are averaged together for smoothing. That is, given reflection point i, the corneal radii of all other reflection points j such that

are averaged with the radius of reflection point i to form a smoothed value for point i.

The subroutine "Get.sub.-- optical.sub.-- axes()" acting as 8e, computes the optical axes system from the smoothed corneal radii. The optical surface center lies near the optical axis. The routine searches the table for a reflection point k with minimum distance

from this line, where rk=.vertline.Prk-Pc.vertline.. The corneal radius for the optical center is rco=rc[k]. The routine next computes the vertical axis by searching the pairing table for the occurrence of two minimal radii surface points separated by more than 30-degrees, as measured by the dot product of the corresponding surface normals, to define the end points of the vertical meridian of the cornea. The points must be each more than 15-degrees from the optical axis. Letting i and j be two surface points such that the corresponding corneal radii--

and the dot product of the corresponding surface normals is--

while that of each point with the optical axis is--

then the vertical meridian is connected by the chord with unit vector Nz[az,bz,cz]--

where dz=.vertline.Pri-Prj.vertline. is the straight line distance between the points.

Finally, the horizontal meridian is defined by two surface points which each more than 15-degrees from the optical axis, are separated by 30-degrees and with corneal radii close in value to the optic center value. That is two points i and j such that .vertline.rci,j-rck.vertline.<acceptable.sub.-- error, and the dot product of the corresponding surface normals,

The unit vector of the horizontal axis is defined by,

As a check on the computations, the optical coordinate system is orthogonal and the dot products of the corresponding unit vectors must be zero, i.e., Ny.Nz=0, No.Nz=0, and No.Ny=0. Furthermore, the optic center corneal radius, rck, must be close to the value derived in calibration, rco, that is, .vertline.rck-rco.vertline.<acceptable.sub.-- error.

The subroutine "View.sub.-- axis" acting as 8f, computes the origin and unit vector for the viewing axis in the display 2a coordinate system from the optical axes computations and known parameters derived in calibration. These parameters include the location of the viewing origin and the angular orientation of the viewing axis in the optical coordinate system. The viewing origin, Pv[xv,yv,zv] is located a fixed distance rx along the optical axis No, a distance ry along the horizontal axis Ny, and a distance rz along the vertical axis Nz. The viewing axis Nv[av,bv,cv] is orientated at a fixed elevation, ev, to the vertical axis and an azimuth, av, to the optical axis in the plane defined by the optical and horizontal axis. The viewing origin, Pv, is given by--

and the unit vector for the viewing direction by--

These two parameters determine the viewing direction in the display coordinate system.

The viewed point in the work space coordinate system is computed by the subroutine "View.sub.-- point()" in accordance with routine 9 described above. The subroutine translates the viewing position and direction in the optical device 2 coordinates to that in the workspace using the headset sensor 6 position and orientation returns. The display 2a coordinate system is centered on the sensor position and aligned with the corresponding axes by design. The sensor returns give the sensor location Ps and the azimuth, as, pitch, bs, and roll, cs, of the sensor in the workspace coordinate system. The sensor Cartesian coordinate system is centered at Ps with the orthogonal axes Ixs[axs,bxs,cxs], Iys[ays,bys,cys], and Izs[azs,bzs,czs] unit vectors, where the corresponding directional cosines are given for Ixs by--

for Iys by--

and for Izs by--

in terms of the orientation angles. The viewing point Pe[xe,ye,ze] in the workspace is given by--

and the viewing direction Ne[ase,bse,cse] is--

A planar surface is located in the workspace by an origin Po[xo,yo,zo] and a surface normal Ndz[az,bz,cz]. The surface faces the user if the dot product of the viewing direction and surface normal is less than zero, i.e., Ne.Ndz<0. The distance, rs, between the viewing point and the surface along the viewing direction is given by--

and the intersection point Pi with the surface is Pi=Pe+rs*Ne. The surface contains a convex polygonal panel described by a set of vertices {xcor,ycor} in the surface coordinate system defined by the orthogonal unit vectors Ndx[ax,bx,cx] and Ndy[ay,by,cy]. The intersection point Pi is transformed into surface coordinates Pd[xd,yd] by--

where ri is the distance between the surface origin, Po, and the intersection point, Pi, ri=.vertline.Pi-Po.vertline., Ndi is the corresponding unit vector, Ndi=(Pi-Po)/ri, and scx and scy are the appropriate coordinate scaling factors. The routine checks that the intersection point is within the polygonal borders by computing the directional cosines of the unit vector between the point and each vertex, and then summing the angles formed between consecutive vertices from the inverse cosine of the dot product of the corresponding unit vectors. The angles will sum to 180-degrees for an enclosed point and zero-degrees for a point outside of the polygon. The polygon that has the shortest viewing distance is seen by the user.

The subroutine "Fixation()" classifies the viewing fixation in accordance with the routine 10 described above. The routine keeps a running record of the azimuth and elevation viewing angles and the corresponding angular velocities. The routine "Get.sub.-- vel()" computes the present velocities from the position record using numerical differentiation following lowpass filtering to remove noise. The routine then computes the variance and velocity and classifies the viewing as fixation if these are less than thresholds. The viewing is classified pursuit if the variance exceeds the threshold but the velocity while greater than that for fixation is less than that for saccade. Finally, the routine classifies the viewing as saccade if the all thresholds are exceeded; the routine "Predict.sub.-- endpt()" predicts the saccadic endpoint by fitting the velocity record to a half-sine wave represented by a second order polynomial.

Subroutine "Gaze.sub.-- point()" reduces the fixation record to a gaze point in accordance with routine 11. The routine keeps a running record of the fixation azimuth and elevation viewing angles and the time between samples. The routine tests the record for the occurrence of a gaze point.

The subroutine "Converge()" computes the convergence points of the left and right eye gaze directions for the dual eyetracker configuration in accordance with routine 12. The convergence points are the points of closest approach between the two gaze lines; the line segment connecting the two lines at these points are perpendicular to both lines. Let this closest approach-line segment joins the left gaze line, with origin Pvl[xvl,yvl,zvl] and unit vector Nvl[avl,bvl,cvl], at the left convergence point Pcl[xcl,ycl,zcl]. The segment joins the right gaze line, with origin Pvr[xvr,yvr,zvr] and unit vector Nvr[avr,bvr,cvr], at the right convergence point Pcr[xcr,ycr,zcr]. Since the line segment is orthogonal to the gaze lines, the dot products of the corresponding unit vectors must be zero--

The distances rl and rr from the gaze line origins to the corresponding convergence points are computed from--

where a=Nvl.(Pvl-Pvr), b=Nvr.(Pvl-Pvr), and c=Nvl.Nvr. The corresponding convergence points are given by--

The gaze convergence point is the average of these two points.

Appendix B is a computer program written in C-language which is used for calibration of the invention. The program computes the corneal radius of curvature, the origin and directional cosines of the viewing direction in the corneal optical axes system, and the separation between the pupil center and the corneal center. The invention is auto-calibrating since the user can quickly and accurately go through the calibration procedure which correlates eye gaze position with eye line of sight.

The program "Calibrate()" controls the calibration process and computes the appropriate constants from the user's viewing response data. The program calls for the sequential display of a series of cue markers from "Draw.sub.-- target()" for the user to look at. As each cue is displayed the program waits for the user's response from "Hold.sub.-- response()" and then collects the response data for that marker. The routine "Get.sub.-- gaze" computes the corneal center and optical axes for each cue marker by calling on the appropriate routines found in the main program of Appendix A. The cue markers are placed in the visual scene so that the eye is in full view to the sensor array. This ensures that the optical axes may be derived from the source to sensor pairing table as described for subroutine "Calib.sub.-- optical.sub.-- axes" above without reference to the pupil image. The location Pt[xt,yt,zt] of the cue marker in the display coordinates is related to the viewing point, Pv, and direction, Nv, by--

where rt is the straight line distance between the two points. In turn, the viewing point, Pv, is located a fixed distance, rx, along the optical axis, No, a distance, ry, along the horizontal axis, Ny, and a distance, rz, along the vertical axis, Nz, and is given by--

The viewing axis, Nv, is orientated at a fixed elevation, ev, to the vertical axis and an azimuth, av, to the optical axis in the plane defined by the optical and horizontal axis and is given by--

The fixed distances rx, ry, and rz, and the fixed angles av and ev are unknowns which are computed in the calibration. These equations are combined to form--

where--

The above expression form a set of three equations in the three unknown a, b, and c, for each cue marker presentation. In turn, the solution of the parameters a, b, and c for three cue markers enables the solution of the parameters rx, ry, rz, av, ev, as well as the three marker to view point distances, rt1, rt2, and rt3.

The distance rp between the corneal center and the pupil center is computed in the calibration process. The routine "Get.sub.-- pupil.sub.-- ctn()" searches the pairing table 4a for a source reflection point that overlaps the pupil image center. In a manner similar to that for the routine "Chk.sub.-- pupil.sub.-- ctn()", consider the corneal surface reflection point 21 which is mapping to the pupil image center 25. The ray from the pupil center which is refracted at this point to the CCD accumulator 5a, coincides with the ray reflected from source 20 to the sensor 22. The reflection angle, ar, is equal to the inverse cosine of the dot product of the unit vectors for the surface normal at point 21, Nr, and the reflected ray, Ne,

In turn, the refraction angle, at, is computed from Snell's law by

where index is the effective optical index of refraction. In turn, the angle ao at the corneal center between the optical axis and the surface normal is given by the inverse cosine of the dot product of the corresponding unit vectors--

The corneal center, Pc, the surface reflection point 21, and the pupil center Pp (assumed to lie on the optical axis) form a triangle with adjacent sides rp and rc, the corneal radius for the point 21, with opposite angles, at, and pi-ao-at, respectively. The distance rp between the corneal center and the pupil center is therefore given by--

in accordance with the trigonometric law of sines.

The invention can be used with video display 2a that are not head mounted but rather mounted in the workspace separate from the user such as panel mounted displays. The display illuminates the user's eye as he views the task related display elements. For example, a near infrared light source array can be positioned within the field of view of the task centered displays. In addition, the sensor system 2b need not be head mounted; rather the sensor system could be mounted in the workspace separate from the user. In this case, the sensor system must be continually oriented toward the reflected light from the eye. For this reason, some sort of head sensor 6 either head mounted or separate from the user must be used to located the eye and align the light sensor array 2b.

FIG. 8 is a schematic showing the invention used to control computerized machinery from video displays. The user of the invention is viewing a visual display 41 which has task related elements 41a pertaining to the control of the computerized machinery 15. The user controls the machinery by gazing at the task related display element 41a which corresponds to the desired control action. The instant invention by design serves as a peripheral to the host computer system 13. The invention determines the display 41 location being gazed and transmits this information by output to the host computer 13. The host computer 13 compares the gazed location to those of the display elements 41a. Once a match is made, the host computer 13 executes the appropriate task related routines updating the state of the controlled machinery 15 and correspondingly, that of the visual display elements 41a as feedback to the user.

The visual display 41 may be either a real visual display seen through the display 2a visor, or an apparent image generated in virtual reality by the head mounted electronics display 2a or a combination of both with the virtual image overlaid on the real. In the latter case, the display driver for the electronic video display 2a is under the control of the host computer 13 to provide virtual reality or overlaid display elements 41a as control cues and task critical information.

When the visual display 41 is generated by the video display 2a, the number of pixels activated in a refresh cycle will vary with the visual display 41 as dictated by the host computer 13 in response to the state of the machinery 15 and the user's control selections. The entire display 2a will be activated for such uses as television camera returns, night vision goggles, and virtual reality displays used in computer graphics visual simulations. In these cases, all display pixels are used as light sources. On the other hand, so as not to interfere with the external scene, a limited number of line structures may be activated to display task critical information in virtual image, heads-up helmet mounted displays used in aircraft and helicopters. However, these line structures are drawn in predetermined patterns. Light filters are incorporated with the visor in heads-up displays to reduce display washout from ambient lighting from the external scene and the other electronic video displays located within the visual workspace. Furthermore, the intensity and radiant spectrum of the display's activation pattern are distinct characteristics readily processed by signal treatment. This is particularly true for the CRT driven phosphorous emitter displays. Finally, the video display 2a may show no discernible image to the user at all; instead the video display 2a may be a sequential series of near infrared light sources used merely to track the user's eye. In this case, the video display 2a is independent of the visual task display 41 and not under the control of the host computer 13. For example, the display 2a may be a LCD driven by a glow lamp through an infrared filter. Or the display 2a may be a matrix of near infrared LEDs. Finally, the display 2a may be a special design including task centered visually discernible elements combined with infrared sources.

The invention output to the host computer 13 in real time is: (1) the eye status flag, (2) the eye pupil size, (3) the gaze status flag, (4) the current gaze point location, (5) the gaze dwell time, (6) a task interest flag, and (7) the head position and orientation from the output of processor 9. The eye status flag is the output of routine 10: fixation, saccade, pursuit, or blink. The gaze status flag indicates either the start of a gaze, continuation, or termination. The gaze location gives the point of regard in both workspace coordinates and workspace surface by surface identifier and surface location. The routine has an initial 100 millisecond delay in detecting the start of a gaze and a 50 millisecond in detecting the end. The gaze dwell time is an indication of the user's cognitive task related interest in the visual element being studied. The dwell times tend statistically to be on the average longer for elements of interest. The task interest flag is reset to "non-interest" when the dwell time is less than a lower threshold value determined in calibration, usually about 150 milliseconds. However, the flag is set to "interest" when the dwell time exceeds an upper bound, usually about 250 milliseconds. The gaze status flag indicates a "saccade" during the large saccades that occur between gazes and "failure" when the processor fails to determine an eye position for more than 200 milliseconds. A further embodiment of the invention is the simultaneous tracking of both eyes with a dual display having a pair of video displays 2a and sensor system 2b. This allows the computation by computer routine 12 of the binocular convergence point in the user's three dimensional workspace for either real or virtual stereographic viewing. The convergence point is defined as the workspace point where the viewing axes are closest. In this case, the above output is made for both eyes separately along with the convergence point in the workspace coordinates.

The host computer 13 upon receiving a set task interest flag compares the gaze point location to those listed for the visual display elements 41. Once a match is made, the host computer 13 executes the appropriate task related routines updating the state of the controlled machinery and correspondingly, that of the visual display elements as feedback to the user. However, in some cases, the selected decision may have critical implications for the task performance. An example is that of engaging hostile fire upon another vehicle in warfare. Note that the relation of gaze dwell time to task elements is a statistical process that may vary under stress as the task conditions change. For this reason, in the event of critical tasks, the host computer 13 as an intermediate control step, will highlight the matched display element 41 with a change in color, brightness, or shape as feedback to the user. The appropriate control routines are then implemented if the user executes a manual action 14 in confirmation of the choice selection. The highlighting for element 41 is dropped if the confirmation 14 is not received within a preset time-out period, say several seconds. The manual confirmation action 14 may be a button push by the user or voice command via an automatic speech recognizer. Of course, the control routine executed along with the accompanying display update is determined by the display element 41 selected and the host computer 13 and associated display driver programs.

The accuracy of the eyetracker may be limited by the angular size of the human fovea (+/- 0.5 degree); the visual scene is integrated in the visual cortex as ocular fixations occur about the scene. However, the operator of a computer driven tactical display may wish to select among display icons of display elements 41 that are closer than this in size and perhaps even overlap. For this reason, a companion display to the tactical display may be shown by toggle switch selection, either adjacent to or as an overlap showing the icons in order of designation separated at 0.5 degree increments. The operator then looks at the companion display to make the selection.

The eye status and pupil size are passed to the host computer 13 for human factors analysis; the gaze point and duration are all that is needed for display element 41 selection. However, there is evidence that pupil dilation and eye blinks are related to information processing stages and user's workload. These are factors which are important in controlling the rate and form of display element presentation by the host computer 13. For example, it has been shown that the pupil will dilate at the start of information processing reaching its maximum just before decision, and contract at the moment of decision making. The extent of the dilation is an inverse function of the intelligence of the viewer and direct function of the complexity of the problem to be solved. However, the pupil change is also a function of extraneous factors including ambient lighting and subject fatigue. For these reasons pupil dilation is a valid measure of the stages of information processing only in constant illumination displays for alert users.

Furthermore, blinks are reported to occur at transitions between stages of information processing. In particular, blinks appear to mark brief breaks that the brain takes at the end of each phase of a mental task, and in this way punctuate the sequence of mental events involved in acquiring and processing information. The occurrence of eye blinks may punctuate the appropriate moments to measure pupil dilation in information processing. Another factor to consider is the level of workload which the user is experiencing. Eye blink rate has been shown to be non-linearity dependent on mental workload. Blink rate and duration are related to boredom, anxiety, and fatigue. Blink rate increases with boredom, anxiety, or other sources of physical or psychological stress. Fatigue increases blink duration. Attention to tasks demanding visual activity decreases the blink frequency. Head movements performed in conjunction with eye movements greater than 12-degrees, are an additional measure of workload. For example, workload information would be needed for the proper functioning of a host computer controlling the display formats for the user of an electronics intelligent pilot-vehicle interface design. ##SPC1##