Eye Finding and Tracking System

1. Technical Field:

This invention relates to a system and method for finding and tracking the location of a subject's eyes.

2. Background Art:

Eye finding and tracking devices are employed for many purposes. For example, such devices are often used in vehicle operator drowsiness and intoxication detection systems. When a person is responsible for operating a vehicle, such as an automobile, it is critical that the person be capable of demonstrating basic cognitive and motor skills that will assure the safe operation of the vehicle. Lack of sleep, boredom or consumption of drugs or alcohol can impair a vehicle operator's ability to safely operate the vehicle. Therefore, it is important when designing an impaired driver detection system to continuously evaluate an operator's ability to control the vehicle. Impaired driver detection systems are useful because they avoid or reduce personal injury and property damage by preventing accidents.

An example of a known apparatus for eye finding and tracking which is employed in a vehicle operator drowsiness detection system employs two images of a driver's face. One of the images is obtained by illuminating the driver's face from a first direction, and the other is obtained by illuminating the driver's face from another direction. The two images are processed to detect three-dimensional positions of the driver's eyes. In another, example of a known eye finding and tracking system, an image of a vehicle driver's head is processed to determine the widest portion of the driver's face. From this determination, calculations are performed based on an analysis of the image and assumed geometric relationships of the eyes to the widest portion of the operator's face to define two rectangular regions which should include the operator's eyes.

Eye finding and tracking devices are also useful for other applications. For instance, these devices can be employed in systems designed to monitor the attentiveness of a machine operator, or a person working at a control console such as an air traffic controller. Another example where eye finding and tracking can be useful is in conjunction with an identification system which employs iris pattern matching techniques.

Although systems employing current eye finding and tracking devices work reasonably well for their intended purpose, it is a primary object of the present invention to provide an improved eye finding and tracking system which can determine the location of a subject's eyes within an image of his or her face and thereafter track the changing locations of the subject's eyes in subsequent images with greater accuracy than has been achieved in the past.

The above-described objectives are realized with embodiments of the present invention directed to a system and method for finding and tracking the location of a subject's eyes. The system and method employ an imaging apparatus which produces digital image frames including the face and eyes of a subject. Each digital image frame comprising an array of pixel values representing the intensity of light reflected from the face of the subject. These intensity representing pixel values are located at positions in the array specified by x and y coordinates. An eye position finding and tracking apparatus is used to average the intensity representing pixel values within respective M.sub.x by M.sub.y pixel blocks of a digitized image frame to create elements of plural output matrices. Then, elements of the output matrices are compared to various threshold values. These threshold values are chosen so as to identify which matrix elements correspond to a M.sub.x by M.sub.y pixel block which includes pixel values potentially representing an image of the subject's pupil and at least the portion of the subject's iris.

Specifically, the aforementioned averaging process involves respectively averaging the intensity representing pixel values within three vertically arranged M.sub.x by M.sub.y pixel blocks in the upper left-hand corner of a first digitized image frame to create a first three-element column of a first output matrix. Next, the intensity representing pixel values within three vertically arranged M.sub.x by M.sub.y pixel blocks, each of which is offset to the right by one column of the array in relation to the last averaged pixel block, are respectively averaged to create a new column of the first output matrix. This last step is repeated until the three vertically arranged M.sub.x by M.sub.y pixel blocks which include pixel values from the last column of the array have been averaged to create a last column of the output matrix. The intensity representing pixel values within three more vertically arranged M.sub.x by M.sub.y pixel blocks are then respectively averaged starting at a position on the left-hand side of the array which is offset downward by one row in relation to a previously averaged pixel blocks. This averaging creates a first three-element column of a new output matrix. Next, the intensity representing pixel values within three vertically arranged M.sub.x by M.sub.y pixel blocks each of which is offset to the right by one column of the array in relation to a last averaged pixel blocks are respectively averaged to create a new column of the new output matrix. This step is then repeated until the three vertically arranged M.sub.x by M.sub.y pixel blocks which include pixel values from the last column of the array have been averaged to create a last column of the new output matrix. Finally, the above-described process of averaging pixel blocks which are offset downward by one row is continued the until the three vertically arranged M.sub.x by M.sub.y pixel blocks which include pixel values from the last column and row of the array have been averaged to create a last column of a last output matrix.

The specifics of the aforementioned comparing process includes comparing each element of each output matrix to a threshold range. The threshold range has a lower limit representing the lowest expected average of the intensity representing pixel values within a M.sub.x by M.sub.y pixel block corresponding to an image of the subject's pupil and at least a portion of the subject's iris for the particular illumination condition present at the time the image was produced. Similarly, the upper limit of the threshold range represents the highest expected average of the intensity representing pixel values within the M.sub.x by M.sub.y pixel block corresponding to the image of the subject's pupil and at least the portion of the subject's iris for the particular illumination condition present at the time the image was produced. Once the threshold range has been employed, any output matrix element which both exceeds a lower limit of the threshold range and is less than an upper limit of the threshold range is flagged. Next, threshold values are compared to the average of the intensity representing pixel values within each M.sub.x by M.sub.y pixel block immediately surrounding any pixel block corresponding to a flagged output matrix element. Any previously flagged output matrix element which has any surrounding pixel block which fails to meet the threshold criteria are de-flagged (i.e. de-selected).

The system and method of the present invention can also include provisions for identifying a group of flagged matrix elements having any intensity representing pixel values associated with the M.sub.x by M.sub.y pixel block corresponding to the flagged matrix element which are shared with another M.sub.x by M.sub.y pixel block corresponding to a different flagged matrix element. It is then determined which of these flagged matrix elements are greater than the others, or are positioned in the center of a group of elements each having identical values. Any of the flagged matrix elements which were not determined to be greater, or in the center position, are de-flagged.

Further, the system and method can include a provision for determining if a M.sub.x by M.sub.y pixel block corresponding to a flagged matrix element is within prescribed horizontal and vertical distance ranges from a M.sub.x by M.sub.y pixel block corresponding to another flagged matrix element. The prescribed horizontal distance range has an upper limit corresponding to the maximum expected horizontal eye separation of a subject and a lower limit corresponding to the minimum expected horizontal eye separation of a subject. Similarly, the prescribed vertical distance range has an upper limit corresponding to the maximum expected vertical eye separation of a subject and a lower limit corresponding to the minimum expected vertical eye separation of a subject.

Once the flagged matrix elements have been determined by the above-described processes, any M.sub.x by M.sub.y pixel block corresponding to a flagged matrix element can be designated as an actual subject eye location. However, it is preferred that further steps be taken to confirm the actual eye location, as well as to track the changing position of the subject's eye. Accordingly, it is preferred the system and method of the present invention include a provision for tracking the location of a center of each M.sub.x by M.sub.y pixel block potentially representing the image of the subject's pupil and at least the portion of the subject's iris in each subsequent image frame produced from the imaging apparatus. Further, a provision is preferably included to detect a blink at the location. Such a blink confirms the potential eye location is an actual eye location.

Generally, the aforementioned preferred provision for tracking and detecting a blink involves extracting a rectangular block of pixels centered at a current potential eye location. The current rectangular block is correlated with a block extracted during the aforementioned eye finding process. If the computed correlation coefficient does not exceed a threshold, then a blink has been detected. The center of the M.sub.x by M.sub.y pixel block corresponding to this location where a blink has been detected is then confirmed as an actual eye location.

More specifically, this preferred tracking and blink detecting provision involves first determining the center of the M.sub.x by M.sub.y pixel block associated with the previously-identified potential eye locations. Next, cut-out pixel blocks are selected in a next consecutive image frame produced by the imaging apparatus. Each of these cut-out pixel blocks respectively corresponds to the location of the M.sub.x by M.sub.y pixel block in the immediately preceding image frame which was identified as a potential eye location (i.e. the pixel block potentially representing the image of the subject's pupil and at least the portion of the subject's iris) and includes a number of pixels which surround the identified pixel block. Preferably, the number of pixels equates to the pixels contained in all surrounding M.sub.x by M.sub.y pixel blocks which are adjacent to the identified pixel block. Next, respective matrices of intensity representing pixel values are formed from an area surrounding the center of each M.sub.x by M.sub.y pixel block identified as a potential eye location in the last preceding image frame in which such a center was determined. Preferably, this area encompasses the identified pixel block and all surrounding adjacent pixel block of the same size. The appropriate one of these matrices is then correlated with each element of the associated cutout block. This correlation is performed by sequentially overlaying the center element of the appropriate one of the matrices onto each pixel value of the associated cut-out block starting with the upper left-hand corner, and performing a correlation between the overlaid matrix and the cut-out block for each overlaid cut-out block pixel location. The result of this correlation is to create a matrix of correlation coefficients associated with each cut-out block. These correlation coefficient matrices are compared to a correlation threshold value, and any correlation coefficient matrix element which exceeds the correlation threshold value in each of the correlation coefficient matrices is flagged. The correlation coefficient threshold value is chosen so as to ensure a substantial degree of correlation between an overlaid matrix and an associated cut-out block. If more than one element of a correlation coefficient matrix is flagged then it is determined which of the flagged correlation coefficient elements has the greatest value. This element having the greatest value in the matrix corresponds to the center pixel value of the M.sub.x by M.sub.y pixel block of the potential eye location in the image frame currently being processed. However, if no elements of a correlation coefficient matrix are flagged, then the lack of flagged elements condition in the matrix is noted and the number of consecutive times this condition has occurred is calculated. The aforementioned process is repeated for each consecutive image frame produced by the imaging apparatus to continuously track any movement of the potential eye locations.

The preferred provision for detecting that a blink has occurred in a location identified as potential eye is accomplished by monitoring the number of consecutive image frames which exhibit the aforementioned lack of flagged elements condition for a particular location. If this number does not exceed a prescribed threshold typical of a blink duration, for example approximately seven times in the tested embodiment, before elements of the correlation coefficient matrix corresponding to the location are once again flagged, then a blink has been detected. The center of the M.sub.x by M.sub.y pixel block corresponding the this location where a blink has been detected is then confirmed as an actual subject eye location.

The system and method may also have a provision for assigning a low confidence status to a location identified as a potential eye location whenever the number of consecutive times the aforementioned lack of flagged elements condition is detected for a correlation coefficient matrix associated with the location exceeds a prescribed threshold value, for example 150 times in the tested embodiment. This low confidence status indicates the location is not likely to be an actual subject's eye. Further, a low confidence status is assigned to a potential eye location whenever a blink is not detected at the location for a prescribed number of consecutive image frames (e.g. 150 frames in the tested embodiment). Finally, a low confidence status is also assigned to a location previously identified as an actual eye location whenever the number of consecutive times the lack of flagged elements condition associated with this location exceeds a prescribed threshold (for example 150 times in the tested embodiment) and there is no other location identified as an actual eye location within the previously-described horizontal and vertical distance ranges from the actual eye location. The system is reinitialized whenever all the locations identified as potential or actual eye locations are assigned a low confidence status.

In addition to the just described benefits, other objectives and advantages of the present invention will become apparent from the detailed description which follows hereinafter when taken in conjunction with the drawing figures which accompany it.

The specific features, aspects, and advantages of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where:

FIG. 1 is a schematic diagram showing one embodiment of an eye finding and tracking system in accordance with the present invention.

FIG. 2 is a preferred overall flow diagram of the process used in the eye finding and tracking unit of FIG. 1.

FIG. 3 is a flow diagram of a process for identifying potential eye locations (and optionally actual eye locations) within an image frame produced by the imaging apparatus of FIG. 1.

FIG. 4 is an idealized diagram of the pixels in an image frame including various exemplary pixel block designations applicable to the process of FIG. 3.

FIG. 5 is a flow diagram of a process for tracking eye locations in successive image frames produced by the imaging apparatus of FIG. 1, as well as a process of detecting a blink at a potential eye location to identify it as an actual eye location.

FIG. 6 is a diagram showing a cut-out block of an image frame applicable to the process of FIG. 5.

FIG. 7 is a flow diagram of a process for monitoring potential and actual eye locations and to reinitialize the eye finding and tracking system if all monitored eye locations are deemed low confidence locations.

In the following description of the preferred embodiments of the present invention, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

FIG. 1 depicts an eye finding system embodying the present invention. The system includes an imaging apparatus 10 which may be a digital camera, or a television camera connected to a frame grabber device as is known in the art. The imaging apparatus 10 is located in front of a subject 12, so as to image his or her face. Thus, the output of the imaging apparatus 10 is a signal representing digitized images of a subject's face. Preferably, the digitized images are provided at a rate of about 30 frames per second. Each frame preferably consists of an 640 by 480 array of pixels each having one of 256 (i.e. 0 to 255) gray tones representative of the intensity of reflected light from a portion of the subject's face. The output signal from the imaging apparatus is fed into an eye finding and tracking unit 14. The unit 14 processes each image frame produced by the imaging apparatus 10 to detect the position of the subject's eye and to track these eye positions over time. The eye finding and tracking unit 14 can employ a digital computer to accomplish the image processing task, or alternately, the processing could be performed by logic circuitry specifically designed for the task. Optionally, there can also be an infrared light source 16 positioned so as to illuminate the subject's face. The eye finding and tracking unit 14 would be used to control this light source 16. The infrared light source 16 is activated by the unit 14 whenever it is needed to effectively image the subject's face. Specifically, the light source would be activated to illuminate the subject's face at night or when the ambient lighting conditions are too low to obtain an image. The unit 14 includes a sensor capable of determining when the ambient lighting conditions are inadequate. In addition, the light source would be employed when the subject 12 is wearing non-reflective sunglasses, as these types of sunglasses are transparent to infrared light. The subject could indicate that sunglasses are being worn, such as by depressing a control switch on the eye finding and tracking unit 14, thereby causing the infrared light source 16 to be activated. Alternately, the infrared light source 16 could be activated automatically by the unit 14, for example, when the subject's eyes cannot be found otherwise. Of course, if an infrared light source 16 is employed, the imaging apparatus 10 would be of the type capable of sensing infrared light.

FIG. 2 is an overall flow diagram of the preferred process used to find and track the location of a subject's eyes. At step 202, a first image frame of the subject's face is inputted from the imaging apparatus to the eye finding and tracking unit. At step 204, the inputted image frame is processed to identify potential eye locations. This is accomplished, as will be explained in detail later, by identifying features within the image frame which exhibit attributes consistent with those associated with the appearance of a subject's eye. Next, in step 206, a determination is made as to which of the potential eye locations is an actual eye of the subject. This is generally accomplished by monitoring successive image frames to detect a blink. If a blink is detected at a potential eye location, it is deemed an actual eye location. This monitoring and blink detection process will also be described in detail later. At step 208, the now determined actual eye locations are continuously tracked and updated using successive image frames. In addition, if the location of the actual eye locations are not found or are lost, the process is reinitialized by returning to step 202 and repeating the eye finding procedure.

FIG. 3 is a flow diagram of the process used to identify potential eye locations in the initial image frame. The first step 302 involves averaging the digitized image values which are representative of the pixel intensities of a first M.sub.x by M.sub.y block of pixels for each of three M.sub.y high rows of the digitized image, starting in the upper left-hand corner of the image frame, as depicted by the solid line boxes 17 in FIG. 4. The three averages obtained in step 302 are used to form the first column of an output matrix. The M.sub.x variable represents a number of pixels in the horizontal direction of the image frame, and the M.sub.y variable represents a number of pixels in the vertical direction of the image frame. These variables are preferably chosen so that the resulting M.sub.x by M.sub.y pixel block has a size which just encompasses the minimum expected size of the iris and pupil portions of a subject's eye. In this way, the pixel block would contain an image of the pupil and at least a part of the iris of any subject's eye.

Once the first column of the output matrix has been created by averaging the first three M.sub.x by M.sub.y pixel blocks in the upper right-hand portion of the image frame, the next step 304 is to create the next column of the output matrix. This is accomplished by averaging the intensity representing values of a M.sub.x by M.sub.y pixel block which is offset horizontally to the right by one pixel column from the first pixel block for each of the three aforementioned M.sub.y high rows, as shown by the broken line boxes 18 in FIG. 4. This process is repeated, moving one pixel column to the right during each iteration, until the ends of the three M.sub.y high rows in the upper portion of the image frame are reached. The result is one completed output matrix. The next step 306 in the process is to repeat steps 302 and 304, except that the M.sub.x by M.sub.y pixel blocks being averaged are offset vertically downward from the previous pixel blocks by one pixel row, as depicted by the dashed and dotted line boxes 19 in FIG. 4. This produces a second complete output matrix. This process of offsetting the blocks vertically downward by one pixel row is then continued until the bottom of the image frame is reached, thereby forming a group of output matrices. The purpose of averaging three M.sub.y high blocks at once is to compensate for any geometrical distortion caused by, for example, head movements due to vibration of the vehicle.

In step 308, each element of each output matrix in the group of generated output matrices is compared with a threshold range. Those matrix elements which exceed the lower limit of the threshold range and are less than the upper limit of this range, are flagged (step 310). The upper limit of the threshold range corresponds to a value which represents the maximum expected average intensity of a M.sub.x by M.sub.y pixel block containing an image of the iris and pupil of a subject's eye for the illumination conditions that are present at the time the image was captured. The maximum average intensity of block containing the image of the subject's pupil and at least a portion of the iris will be lower than the same size portion of most other areas of the subject's face because the pupil absorbs a substantial portion of the light impinging thereon. Thus, the upper threshold limit is a good way of eliminating portions of the image frame which cannot be the subject's eye. However, it must be noted that there are some things that could be in the image of the subject's face which do absorb more light than the pupil. For example, black hair can under some circumstances absorb more light. In addition, if the image is taken at night, the background surrounding the subject's face could be almost totally black. The lower threshold limit is employed to eliminate these portions of the image frame which cannot be the subject's eye. The lower limit corresponds to a value which represents the minimum expected average intensity of a M.sub.x by M.sub.y pixel block containing an image of the pupil and at least a portion of the subject's iris. Here again, this minimum is based on the illumination conditions that are present at the time the image is captured.

Next, in step 312, the average intensity value of each M.sub.x by M.sub.y pixel block which surrounds the M.sub.x by M.sub.y pixel block associated with each of the flagged output matrix elements is compared to an output matrix threshold value. In one embodiment of the present invention, this threshold value represents the lowest expected average intensity possible for the pixel block sized areas immediately adjacent the portion of an image frame containing the subject's pupil and iris. Thus, if the average intensity of the surrounding pixel blocks exceeds the threshold value, then a reasonably high probability exists that the flagged block is associated with the location of the subject's eye. Thus, the pixel block associated with the flagged element is designated a potential eye location (step 314). However, if one or more of the average intensity values for the blocks surrounding the flagged block falls below the threshold, then the flagged block is eliminated as a potential eye location (step 316). This comparison concept is taken further in a preferred embodiment of the present invention where a separate threshold value is applied to each of the surrounding pixel block averages. This has particular utility because some of the areas immediately surrounding the iris and pupil exhibit unique average intensity values which can be used to increase the confidence that the flagged pixel block is good prospect for a potential eye location. For example, the areas immediately to the left and right of the iris and pupil include the white parts of the eye. Thus, these areas tend to exhibit a greater average intensity than most other areas of the face. Further, it has been found that the areas directly above and below the iris and pupil are often in shadow. Thus, the average intensity of these areas is expected to be less than many other areas of the face, although greater than the average intensity of the portion of the image containing the iris and pupil. Given the aforementioned unique average intensity profile of the areas surrounding the iris and pupil, it is possible to chose threshold values to reflect these traits. For example, the threshold value applied to the average intensity value of the pixel blocks directly to the left and right of the flagged block would be just below the minimum expected average intensity for these relatively light areas of the face, and the threshold value applied to the average intensity values associated with the pixel block directly above and below the flagged block would be just above the maximum expected average intensity for these relative dark regions of the face. Similarly, the pixel blocks diagonal to the flagged block would be assigned threshold values which are just below the minimum expected average intensity for the block whenever the average intensity for the block is generally lighter than the rest of the face, and just above the maximum expected average intensity for a particular block if the average intensity of the block is generally darker than the rest of the face. If the average intensity of the "lighter" blocks exceeds the respectively assigned threshold value, or the "darker" blocks are less than the respectively assigned threshold value, then the flagged pixel block is deemed a potential eye location. If any of the surrounding pixel blocks do not meet this thresholding criteria, then the flagged pixel block is eliminated as a potential eye location.

Of course, because the output matrices were generated using the previously-described "one pixel column and one pixel row offset" approach, some of the matrices will contain rows having identical elements as others because they characterize the same pixels of the image frame. This does not present a problem in identifying the pixel block locations associated with potential eye locations as the elements flagged by the above-described thresholding process in multiple matrices which correspond to the same pixels of the image frame will be identified as a single location. If fact, this multiplicity serves to add redundancy to the identification process. However, it is preferred that the pixel block associated with a flagged matrix element correspond to the portion of the image centered on the subject's pupil. The aforementioned "offset" approach will result in some of the matrices containing elements which represent pixel blocks that are one pixel column or one pixel row removed from the block containing the centered pupil. Thus, the average intensity value of these blocks can be quite close, or even identical, to that of the block representing the centered pupil. Thus, the matrix elements representing these blocks may also be identified as potential eye locations via the above-described thresholding process. To compensate, the next step 318 in the process of identifying potential eye locations is to examine flagged matrix elements associated with the previously-designated potential eye locations which correspond to blocks having pixels in common with pixel blocks associated with other flagged elements. Only the matrix element representing the block having the minimum average intensity among the examined group of elements, or which is centered within the group, remain flagged. The others are de-selected and no longer considered potential eye locations (step 320).

Any remaining potential eye locations identified in the above process could be considered the actual eye locations with a reasonable amount of confidence, especially, if the more sophisticated and preferred thresholding processes are employed (step 322). If, however, no remaining potential eye locations exist, then the system is reinitialized in step 324 by inputting a next image frame and starting over at step 302.

The confidence level could optionally be further increased by determining if two of the remaining potential eye locations have a certain geographic relationship to each other (step 326). If so, these two potential eye locations are designated as actual eye locations (step 328). An example of a process for determining if two potential eye locations have the desired geographic relationship involves determining if the locations are within a prescribed horizontal and vertical (i.e. to allow for the head being cocked) distance range from one another. These distance ranges represent the difference between the minimum and maximum expected horizontal and vertical eye separation typical of the subject being monitored (e.g. an adult vehicle operator). If the potential eyes are within these distance ranges, then the chances that these locations are actual eyes is increased. Performing the required distance measurements from a digitized image can be accomplished using any appropriate conventional imaging technique. In the case where none of the remaining potential eye locations are within the aforementioned geographic relationship to one another, then the system is reinitialized in step 330 by inputting a next image frame and starting over at step 302.

However, in a more preferred embodiment of the present invention, it is desired that the confidence level be increase even further by observing subsequent image frames in order to detect a blink, i.e. a good indication a potential eye location is an actual eye location. A preliminary determination in this blink detecting process (and as will be seen the eye tracking process) is to identify the image pixel in the original image frame which constitutes the center of the pupil of each identified potential eye location. As the pixel block associated with the identified potential eye location should be centered on the pupil, finding the center of the pupil can be approximated by simply selecting the pixel representing the center of the pixel block. Alternately, a more intensive process can be employed to ensure a the accuracy of the identified pupil center location. This is accomplished by first, comparing each of the pixels in an identified block to a threshold value, and flagging those pixels which fall below this threshold value. The purpose of applying the threshold value is to identify those pixel of the image which correspond to the pupil of the eye. As the pixels associated with the pupil image will have a lower intensity than the surrounding iris, the threshold value is chosen to approximate the highest intensity expected from the pupil image for the illumination conditions present at the time the image was captured. This ensures that only the darker pupil pixels are selected and not the pixels imaging the relatively lighter surrounding iris structures. Once the pixels associated with the pupil are flagged, the next step is to determine the geographic center of the selected pixels. This geographic center will be the pixel of the image which represents the center of the pupil, as the pupil is circular in shape. The geographic center of the selected pixels can be accomplished in a variety of ways. For example, the pixel block associated with the potential eye location can be scanned horizontally, column by column, until one of the selected pixels is detected within a column. This column location is noted and the horizontal scan is continued until a column containing no selected pixels is found. This second column location is also noted. A similar scanning process is then conducted vertically, so as to identify the first row in the block containing a selected pixel and the next subsequent row containing no selected pixels. The center of the pupil is chosen as the pixel having a column location in-between the noted columns and a row location in-between the noted rows. Any noise in the image or spots in the iris, which are dark enough to be selected in the aforementioned thresholding step, can skew the results of the just-described process. However, this possibility can be eliminated in a number of way, for example by requiring there be a prescribed number of pixel columns or rows following the first detection before that column or row is noted as the outside edge of the pupil.

A blink at a potential eye location represents itself as a brief period where the eyelid is closed, e.g. about 2-3 image frames in length based on an imaging system producing about 30 frames per second. This would appear as a "disappearance" of a potential eye at an identified location for a few successive frames, followed by its "reappearance" in the next frame. The eye "disappears" from an image frame during the blink because the eyelid which covers the iris and pupil will exhibit a much greater average pixel intensity. Thus, the closed eye will not be detected by the previously-described thresholding process. Further, it is noted that when the eye opens again after the completion of the blink, it will be in approximately the same location as identified prior to the blink if a reasonable frame speed is employed by the imaging system. For example, a 30 frames per second rate is adequate to ensure the eye has not moved significantly in the 2-3 frames it takes to blink, Any slight movement of the eye is detected and compensated for by a correlation procedure to be described shortly.

The subsequent image frames could be processed as described above to re-identify potential eye locations which would then be correlated to the locations identified in previous frames in order to track the potential eyes in anticipation of detecting a blink. However, processing the entire image in subsequent frames requires considerable processing power and may not provide as accurate location data as the preferred method of the present invention. FIG. 5 is a flow diagram of the preferred eye location tracking and blink detection process used primarily to identify and track actual eye locations among the potential eye locations identified previously (i.e. steps 302 through 320 of FIG. 3). This preferred process uses cut-out blocks in the subsequent frames which are correlated to the potential eye locations in the previous frame to determine a new eye location. Processing just the cutout blocks rather than the entire image saves considerable processing resources. The first step 502 in the process involves identifying the aforementioned cut-out blocks within the second image frame produced by the imaging system. This is preferably accomplished by identifying cut-out pixel blocks 20 in the second frame, each of which includes the pixel block 22 corresponding to the location of the block identified as a potential eye location in the previous image frame, and all adjacent M.sub.x by M.sub.y pixel blocks 24, as shown in FIG. 6. Next, in step 504, a matrix is created from the first image for each potential eye location. This matrix includes all the represented pixel intensities in an area surrounding the determined center of a potential eye location. Preferably, this area is bigger than the cut-out block employed in the second image. For example, an area having a size of 100 by 50 could be employed. The center element of each matrix (which corresponds to the determined center of the pupil of the potential eye) is then "overlaid" in step 506 on each pixel in the associated cut-out block in the second image frame, starting with the pixel in the upper left-hand corner. A correlation procedure is then performed between each matrix and the overlaid pixels of its associated cutout block. This correlation is accomplished using any appropriate conventional matrix correlation process. As these correlation processes are known in the art, no further detail will be provided herein. The result of the correlation is a correlation coefficient representing the degree to which the pixel matrix from the first image frame corresponded to the overlaid position in the associated cutout block. This process is repeated for all the pixel locations in each cut-out block to produce a correlation coefficient matrix for each potential eye location. In step 508, a threshold value is compared to each element in the correlation coefficient matrices, and those which exceed the threshold are flagged. The flagged element in each of these correlation coefficient matrices which is larger than the rest of the elements corresponds to the pixel location in the second image which most closely matches the intensity profile of the associated potential eye location identified in the first image, and represents the center of the updated potential eye location in the second image frame. If such a maximum value is found, the corresponding pixel location in the second image is designated as the new center of the potential eye location (step 510). The threshold value was applied to ensure the pixel intensity values in the second frame were at least "in line" with those in the corresponding potential eye locations in the first image. Thus, the threshold is chosen so as to ensure a relatively high degree of correlation is observed. For example, a threshold value of at least 0.5 could be employed.

If none of the correlation coefficients exceeded the correlation threshold in a given iteration of the tracking procedure, then this is an indication the eye has been "lost", or perhaps a blink is occurring. This "no-correlation" condition is noted. Subsequent frames are then monitored and the number of consecutive times the "no-correlation" condition occurs is calculated in step 512. Whenever, a no-correlation condition exists from a period of 2-3 frames, and then the potential eye is detected once again, this is indicative of a blink. If a blink is so detected, the status of the potential eye location is upgraded to a high confidence actual eye location (step 514). This is possible because an eye will always exhibit this blink response, and so the location can be deemed that of an actual eye with a high degree of confidence. The eye tracking and blink detection process (of FIG. 5) is repeated for each successive frame generated by the imaging apparatus with the addition that actual eye locations are tracked as well as the remaining potential eye locations (step 516). This allows the position of the actual and potential eye locations to be continuously updated. It is noted that the pixel matrix from the immediately preceding frame is used for the aforementioned correlation procedure whenever possible. However, where a no-correlation condition exists in any iteration of the tracking process, the present image is correlated using the pixel matrix from the last image frame where the affected eye location was updated.

Referring now to FIG. 7, if a potential eye location does not exhibit a blink response within 150 image frames, it is still tracked but assigned a low confidence status (i.e. a low probability it is an actual eye location) at step 702. Similarly, if a potential eye location becomes "lost" in that there is a no-correlation condition for more than 150 frames, this location is assigned a low confidence status (step 704). Further, if a blink has been detected at a potential eye location and its status upgraded to an actual eye location, but then this location is "lost", its status will depend on a secondary factor. This secondary factor is the presence of a second actual eye location having a geometric relationship to the first, as was described previously. If such a second eye location exists, the high confidence status of the "lost" actual eye does not change. If, however, there is no second eye location, then the "lost" actual eye is downgraded to a low confidence potential eye location (step 706). The determination of high and low confidence is important because, the tracking process continues for all potential or actual eye locations only for as long as there is at least one remaining high confidence actual eye location or an un-designated potential eye location (i.e. a potential eye location which has not been assigned a low confidence status) being monitored (step 708). However, if only low confidence locations exist, the system is re-initialized and the entire eye finding and tracking process starts over (step 710).

While the invention has been described in detail by reference to the preferred embodiment described above, it is understood that variations and modifications thereof may be made without departing from the true spirit and scope of the invention.