Intelligent eye

[0001] Not applicable.

[0002] Not Applicable

[0003] Not Applicable

[0004] Machine human interactions have got more efficient over the last decade. However, while visual machine outputs have evolved substantially with user interfaces that mimic 2-d real world views, there has not been a corresponding advance in machine inputs for human interface. The background art describes a variety of input devices such as mice, joy sticks, and other hand and voice operated devices, however there is nothing in the background art that mimics the human interaction with the natural environment with vision and then extends it to what a virtual space can provide in terms of selectable points in the virtual scape.

[0005] Although input devices such as mice and joy sticks have been very usefull in many applications, the present invention departs from the prior art in providing unique new approach for man-machine interaction.

[0006] In view of the prior art, it would be useful to have a device that can provide direct visual instruction to machines given visual stimuli using the information in the direction and depth of gaze as the motion of the eye lids which is closely related to ocular control by the brain. The present invention comprises such a device.

[0007] Some of the objects and advantages of the present invention are that the visual stimuli from computers or other machines, can be directly responded to by the human user, with navigation to the point of interest and action by the eyes and the eyelids for quick interaction with computers and other machines with visual output. Yet another advantage is that the mental processing for interaction is reduced as there in no need for hand eye coordination or voice recognition but a direct use of the eyes and their appendages in feedback to the machine.

[0008] FIG. 1 illustrates an embodiment of the present invention that has a single eye camera and video display encased together with a reflective surface for projection to the eye. The controller unit rests at the back of the head. The embodiment has a microphone and a ear phone.

[0009] 101-Reflector for virtual screen and eye projection to camera

[0010] 102-Housing for camera and video projection unit

[0011] 103-Microphone

[0012] 104-Earphone

[0013] 105-Controller unit-broadband wireless/video decompression/camera interpretation

[0014] 106-Headband

[0015] The quality of Man-machine interaction has improved sharply in the last two decades. Computer visual outputs have been made to mimic the natural stimuli that the human user has observed and has learned to respond to through a lifelong learning experience in interactions with the natural world. However, while machine inputs from the human user have developed substantially in the last two decades, there has remained a significant gap in the visual process of human command to the machine that has a counterpart in the natural world paradigm. In the natural visual interaction of the human with the real world, the human can view and select stimuli by navigating the eyes and the remaining anatomy appropriately. While body movements to this end are effective, the most economical in terms of mental processing is the movement of the eye and appendages (including the eye lids for removing and reinstating stimuli by closing and opening). The control of the eyes and the eyelids by the nervous system are closely related. Therefore the use of the eye and the eyelids in machine input results in much faster navigation and selection of stimuli due to the reduction of the mental processing required, the speed of accurate movement of the eye based on the natural visual selection process of stimuli presented in the field of view, and finally the ease of 3-dimensional navigation. The present invention meets this need. It comprises the machinery to process human eye and eye appendage movements, to navigate and select stimuli as desired by the human user.

[0016] Some embodiments of the present invention may consist of one or more cameras that monitor the eyes and the region around the eyes of the user in real time under ambient light conditions and/or infrared/visible supplementary light. Particular areas of interest for embodiments that monitor a single eye are the position of the pupil and cornea of the eye relative to the overall eye perimeter, the size of the pupil, the position of the eyelids. The monitoring of a single eye is adequate for navigation in a 2-dimensional space. However, when the field is extended to a three dimensional space for navigation, it is necessary to have one or more cameras monitoring each of the two eyes in the stated manner. In addition the present invention has software algorithms and/or hardware to assess the relative positions of the pupil and cornea of the eyes each with a reference to the fixed parts of the face or the perimeters of each of the eyes. This allows measurement of depth of focus of the eyes an d will therefore be able to estimate the zoom-in and the zoom-out of the user as he/she navigates the virtual space stimuli of the machine visual interface. The cameras may be mounted to face the eye or in an alternative architecture utilize the mirror or partial mirror of a reflecting display system thereby allowing the mounting of the camera to face the reflecting mirror and in the vicilnity of the light source of the display system.

[0017] There is extensive background art that supports the implementation of the present invention including the the techniques and methods for object recognition and object tracking in machine vision and computer image processing literatures. There is in the present invention a calibration phase where the motion to be identified and recognized is defined. The object recognition and tracking algorithms are at this time set up with the required parameters for the specific users to the users preference. The algorithms are then able to estimate the likelihood of signals. Suitable filters (some defined below) whose implementation is available in the background art, are used to maximize the signal to noise ratio of the information that the cameras detect. Once a signal is created as the output from a object tracking algorithm, it can be used very much like the signal in a computer mouse or in a computer joy stick as an input signal.

[0018] The present invention uses the motion of the eye-lids in the wink or blink operation (of the left or right or both eyes) to signal events that the user wishes to transmit to the computer. (this may correspond to the click of a mouse button). There are a number of possible such events or signals such a single blink or wink of one eye (useful if both eyes are monitored) and of course multiple blinks or winks in quick succession. The timing and speed of these actions to be recognized by the present invention are adjustable by the user during a calibration process for the present invention defined below. (single or multiple blink and wink signals by may also be called Vclicks herein).

[0019] Yet another feature of the present invention is the selection and deselection of the screen (on/off) with an object outside the screen or virtual screen. Here the user would visually select by looking at a predefined position outside the screen or virtual screen field of view and provide a predefined signal such as a Vclick to select or deselect the screen. This is particularly useful for headset screens that may be used part time only and are transparent within the field of view when not in use.

[0020] The cursor in some embodiments of the present invention is user definable to minimize the intrusion to the field of view. One such embodiment may have a point and a fine blinking circle. Moreover, during the calibration phase the user may select a suitable convention for cursor interaction as detailed below.

[0021] In the case of 2-dimensional navigation there is limited need for a two eye version of the present invention as the movement of a single eye is adequate for identifying the focus information of the user in 2 dimensions. Some embodiments of the present invention may use the convention of having the cursor following the gaze of the user. Selection signals are defined as single eye or double eye Vclicks. Yet other embodiments use another convention where the cursor does follow the gaze but where the cursor is selectable by a predefined Vclick sequence. For example is some embodiments a single Vclick could enable the cursor and a second Vclick after enabling it would select the object on the field of view. Further movement of the eyes in these embodiments will move the selected object as desired and when in the correct destination position the cursor may be deselected with a predetermined pattern of VClicks. The cursor at this time could either follow the eye or be deselected to be reselected later.

[0022] In some embodiments the the movement of the eye to the edge of the virtual screen/screen will activate a scrolling signal that will scroll the screen further to the direction of the edge that the gaze of the user has reached. Scrolling can be stopped by the gaze moving away from the above edge of the screen.

[0023] 3-Dimensional navigation in yet other embodiments of the present invention will in addition estimate the depth of gaze of the user. While it is well recognized that two eye movement should be able to perceive depthm the present invention defines for the first time, a practical approach to achieve this considering the process needed for 3-D navigation. Some of these embodiments use the relative focal direction of the two eyes each eye's focal direction being determined by the position of the pupil and/or the cornea relative to the periphery of the eye. In the case of a single (mono) image prohjected both eyes, the user may zoom-in by relaxed gazing or relazing the eyes to what is substantially infinite focal length. All that needs to be determined is that the eyes are focusing at a point behind the screen/ virtual screen. This event in some embodiments of the present invention will zoom the field inwards as long as the focus of the gaze remains behind the screen/virtual screen. Notably the gaze of the user can be focused behind the screen/virtual screen and still have a direction. Such a direction will be detected by the present invention to allow directional zooming in the direction of the gaze that the user chooses. As soon as the user focuses on the content of the screen, the zoom process slows to a standstill and remains at the standstill view of the image. The present invention may have a user settable time constant to the slowing down and speeding up of the zoom as the user changes his/her gaze from behind the screen/ virtual screen to focus on the screen and vise versa. This time constant will enable the user to periodically check the distance he/she has navigated in the z-direction (zoom direction) and continue or stop the zoom. A similar approach is used in some embodiments for zooming out. Here the depth of gaze is reduced by the user to stare at a point in front of the screensay his her nose. The relative positions of the eyes that are estimated will inform the present invention that the eyes are focusing in front of the screen/virtual screen and the direction of the gaze, and will commence a zoom-out process in the direction of the gaze. Moreover, a similar approach may be adopted for the zoom-out as in the zoom-in, where the user may check the level of zoom out periodically by focusing on the screen/virtual screen, which will slow the zoom eventually to a standstill. The time constant for this transition as the user switches from the point of focus in front of the screen to the point of focus on the screen/virtual screen will determine the rate of slow down that is preferred by the user and may be set or modifies on a setup panel for this input device that is the present invention. In the case of stereo images projected to the two eyes, there will be greater resolution in direction for zoom movements, as the eyes will focus on the desired object at the depth plane desired. Therefore the zoom process can target this point based on the relative direction of focus of the eyes. Other factors such as time constant for the zoom etc. can follow the same design as in the mono-image case for two eye screens or a single screen for both eyes.

[0024] The actual algoriths for zooming in and out in virtual space may follow the extensive background art on such approaches for zooming and moving within 3-d virtual spaces as in video games. Open GL algoriths is one such source. Yet another feature of the present invention is a set of filters that filter out natuaral behavior of the user that are not intended for interpretation by the present invention as events for input to the machine. Such filters in some embodiments of the present invention include a neutral-gaze filter which filters out the user's natural gaze while relaxing or thinking. This filter will be setup at calibration time by observing the user when relaxing or in thought and defines the set of possible relative and/or absolute eye positions that are encountered in this mental state. When the present invention is operational the cursor is temporarily deactivated when the user's gaze enters this set of possible gaze parameters. This temporary deactivation is cancelled when the gaze reverts to the plane of the screen/virtual screen. Here again some embodiments may have time constants to enter the natural gaze state and to get back into the active state at the end of the natural gaze period.

[0025] Yet another filter is the natural blink filter. This is set up again at calibration time to observe the user and the eye lid movements of the user when not working with the machine. Once the present invention has observed the nature of movement in and the time pattern of the eye lids in natural blinks, these embodiments of the present invention at operation time will estimate the probability of an observed stimulus of a blink to be a signal or otherwise part of the natural blink pattern and accordingly use the signal as an input signal or not. The user may define patterns of blinks/winks that are clearly different from the natural pattern to increase the signal to noise tatio in this process and thereby improve the communication with the present invention as an input device. There is extensive background art in the literature in pattern recognition an signal processing that cover the basic algoriths that may be used to construct the present invention.

[0026] While some embodiments of the present invention will be mounted on headsets with screens/virtual screens, and thereby rigidly connected to the screens/virtual screens, they will need to estimate only the positions of the pupil/cornea relative to the perimeter of the eye. However, in embodiments that use fixed cameras and fixed screens/virtual screens, there is the additional need to estimate the position of the head as well and the eyes within the head. These embodiments use additional object and motion detection algoriths that process the camera signals from the face as well to produce the required input signals.

[0027] In embodiments that use a headset, it will be normal to have a virtual or real headset mounted display (well covered in the background art) and for the purpose of voice recognition a microphone (well covered in the background art). These together will provide a powerful embodiment of the present invention that allows the user to navigate and command using the present invention and and provide additional instruction to the computer using voice recognition. Through the microphone.

[0028] A yet more powerful and convenient embodiment will use wireless communication technologies such as blue tooth and enhancements thereof in th future, to provide the video and audio signals to the headset, and to provide the channels to input commands from the present invention and the microphone to the computer.

[0029] Another aspect of the present invention is the visual field that is used for navigation in 3-dimensions, these may be natural spaces as perceived by the user as in video games that mimic natural scapes, or virtual spaces that are created for specific purposes. Some embodiments of the present invention may include one of the following virtual scapes as alternatives to natural scapes:

[0030] The switching space: In these embodiment sthe zoom or z-direction navigation switches from one plane to another in discrete steps. Each of these plases represent an application that is open on the machine, or an alternative view within an application, such as in a CAD program. It is therefore possible for the user to switch views by navigating forward or backwards. These embodiments may also have a circular arrangement of the planes so that navigation in one direction will circulate through all the available planes.

[0031] n-D virtual space: In these embodiments, navigation is within 3 dimensional space where objects are placed in the space using a convention for z-direction distribution as well as the usual x-y plane distribution. For example directory structures may first expand in the x direction then in the y direction and finally in the z direction to give a more intuitive tree structure in 3 dimensions. Such conventions can be made made to be even more intuitive by making the synthetic space to be non-linear wherein the areas where the user is browsing is expanded and the remaining space is contracted to give a greater level of resolution in the region of interest of the user. Finally, n-dimensional spaces may be navigated by ordering them as a sequential set of nested 3-D spaces where the user can select points or nodes in each 3-D space to enter the next 3-D space. Selection and navigation can be effectively done in this environment by the present invention.

[0032] Finally the object recognition programs may include specific information about the user's eyes such as the iris or the retinal image that becomes a signature for logging into the system when the user commences use fothe present invention.

[0033] The Calibration and set up of the present invention includes the learning pocesss for the object recognition and motion detection algorithms using the user's data. The system will begin with default values and refine them to the characteristics of the user. This process will synthesize the actions or sets of actions of the user and map them into the desired actions of the user. Several of these actions are listed below. Howeever the process of calibration may include other factors as well.

[0034] 1. Neutral-Gaze filters are calibrated by the calibration program requesting the user to perform one or more gaze patterns that that (s)he performs when in thought or when (s)he wants to focus on other maters besides the visual display. Onces one or more of these gaze data sets are processed the parameters of these gaze patterns are maintained and compared with real time data from the user for actions as noted.

[0035] 2. Blink pattern filters are calibrated by the calibration program requesting the user to simply use the present invention with no visual stimuli for a period of time. This can include several periods that the user is relaxed and focusssed (as blink patterns could vary depending on mental conditions.). Once one or more of these blink data sets are processed the parameters of these blink patterns are maintained and compared with real time data to filter out this information in real time so that the signal to noise ratio of the data used to drive the input is raised by avoiding natural blink patterns being interpreted as Vclicks.

[0036] b 3. Gaze definition for Zoom-in and Zoom-out. Here the user inputs his/her preferred gaze sets for each of these interpreted inputs. Each of the zoom-ins and zoom-outs can have multiple directions and the user can define these to be the prefereed gazes. Notably some embodiments of the present invention will flag up conflicts between the natural gaze and the chosen zoom out gazes so that the user may choose other zoom out gazes to minimize error in operation. Moreover, the Zoom-in and zoom-out will usually have time constants assigned to each of them so that when the user switches from focus on the scrren to zoom and vise versa, the zooming of the field will slow and then stop giving the user time to focus periodically on the field of the screen while zooming in either direction. In the case of a stereo field of view, with two screen for each of the eyes, 3-D objects may be used in the calibration process for the eyes to focus on for the convergence measurements. This may be done at different depths at different locations in the x-y field.

[0037] 4. The present invention needs to have a calibrated 2-d field of view so that movements in the eye can be interpreted as changes in the focus of the eye to different points on the screen/virtual screen. Therefore the calibration process includes stimuli that are places on the screen/virtual scrren that the user looks at and the camera notes the position of the eyes and thereby maps the eye positions for each of the screen stimuli and thereby permits interpolation for estimating the exact focus of the eye from the position of the eye.

[0038] 5. Vclicks need to be defined. The user may choose a set of several signals with winks and/or blinks to signal different input signals to the computer. (very much like left, right and double clicks on the mouse). This is done at calibrate time.

[0039] 6. The user may define the convention for selecting and deselecting the cursor when the present invention is in use. For example the one convention would be where the cursor follows the users gaze on the screen/virtual screen. In others the user is able to deselect the cursor with Vclicks and then reselect the cursor when desired again with Vclicks.

[0040] 7. The user may define a position where he/she may look outside the screen/virtual screen and implement a Vclick to deactivate and reactivate the screen/virtual screen.

[0041] 8. the user may define regions on the the edge of the screen/virtual screen where a gaze would imply a command to scroll in the direction of the gaze.

[0042] Notably all the actions that are defined in the present invention may be implemented using algorithms that are currently in use for other input devices, in algorithis in the computer vision and image processing literature, in algorithosn in the signal processing literature and in the stochastic control literature.

[0043] The following is a detailed description of some of the components of this embodiment. The preferred embodiment of the present invention comprises a headset that has a single camera mounted besides a light source of a virtual screen, for a single eye virtual screen system. Both the camera and the light source are placed outside the field of view required of the virtual screen system. The reflective screen of the virtual screen system will also be reflective for light that falls on the eye and reflected back to the camera. The camera orientation will be such that it can see the eye and its immediate surroundings including the eye lids. The camera and the light source may be mounted in the same housing. The signals from the camera are conducted to a microprocessor system that analyses the data to provide the as output the inputs that will go into the computer. This microprocessor unit will need to be calibrated to the user for some of the functions as will the computer for others. Inputs to the visual display and the outputs from the microcontroller are conducted by wireless infrastructure such as Bluetooth or extensions thereof to relevant computer input port.

[0044] Additional embodiments to the preferred embodiment has a two eye system with the virtual screen system duplicated and each of the two eyes monitored for movement. The microprocessor system will take as inputs the signals from both cameras and output in addition, the gaze depths as required for 3-d browsing. The microprocessor system will also interpolate the inputs from the two cameras to better estimate the direction of focus of the user on the virtual screen.

[0045] Another additional embodiment has microphone and earphones as well to provide a complete interface for the computer system.

[0046] The microprocessor based controller may be mounted on the headset prefereably on the headband that bends down to the base of the skull, so that the weight of the module may be supported at that point largely by the neck. In this single eye embodiment the headband extends only to behind the ear opposite the ear with a ear piece and the side where the display and camera are mounted. Whereas in the double eye embodiment there are two display booms that include the camera extending from the ear pieces. The microphone boom extends from the ear piece of the headset as is the common design. In both cases the headset may rest on the top of the ears and the base of the skull, on the neck.

[0047] In an alternative embodiment to the preferred embodiment, the cameras that are used to record and interpret movement of the eye and eye region are mounted on the outer side of the field of view of the user, pointing directly at the eye, rather than use the reflective properties of a virtual screen to get the image of the eye as in the preferred embodiment.

[0048] In another alternative embodiment the headset is connected by wires with the required input and output signals rather than use a wireless communication system.