Method and apparatus of wearable eye pointing system

[0001] The present application for patent claims priority under 35 U.S.C. §119 from U.S. provisional patent application Ser. No. 62/039,071, entitled “Method and Apparatus of Wearable Eye Pointing System,” filed Aug. 19, 2014, the entire subject matter of which is incorporated herein by reference.

[0002] The described embodiments relate to interactive, wearable computing systems.

[0003] The advent of mobile computing devices such as smart phones and tablet computers has enabled users to perform computing functions away from their desks and offices. Users operate mobile computing devices while they are walking, driving, etc. However, traditional mobile computing devices employ a fixed display and touch screen command interface. Operating these mobile computing devices while moving can create a dangerous situation as the users hands and eyes should be focused on their mobility task (e.g., walking or driving), rather than operating their mobile computing device.

[0004] To reduce the risks associated with operating a mobile computing device while moving, hands-free command interfaces based on voice recognition have been developed. Voice recognition allows the user to operate the mobile computing device while their hands and eyes are focused on more important tasks. However, compared to standard point-and-click operation, voice recognition is slow, unreliable, and limited to a simple, lengthy string of questions. If the user has to listen to a string of selection choices and then speak to the machine, the interface often frustrates the user and is left unused.

[0005] Alternative hands-free command interfaces based on tracking eye movement have been explored. Human eyes can move much faster than hands. As a method of pointing, eyes work much faster than a computer mouse, and many orders-of-magnitude faster than voice recognition. However, it is recognized that while eyes can move quickly, the movements are often spontaneous and even erratic. Therefore, eye movements alone are not a reliable command input.

[0006] Nonaka introduces an interface to a fixed computer system that combines the simultaneous tracking of the direction of gaze and head gesture as a command input. Additional details are presented in “Communication Interface with Eye-Gaze and Head Gesture Using Successive DP Matching and Fuzzy Inference,” by Hidetoshi Nonaka, Journal of Intelligent Information Systems, 21:2, 105-112, 2003, the content of which is incorporated herein by reference in its entirety. The display of the computing system subject to control is fixed in the environment. To resolve a fixed gaze point, the system simultaneously tracks both eye and head movement to identify whether the head and eye movements are coordinated by the vestibulo-ocular reflex (VOR). If VOR is detected, the assumption is that the gaze is fixed and the head gesture is deliberate. In other words, evidence that the user is engaged in a VOR is used as an indicator of a human decision. Although this approach is suitable when the display of the computing system is fixed in the environment, it is unworkable when the display of the computing system subject to control is wearable.

[0007] In wearable display systems, a fixed gaze line indicates focus on an object in the environment, rather than an image projected by the display. Thus, detecting the VOR reflex during head movement cannot serve as an effective interface between the user and the wearable display system. Thus, improvements in the interface between a wearable computing system and its user are desired.

[0008] Methods and systems for providing a reliable hands-free interface between a user and a computing device are presented herein. The interface enables the user to express his/her intention by navigating a variety of selection options in a fast, reliable manner. Such a hands-free interface is suitable for controlling wearable computers, phones, robots, and assistive devices.

[0009] The interface system includes a gaze tracking system, display system, and head motion tracking system all attached to a frame that is mountable to the head of a user. In one aspect, eye movements are tracked by the gaze tracking system to determine when the gaze of a user has settled on a particular image displayed by the head-mounted display system. After a fixed gaze has been identified, it is determined whether the user is moving his/her head in a confirmatory head gesture.

[0010] The user selection is verified by determining whether the gaze of the user remains fixed to the selected image while executing the confirmatory gesture. This is accomplished by determining that the movement of the eye relative to the head of the user remains below a threshold value while the user is executing the confirmatory gesture. This ensures that the gaze of the user is focused on a particular selection option displayed on the head mounted display. Thus, the eye pointing event is verified by discerning whether the user is focused on an image projected by the display or on an object in the surrounding environment during the confirmatory head gesture. To maintain eye focus on the image displayed by the head mounted display while moving the head in a confirmatory gesture requires that the user consciously suspend the vestibulo-ocular reflex (VOR). Hence, when substantially no movement of the eye is detected relative to the moving head, it is a clear indication that the user has consciously decided to suspend his/her VOR reflex for purposes of communicating his/her intent. This physiological switch is both deliberate and detectable, thus minimizing false confirmation events.

[0011] The foregoing is a summary and thus contains, by necessity, simplifications, generalizations and omissions of detail. Consequently, those skilled in the art will appreciate that the summary is illustrative only and is not limiting in any way. Other aspects, inventive features, and advantages of the devices and/or processes described herein will become apparent in the non-limiting detailed description set forth herein.

[0019] Reference will now be made in detail to background examples and some embodiments of the invention, examples of which are illustrated in the accompanying drawings.

[0020] Methods and systems for providing a reliable hands-free interface between a user and a computing device employing a head-mounted display are presented herein. The interface enables the user to express his/her intention by navigating a variety of selection options in a fast, reliable manner. Such a hands-free interface is suitable for controlling wearable computers, phones, robots, and assistive devices.

[0021] In one aspect, eye movements are tracked to determine when the gaze of a user has settled on a particular direction. After a fixed gaze direction has been identified, it is determined whether the user is moving his/her head in a confirmatory head gesture. Thus, an eye pointing event is confirmed by a subsequent confirmatory head gesture.

[0022] In a further aspect, the user selection is verified by determining whether the gaze of the user remains fixed to a particular portion of the head-mounted display while executing the confirmatory gesture. This is accomplished by determining that the movement of the eye relative to the head of the user remains below a threshold value while the user is executing the confirmatory gesture. This ensures that the gaze of the user is focused on a particular selection option displayed on the head mounted display. In other words, when the eye tracks the image on the head mounted display while the head is moving, the eye does not move relative to the head. This also ensures that the gaze of the user is not focused on an object in the surrounding environment. If the eye is not moving relative to the head while the head is moving, the eye is not tracking an object fixed in the environment. Thus, the eye pointing event is verified by discerning whether the user is focused on an image projected by the display or on an object in the surrounding environment during the confirmatory head gesture.

[0023] Moreover, the user must consciously suspend the vestibulo-ocular reflex (VOR) to maintain eye focus on the image displayed by the head mounted display while moving the head in a confirmatory gesture. Hence, when substantially no movement of the eye is detected relative to the moving head, it is a clear indication that the user has consciously decided to suspend his/her VOR reflex for purposes of communicating his/her intent. This physiological switch is both deliberate and detectable, thus minimizing false confirmation events.

[0024] FIG. 1 is a diagram illustrative of an embodiment of a wearable computing system mounted to the head of user 10. The wearable computing system includes a gaze tracking system 103, a display system 102, a head motion tracking system 104, computing system 110, and wireless communications system 150 all mounted to a frame 101. Frame 101 is mounted to the head of user 10. In this manner, gaze tracking system 103, display system 102, head motion tracking system 104, computing system 110, and wireless communications system 150 are all fixed in position relative to the head of user 10. In some embodiments, frame 101 is an eye-glass type frame. In some embodiments, the head motion tracking system 104 includes an accelerometer and gyroscopic sensor combination suitable for tracking the motion of the head over time. In some embodiments, the display system is a wearable display system such as Google Glass™.

[0025] In one example, gaze tracking system 103 includes an eye-mark recorder system, such as the eye-mark recorder system model number EMR-7 manufactured by NAC Image Technology, Incorporated, Simi Valley, Calif. (USA). In this example, the eye mark recorder includes a head mounted eye-tracking unit and camera controller. Eye movement is measured by an infrared corneal reflex method in the eye tracking unit at a frequency of 30 frames per second. The camera controller embeds the measured data in the vertical blanking period of the NTSC composite video signal of the user's scope image. The measured data are extracted from the composite video signal as 9-bit data for the horizontal direction and 8-bit for the vertical direction, for example, by computing system 110. In one example, head motion tracking system 104 includes a 3D motion tracking system such as the 3D motion tracking system, 3SPACE FASTRAK, manufactured by Polhemus, Incorporated, Colchester, Vt. (USA). In this example, FASTRAK includes a system electronics unit, a transmitter, and a receiver. The motion of the receiver, i.e., the motion of the head, is measured by the system electronics unit based magnetic field measurements. The measured head motion in six degrees of freedom (e.g., x, y, z, Rx, Ry, Rz) is communicated, for example, to computing system 110.

[0026] FIG. 2 depicts a block diagram illustrative of wearable computing device 100 including a head mountable display 102, a head mountable gaze tracking device 103, and a head mountable head motion tracking device 104, such as those depicted in FIG. 1, in one exemplary operational scenario. Wearable computing device 100 also includes a computing system 110 and wireless communications system 150 that may be head mountable as described with reference to FIG. 1, or alternatively, may be carried by a user, but not head mounted.

[0027] As depicted in FIG. 2, computing system 110 communicates signals 105 to display system 102 that cause display system 102 to present one or more selectable images within the field of view of the user. In general, display system 102 may be any head-mountable display system that displays images within the field of the user such that the images move with the head of the user. In other words, the images remain in the same location with respect to a coordinate frame (e.g., coordinate frame XYZ) attached to the head of the user. As depicted in FIG. 3 in one embodiment, display system 102 projects one or more selectable images 111A-D onto the retina of the eye of a user such that the user perceives the one or more images in the field of view of the user.

[0028] As depicted in FIG. 2, computing system 110 receives signals 106 from gaze tracking system 103 that indicate an amount of movement of an eye of the user relative to the head of the user. Based on these signals, computing system 110 determines whether the gaze of the user is fixed on a selected image for a predetermined period of time. For example, as depicted in FIG. 3, gaze tracking system 103 observes the movement of the eye 11 of the user 10 and generates signals 106 that indicate the direction of gaze 113 of the user at a particular moment in time. FIG. 6 depicts a representative example of signals 106 indicative of gaze direction over time. Computing system 110 determines whether the direction of gaze 113 remains fixed on a particular image 111A within a small tolerance band, ΔGAZE, for a predetermined period of time, TGAZE. TGAZE is selected such that exploratory eye movements can be separated from focused gaze (e.g., a period of time greater than 500 milliseconds). ΔGAZE is selected to allow for signal noise and spurious eye movements that are not indicative of a change of focus of eye gaze.

[0029] If computing system 110 determines that the direction of gaze 113 remains fixed on selected image 111A for a period of time, T_GAZE, the computing system 110 begins to analyze signals 107 received from head motion tracking system 104. Head motion tracking system 104 generates signals 107 that indicate the motion of the head of the user. Computing system 110 determines whether the motion of the head of the user is consistent with a predetermined confirmatory head gesture (e.g., a head nod, a head shake, a head tilt, etc.) based on signals 107 during a period of time after T_GAZE.

[0030] If computing system 110 determines that the motion of the head of the user is consistent with a predetermined confirmatory head gesture, computing system 110 analyzes signals 106 during the period of confirmatory head gesture to verify that the direction of gaze remains fixed on the selected image during the confirmatory head gesture. For example, as depicted in FIG. 6, computing system 110 determines whether the direction of gaze 113 remains fixed on a particular image 111A within a small tolerance band, Δ_GAZE, for a predetermined period of time, T_GEST, after time period, T_GAZE. In this manner, computing system 110 analyzes signals 106 to verify the confirmatory head gesture for a period of time subsequent to the period of time used to identify that the direction of gaze was originally fixed on a particular image. Δ_GAZE is selected to be a small value that indicates little or no movement of the eye relative to the head. The allowable movement of the eye relative to the head during the confirmatory head gesture is chosen to ensure that the gaze remains fixed on the selected image for the confirmatory time period, T_GAZE.

[0031] For example, as depicted in FIG. 4, user 10 executes a confirmatory head gesture by nodding his/her head (i.e., rotating his/her head about the Z-axis of the head-fixed coordinate frame XYZ). Furthermore, the gaze of user 10 remains fixed on selected image 111A during execution of the confirmatory head gesture. In this example, computing system 110 determines that signals 106 indicate movement of the eye 11 relative to the head 10 within tolerance band, Δ_GAZE, during the period of time, T_GEST. In response, computing system 110 executes the command indicated by the image selected by the user (e.g., the “retrieve” command illustrated in FIG. 4). In one example, computing system 110 communicates a message 108 to an external computing device via wireless communications device 150. Message 108 includes a request to “retrieve” information from the external computing system and send the information back to computing system 110 for display to the user.

[0032] As depicted in FIG. 4, the display system and gaze tracking system move with the head as the head moves because the head-mounted display is securely fixed to the head. Thus, no movement of the eyes relative to the head is observed by the gaze tracker as long as the gaze of the user remains focused on the selected image 111A.

[0033] In another example depicted in FIG. 5, user 10 executes a confirmatory head gesture by nodding his/her head (i.e., rotating his/her head about the Z-axis of the head-fixed coordinate frame XYZ). However, the gaze of user 10 remains fixed on image 112 located in the surrounding environment during execution of the confirmatory head gesture. In other words, when the eye of the user is focused on an object in the surrounding environment (i.e., the environment that is not moving with the head) the vestibulo-ocular reflex (VOR) causes the eye to move antagonistically with the head to maintain focus on the object while the head is moving. In this scenario, the eye moves with respect to the head to maintain the direction of gaze on the object. In this example, computing system 110 determines that signals 106 indicate movement of the eye 11 relative to the head 10 that is outside tolerance band, Δ_GAZE, during the period of time, T_GEST. In response, computing system 110 does not execute any command indicated by any of the images presented to the user by display system 102.

[0034] Computing system 110 is communicatively coupled to the head mounted display 102, gaze tracking device 103, and head motion tracking device 104 by a wired, or wireless communications link. In addition, computing system 110 is communicatively coupled to a wireless communications device 150 by a wired communications link. In turn, wireless communication device 150 is operable to maintain a wireless communication link with an external computing device with access to a network such as a local area network, wide area network, the internet, etc. In the example depicted in FIG. 2, wireless communications device 150 maintains a wireless communications link operating in compliance with any suitable wireless communications protocol (e.g., Bluetooth®, WiFi, ZigBee®, any cellular network based protocol, or other high frequency communications network).

[0035] In one embodiment, wireless communications device 150 is operable to maintain a wireless communication link with an external computing device (e.g., router) within a network such as a local area network, wide area network, the internet, etc. In this example, wireless communications device 150 maintains a wireless communications link with a network communications device such that data (e.g., message 108) may be communicated between computing system 110 and the network.

[0036] In another embodiment depicted in FIG. 2, wireless communications system 150 is configured to communicate digital signals from computing system 110 to a portable computing device 180. As depicted in FIG. 2, wireless communications system 150 transmits a radio frequency signal 152 over antenna 151. The radio frequency signal 152 includes digital information indicative of the digital signals (e.g., message 108) to be communicated from computing system 101 to the portable computing device 180. The radio frequency signal 152 is received by wireless communications device 180 over antenna 181.

[0037] As depicted in FIG. 2, by way of non-limiting example, portable computing device 180 is an internet enabled electronic device that is further configured to communicate the digital signals to another computing system (not shown) that is communicatively coupled to device 180 over the internet 190. In some other examples, portable computing device 180 is a network enabled electronic device that is further configured to communicate the digital signals to another computing system (not shown) that is communicatively coupled to device 180 over a local area network.

[0038] In a further aspect, computing system is configured to control the supply of electrical power to one or more of the head mounted display 102, gaze tracking device 103, and head motion tracking device 104. As depicted in FIG. 2, computing system 110 includes a battery 170 that supplies power to electronic components of computing system 110. However, battery 170 is also configured to supply electrical power 171 to power supply 160. Power supply 160 is configured supply electrical power signals to the head mounted display 102, gaze tracking device 103, and head motion tracking device 104. For example, power supply 160 supplies electrical power signal 163 to head mounted display 102, electrical power signal 162 to gaze tracking device 103, and electrical power signal 161 to head motion tracking device 104. Power supply 160 is attached to bus 140 and is controlled by processor 120. Processor 120 is configured to detect the power supply requirements (if any) of each attached system and configure power supply 160 to supply appropriate electrical power signals.

[0039] As depicted in FIG. 2, computing system 101 includes a processor 120, a memory 130, and a bus 140. Processor 120 and memory 130 are configured to communicate over bus 140. Memory 130 includes an amount of memory 131 that stores data collected from gaze tracking system 103 and head motion tracking system 104, along with threshold values such as T_GAZE, T_GEST, and Δ_GEST as described hereinbefore. Memory 130 also includes an amount of memory 132 that stores program code that, when executed by processor 120, causes processor 120 to implement hands-free interface functionality as described herein.

[0040] In some examples, processor 120 is configured to store the digital signals generated by gaze tracking device 103 and head motion tracking device 104 onto memory 131. In addition, processor 120 is configured to read digital signals stored on memory 131 and transmit the digital signals to display system 102, wireless communication transceiver 150, and programmable power supply 160.

[0041] FIG. 7 illustrates a flowchart of a method 200 implementing hands-free interface functionality as described herein. In some embodiments, wearable computing system 100 is operable in accordance with method 200 illustrated in FIG. 7. However, in general, the execution of method 200 is not limited to the embodiments of wearable computing system 100 described with reference to FIG. 2. This illustration and corresponding explanation are provided by way of example as many other embodiments and operational examples may be contemplated.

[0042] In block 201, a plurality of selectable images 111A-D are projected in a field of view of a user 10 of a head-mountable computing system 100.

[0043] In block 202, an indication (e.g., signals 106) of an amount of movement of an eye 11 of a user 10 relative to a head of the user is received by computing system 110.

[0044] In block 203, a gaze of the user 10 is determined to be fixed on a selected image 111A of a plurality of selectable images 111A-D for a first period of time, T_GAZE, based at least in part on the amount of movement of the eye 11 of the user 10 relative to the head of the user.

[0045] In block 204, an indication (e.g., signals 107) of an amount of motion of the head of the user 10 is received by computing system 110.

[0046] In block 205, the user 10 is determined to be executing a confirmatory head gesture based on the motion of the head of the user for a second period of time, T_GEST, after the first period of time, T_GAZE.

[0047] In block 206, the user selection of the selected image is verified by determining that the amount of movement of the eye 11 of the user 10 relative to the head of the user 10 is below a predetermined threshold value, Δ_GEST, for the second period of time, T_GEST.

[0048] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0049] Although certain specific embodiments are described above for instructional purposes, the teachings of this patent document have general applicability and are not limited to the specific embodiments described above. Accordingly, various modifications, adaptations, and combinations of various features of the described embodiments can be practiced without departing from the scope of the invention as set forth in the claims.