Fingertip visual haptic sensor controller

[0001] This application claims priority under 35 U.S.C. §119(e) to provisional application Ser. No. 60/894,345, filed Mar. 12, 2007, the contents of which are incorporated by reference herein in their entirety.

[0002] 1. Field of the Invention

[0003] The present invention relates generally to devices and methods for remotely sensing and controlling the environment, and, more specifically, the present invention is directed to visual sensing devices that provide tactile sensation of edges and objects in the environment and control of recognized targets in the environment.

[0004] 2. Description of the Background

[0005] Devices that transfer input from the visual environment to another sense have been developed. For example, visually impaired individuals have been presented with a wide variety of devices that transfer to another sense at least some of the vision-related capabilities of normally-sighted individuals. The primary assistive technologies presently used by the blind to navigate through the environment are essentially unchanged from those used twenty years ago, namely, white canes and guide dogs. Although these two methods can provide the ability to travel safely in a wide variety of environments, neither provide the kind of assistance needed to straighten a picture frame on the wall or find a can of soup on a counter-top.

[0006] Electronic navigation aids are finding some acceptance, especially laser and ultrasound canes, as well as portable computers with global positioning systems (GPS) and electronic Braille or speech interfaces. Replacing the more general capabilities of vision to provide detailed information about objects in the environment has proven much harder.

[0007] Existing products capture an image and then display that image on a tactile screen that could be worn on a belt.

[0008] The fingers can then be used to interrogate the image depicted on the tactile screen in an effort to “visualize” the image depicted on the screen.

[0009] There are also a number of related technologies that monitor hand, eye, and body motion from a fixed camera to interpret gestures, commands, gaze direction. These are generally not worn by the operator and thus only operate within a limited space, focusing on the operator himself rather than his interrogation of whatever environment he happens to be in.

[0010] There exists a need, however, for a device that allows for active interrogation and sensing of the three dimensional environment surrounding the operator as he moves through daily life. There also exists a need for a device that not only allows the user to sense the environment but also provides control of specific aspects of the environment that are sensed by the device. There exists a need for a portable device providing this ability to interrogate and control the environment which is not limited to any specific environment. Through its many preferred embodiments, the present invention addresses one or more of the above limitations in the art to provide a variety of different sensing and control devices and methods for use as a surrogate for vision.

[0011] The present invention is directed to providing a user with the ability to sense visual input and control remote devices via the fingertip. In at least one preferred embodiment, a small image sensing device (such as a video camera or laser range finder) and a haptic stimulator (such as a small, mechanical vibrator) mounted on a fingertip provides tactile sensations given certain input to the image sensor. In one application, the image sensing device may be swept across a “field of view” and, when it encounters a known condition (such as sensing a physical edge), provides tactile feedback to the user through the vibrator mounted on the user's finger(s).

[0012] In one preferred embodiment, an image sensing device is mounted on the dorsal aspect (top) of one or more fingertips. Each of these sensors captures image data and communicates the image data to a computer vision system. Preferably, each image sensing device works independently, thereby providing the user feedback (and potential control) over a wide variety of situations at one time. The computer vision system processes the captured image data from each image sensing device according to preplanned or adaptive rules, and the system transmits a signal back to the haptic stimulator (which may also be mounted on the user's fingertip) corresponding to each respective image sensing device. Therefore, each fingertip is able to individually interrogate the three dimensional environment and receive feedback about what the finger “sees.”

[0013] In some embodiments, the image sensing device may also be used to control remote devices via the fingertip. In these instances, the computer vision system can recognize specific targets (such as a light switch or lamp) which can be controlled by the flick of a finger or turn of the hand. The applications for both the sensing/recognition of real-world objects and conditions, as well as the control of the same, are almost limitless within the teachings of the present invention.

[0020] It is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clear understanding of the invention, while eliminating, for purposes of clarity, other elements that may be well known. Those of ordinary skill in the art will recognize that other elements are desirable and/or required in order to implement the present invention. However, because such elements are well known in the art, and because they do not facilitate a better understanding of the present invention, a discussion of such elements is not provided herein. The detailed description will be provided hereinbelow with reference to the attached drawings.

[0021] The present invention, in at least one preferred embodiment, provides an image sensing device and haptic stimulator mounted on a user's fingertip or other appendage in order to facilitate a “visual” representation of the user's environment. Specifically, the image sensing device, which may be in the form of a miniature camera, laser range finder or other device, is aimed or swept throughout the environment by the user while it captures data about the environment. This captured data is sent to a computer processor (which may include a computer vision system) which is programmed to recognize certain environmental features (such as the edge of a doorway or a specific item) when these features are encountered within the field of view of the image device.

[0022] Upon detection, the computer processor then preferably signals back to the haptic stimulator (which may be mounted on the same finger as the image sensing device) causing a small, mechanical vibrator or other physical actuator (or any other method of communicating with the sensory system, such as through electrical stimulation or temperature) to pulse and alert the user that the environmental feature was detected. In a simple form, the image sensor/haptic stimulator may act as a “virtual cane” for the blind-sensing edges and obstructions in the environment. Through the use of more robust computer hardware and vision systems, the capability to recognize precise objects provides a greater degree of specificity. By building on the natural skills inherent to exploring the tactile environment with one's hands, the user will benefit from an ease of learning and interpretation when applying these skills to the visual environment.

[0023] FIG. 1 shows an enlarged perspective view of one exemplary embodiment of the present invention 100 where the image sensing device is a camera 101. The camera 101 is shown mounted on the dorsal (top) portion of a finger with the image capturing field of view of the camera 101 extending out from the fingertip. As the user extends and sweeps the fingertip in front of the user, the camera collects either continuous or periodic (e.g., sampled) images that are sent to a separate processor (described below).

[0024] The device 100 of FIG. 1 also shows a haptic stimulator 102 mounted below the camera 101. The haptic stimulator 102 is preferably mounted to the same fingertip as its corresponding image sensor 101, and, more specifically, to the top surface of the fingertip (dorsal aspect of the distal phalanges) leaving the distal and ventral aspects free to be used in normal touching and gripping. In this embodiment, the camera 101 and haptic stimulator 102 are attached to the finger by an elastic cuff 104, which encircles the finger at the distal inter-phalangial joint. Other attachment mechanisms could be used, as understood by those skilled in the art, such as an adhesive to attach to the fingernail.

[0025] The image sensing device (camera) 101 and haptic stimulator 102 are preferably connected to a computer processor or computer vision system (shown in FIG. 4) by a communication link 106. The communication link 106 may be a wire as depicted in FIG. 1 or could be any type of wireless or other communicative connection. Through this communication medium, the signal from the image sensing device 101 may be transmitted to the processing system. Likewise, after the sent image data is processed by the computer vision system, a responsive signal may be sent from the computer processor back to the haptic stimulator over the same communication medium to mechanically actuate the stimulator against the user's fingertip (as described below). The computer vision system could also be miniaturized sufficiently so that it could be mounted on the finger along with the camera and haptic stimulator, eliminating the need for a communication system.

[0026] Multiple transmitters and receivers may exist on the device worn by the user to relay signals between the image sensing device, computer vision system and haptic stimulator. Likewise, multiple transmitters and receivers may also exist in the environment to relay control signals to and from remotely controlled targets in the environment.

[0027] Audio transmitters and receivers may also exist on the device to audibly alert the user of specific features and targets in the environment and to allow the user to communicate with the device by voice command. Thus, a user can verbally communicate to the finger a specific feature or target in the environment that the user desires to locate. The user can then move his fingers over the environment and receive an audio communication back from the finger once the specific feature or target in the environment is detected. The user may also be alerted through the haptic stimulator when the specific features or targets are identified in such a way that their location may be determined and/or control over them may be exerted.

The Image Sensing Device

[0028] FIG. 2 depicts a simplified block diagram of one preferred system according the present invention. As shown in FIG. 2, an image sensing device 200 is in communication with a computer vision system 202 via a communications link 204. The image sensing device 200 captures visual features in the environment. The image sensing device is any device that captures information about points in space along a set of projected lines, such as a video camera. The image sensing device may also capture information about range along each line, as in the case of a laser rangefinder, set of stereo cameras, or sonar scanner. Other devices such as infrared cameras and range finders or still cameras with rapid image capture may also be utilized. The image sensing device encompasses any modality that interrogates space relative to the location of the fingertip.

[0029] One preferred example of an image sensing device 200 is a video camera. Video cameras are well known devices used to capture images. The first cameras were relatively large devices and cumbersome to use. Currently, miniature cameras are common, and are small enough and light enough to be mounted on a fingertip.

[0030] A camera captures the visual environment and converts the three dimensional world into a two dimensional image map made up of numerous pixels. The signal from the image sensing device 200 is then transmitted to a real-time computer vision system 202 where the pixels are analyzed to detect a particular feature, such as an edge with a particular orientation across the center of the image, or to identify certain targets in the image, such as a coffee cup.

[0031] There are various ways in which to implement the imaging sensor as a camera or other device. As shown in FIG. 1, the image sensing device could be a miniature camera.

[0032] For example, the image sensing device could be a very small (e.g., ¼″ by ½″) video camera mounted to the fingertip. However, in its simplest form, the image sensing device could be a miniature single pixel sensor that focuses on a single pixel of an image. This could be as simple as a lens with a pinhole at the plane of focus, behind which a photosensor would detect only that particular location in the image. Such a sensor may be cheaper and smaller than an entire camera, and with motion of the camera could report a temporal signal that could be processed to accumulate and identify spatial patterns such as edges. Alternative means of implementing the image sensing device include a laser (or laser range finder) or ultrasonic transducer.

The Computer Vision System

[0033] Computer vision systems are well known in the art and allow for perception of information from image sensing devices. FIG. 3 shows one exemplary computer vision system that may be used as part of the present invention. While the computer vision system 300 depicted in FIG. 3 is a conventional desktop computer, the computer vision system of the present invention may include any image processing system including, for example, laptop computers and handheld or pocket devices. In some embodiments, the computer vision system may be a specially programmed microprocessor embodied in a portable device that may be worn by the user during use, or mounted directly on the finger with the image sensing device.

[0034] Computer vision is a well-established and broad field with many techniques that may be applied to the present invention, such as optical flow, feature analysis, target recognition. Any and all aspects of the computer vision field may be utilized in connection with the fingertip image sensing device.

[0035] Referring back to FIG. 2, the computer vision system 200 receives a signal 204 from the image sensing device 200 which includes visual images of the environment. The computer vision system 202 uses a vision algorithm to obtain information from images of the environment or other sensory data received from the environment. The images or sensory data are processed by the computer vision system 202, and the computer vision system detects features in the environment. These features are localized areas having particular characteristics such as an edge, a color, or texture. The computer vision system 202 may also detect entire targets in the environment that it is programmed to identify.

[0036] For example, a computer vision system 202 may identify edges in the environment such as door frames, tables and countertops, or objects such as a coffee cup or pencil. In addition, the computer vision system may be programmed to identify specific targets such as a light switch, lamp or television set. These targets may also include a dot, cursor, bar, icon, or other pattern on a computer screen of other type of monitor (as described in more detail below) or specific targets in open surroundings such as markers, buoys, or a lighthouse. As is known in the art, the capabilities of computer vision systems are expansive; such systems allow for detailed perception of the environment such as face recognition and optical character recognition.

[0037] Several techniques have been well explored in computer vision for identifying targets in the environment, including combining features with particular geometric relationships, as well as image-matching schemes based on libraries of pre-recorded images of many different targets, to name just two. These and other techniques that are readily understood by those skilled in the art could all be utilized as part of the present invention.

[0038] The computer vision system of the present invention may receive instructions as to specific features to identify in the environment at any given time. The identifying features of certain detectable objects may be preprogrammed in the computer processor or vision system, and others may be programmed based on the use of the system by a particular user at any time. These instructions may be input into the computer vision system manually, such as by keyboard, by voice command from the user, or by other means including motions of the hand and fingers. Changes in the features and targets to be identified may also be automatically controlled given, for example, the location, time of day, or mode of user operation.

[0039] For example, a user may be able to verbally say the word “pencil” as a way of commanding the vision computer system to detect a pencil in the environment. The computer vision system will then process images being received from the image sensing device and identify any pencils in the environment (e.g., by matching the received image data to stored data about a pencil). If a pencil is detected, the computer vision system can transmit a signal to the haptic stimulator mounted on the fingertip causing it to vibrate. Location within the image relative to the central pixel (direction of finger pointing) may be encoded in the haptic feedback, or otherwise communicated to the user, so as to permit “homing in” on the target (such as a pencil) without initially having to have been pointing directly at it.

[0040] Certain targets may also be used as navigational beacons, providing the user with detailed information of his/her location relative to the beacon, including distance and orientation. This technology is well known to the field, and includes special graphical patterns designed to permit the extraction of such distance and orientation information of the video camera from the image captured by the camera. Thus the user could know where he/she is with respect to a target within a room and receive feedback helpful to navigation. For example, if these targets were on a computer display, they could be adaptive, facilitating effective localization of the camera. Active light-source arrays, well known to the field, could also serve as effective targets for this purpose, given the appropriate camera system.

Tactile Sensation Output (The Haptic Stimulator)

[0041] As described above, and shown in FIG. 1, a haptic stimulator (most simply, a vibratory transducer) is preferably mounted to a fingertip, and, more specifically, to the top surface of the fingertip (dorsal aspect of the distal phalanges). When the computer processor detects an environmental feature or object for which the user is searching, the processor sends a signal to the appropriate haptic stimulator which then provides the user tactile feedback about the user's visual environment.

[0042] Referring to FIG. 2, the haptic stimulator 206 is connected to the computer vision system 202 by a communication link 208 which may take the form of any wired or wireless communications medium, or as described previously the computer vision system could be miniaturized to fit on the finger, eliminating the need for the communication system. When an edge, object or other programmed feature or target in the environment is identified by the computer vision system 202, the computer vision system transmits a signal via communication link 208 to the haptic stimulator 206 attached to the finger causing the stimulator to vibrate.

[0043] Haptic devices beyond simple vibratory transducers have been developed and may be utilized as a haptic stimulator, including devices based on multiple vibratory “pins” and ones that move a looped or spherical surface repeatedly past the fingertip as the fingertip moves to simulate motion across a stationary surface. Direct electrical stimulation has also been used, as well as transducers that change temperature. Other embodiments may include pneumatic devices having an array of tiny holes that blow air across the fingertip making waves that create the perception of moving textures.

[0044] One significant feature of the present invention is that it allows for independent interrogation of the environment by multiple fingers simultaneously. Each fingertip is able to do its own interrogation of the environment. Each fingertip having an image sensing device and haptic stimulator is preferably treated separately by the computer vision system, which permits potentially up to ten or more image sensing devices to be used simultaneously. The computer vision system may take advantage of the multiple images to combine information in useful ways, such as to interpret the relative orientations of the fingertips and to enlarge the overall understanding of the environment. Such haptic stimulators mounted on multiple fingers may give a coordinated sense that can be quickly swept through the environment using motions of the hands and fingers.

[0045] Using multiple image sensing devices together could also allow a user to more easily determine the orientation or shape of an edge or object in the environment. For example, if the haptic stimulators are made to vibrate when finding an edge with a given orientation, by rotating the hand around the target object remotely or moving the fingers, a user can determine whether the detected edge is a horizontal or vertical edge such as those edges representing doorways or tables, or determine whether the shape of an object on a desk was square or round.

[0046] A user wearing multiple image sensing devices can also search the environment for multiple features simultaneously. For example, a user can program the computer vision system to utilize the image sensor on the right index finger to search for a “pencil” and the right middle finger to search for a “coffee cup” (e.g., by using a voice command to the vision system. Then, by waving the right hand throughout the environment, the user simultaneously searches for both items at the same time. As each item is detected by the vision system, the appropriate haptic stimulator (right index finger for “pencil” and right middle finger for “coffee cup”) will be actuated. By using voice commands to continually change the “object” for which each finger is searching, a very complex and flexible visual search system may be employed through the use of the present invention.

Controller

[0047] In addition to, or as an alternative to, providing tactile sensation, the image sensing device according the present invention can be used to control remote targets in the environment. The controller preferably operates in conjunction with the haptic stimulator, though the haptic stimulator is not necessary for operation of the controller. Embodiments may include both the haptic stimulator and controller to provide the user with both tactile sensation and control of features in the environment.

[0048] The location of targets in the environment is captured by the image sensing device and recognized by the computer vision system according to the methodology set forth above. However, when the image sensing device is pointed at a specific target that has been located, control signals can be generated by specific motions of the image sensing device (e.g., bending a finger up and down) that control aspects of the environment (e.g., turning a light switch on and off). Such motions may be determined from the image itself, or in addition, or instead, by other sensors on the finger including inertial or joint-angle sensors as are well known in the field.

[0049] The control features of the device are practically unlimited. For example, the controller may be used simply to control a cursor on a computer screen by movement of the finger. Or, in more complex embodiments, the controller may involve pointing all of the fingers at a robot to control the robot like a puppet by movement of the fingers.

[0050] In the simple scenario of the cursor, the controller can keep the cursor with the same location and orientation with respect to the camera's field of view. Thus the curser moves with the finger. The controller may move the cursor around the screen, or the cursor may move as a slide pot so that when the finger moves side to side the cursor on the screen moves side to side along a line, or the controller may be used as a virtual knob that determines and controls the angle of a cursor shaped like an arrowhead.

[0051] The concept of control could involve special targets that would be recognized by a computer vision system that is connected to the image sensing device, such that when the finger is pointed at the target, the control signals could be issued by specific motions of the image sensing device.

[0052] As one example, the image sensing device could be a fingertip camera connected to a computer vision system and the target could be a computer screen containing a dot that could be moved by the fingertip using a control feedback loop to keep the dot in the center of the camera image. In this example, the computer vision system uses an algorithm to identify the dot on a computer screen, and to control the location of that dot by moving the dot in order to maintain the dot in the center of the camera's field of view. Thus, movement of the fingertip camera will cause the dot to move around on the screen.

[0053] As described above, this aspect of the invention can be implemented with a camera as the image sensing device. The miniature camera is mounted to the fingertip of the user and when pointed at the dot, the dot is detected by the computer vision system. The computer vision system then locks onto the dot and causes the finger to vibrate via the haptic stimulator indicating that the dot is locked. The location of the dot can then be controlled remotely by the user by movement of the finger.

[0054] As one example of this implementation, the camera can be pointed at a black computer screen having a single white dot at the center point of the screen. The center point of the camera image may be designated as coordinates (0,0). Taking an image from the camera, the computer vision system can examine each pixel on the cameral image and identify the dot by determining the pixel with maximum brightness. This point is then designated as coordinates delta X, delta Y. By causing the dot to move by −delta X, −delta Y, the computer can place the dot in the center of the camera image. If this is done repeatedly as the camera moves, the dot will effectively track finger movement and continually move to where the finger is pointing. If the finger moves beyond edge of the screen, the dot may be placed back to its original center point on the screen or remain at the edge of the screen nearest to where the finger moved past edge of the screen. Many variations are possible. For example, the dot can be a single pixel, a clump of pixels, or a blur. Haptic feedback can be supplied as to whether tracking is occurring.

[0055] The control aspect of the invention may be applied to any number of targets recognized by the computer vision system. For example, the dot could be a bar that can be rotated by turning the hand which in turn rotates the camera attached to the fingertips. Such rotation of the bar could be linked to specific lighting or a dimmer switch and thus act as a dimmer dependent on the extent that the bar is rotated. Other such commands can be located on the screen and programmed into the computer vision system.

[0056] Another implementation could utilize an actual light switch or lamp rather than a computer screen. The computer vision system could detect a specific light switch, lamp, television set, or any other mechanical or electronic device that is captured by the image sensing device. Once captured, the finger would vibrate via the haptic stimulator, and then a flick of the finger or turn of the hand could generate a control signal to manipulate the target device, so as to turn the target device on or off, adjust its volume, change the station if the target device is a television or radio, or any other functionality of the target device. This aspect of the invention can be extended to many other real world mechanical or electronic manipulations including remote sensing and control of video games or other types of multimedia.

[0057] For example, in the case of controlling a light switch, the light switch could, but would not itself have to be an active device that is directly controlled by the controller. The light switch need only be a target visually recognizable by the vision computer system. The controller can then communicate with a responsive control system in the light switch or a separate control system, such as a circuit to dim the lights, to effect control over the system and turn the lights off or on or adjust brightness of the lights.

Other Exemplary Embodiments

[0058] FIG. 4 depicts an exemplary embodiment of the invention that uses a laser as the image sensing device. In this embodiment, the image sensing device is a laser 400 that is used to illuminate features in the environment and the reflections from the laser are detected by a phototransistor 402 attached to the laser 400. Attached to the underside of the laser 400 are miniature magnets 404 and a coil 406 which cause the laser 400 to vibrate when activated by reflections detected by the phototransistor 402 when features are identified in the environment.

[0059] FIG. 5. depicts a block diagram of the system shown in FIG. 4. In this embodiment, a target having an edge 500 is depicted as an abrupt transition between black and white regions. To avoid interference from ambient light, the laser 502 is modulated with a local oscillator 504 such as a 10 KHz square-wave oscillator. Reflected laser light is detected using a phototransistor 506 and the signal is filtered and amplified by a receiver circuit 508. Since both the frequency and phase of the modulated laser signal are known, autocorrelation using a correlator 510 can be effectively used to detect its presence in noise. The phase of the local oscillator 504 is adjusted empirically by a phase adjuster 512 to optimally compensate for phase shifts in the overall system, and the DC component of the autocorrelation is blocked by a DC blocker 514 leaving only the changes caused by movement across the optical edge 500. These changes are used to power a solenoid driver 516 (consisting of a coil 518 mounted on the fingertip and a stack of miniature magnets 520 attached to the laser) to produce vibration via regenerative feedback in the laser when a properly oriented edge is found.

[0060] Nothing in the above description is meant to limit the present invention to any specific materials, geometry, or orientation of elements. Many part/orientation substitutions are contemplated within the scope of the present invention and will be apparent to those skilled in the art. The embodiments described herein were presented by way of example only and should not be used to limit the scope of the invention.

[0061] Although the invention has been described in terms of particular embodiments in an application, one of ordinary skill in the art, in light of the teachings herein, can generate additional embodiments and modifications without departing from the spirit of, or exceeding the scope of, the claimed invention. Accordingly, it is understood that the drawings and the descriptions herein are proffered only to facilitate comprehension of the invention and should not be construed to limit the scope thereof.