US 5,594,469 AGrant
Hand Gesture Machine Control System
Issue Date:1997-01-14
•16 Claims
•9 Drawing Sheets
Abstract
A system for the control from a distance of machines having displays incls hand gesture detection in which the hand gesture causes movement of an on-screen hand icon over an on-screen machine control icon, with the hand icon moving the machine control icon in accordance with sensed hand movements to effectuate machine control. In one embodiment, TV control led by hand signals includes detecting a single hand gesture and providing a hand icon on the screen along with the provision of icons representing TV controls such as volume, channel, color, density, etc., in which a television camera detects the hand in a noisy background through correlation techniques based on values of local image orientation. In order to trigger the system into operation, a trigger gesture such as the "how" sign is distinguished from the background through the utilization of orientation angle differences. From correlation values based on correlating local orientations between a mask defining a particular hand and the later acquired image of the hand, normalized correlation scores for each pixel are obtained, with the correlation peak being detected and then thresholded to eliminate false alarms.
Metadata
Assignee
- Mitsubishi Electric Information Technology Center America Inc.
Inventors
- William T. Freeman
- Craig D. Weissman
Application Information
Application Number:US 3919552
Filing Date:1995-02-21
Priority Date:1995-02-21
Art Unit:269
Classifications
IPC:
G09G 508
Field of Search:
345395382157;156;158;145;146159291;203
Patent Drawings (9 sheets)
Description
Field of Invention
This invention relates to the control of machines by hand signals and more particularly to a hand gesture recognition system for control from a distance.
Background of the Invention
For some time it has been possible to control machines through the use of remote signaling devices in which infrared or RF signals are used to control machine function. From VCRs, TVs, to stereo equipment and computers, battery-powered hand-held devices eliminate the requirement for the individual to move to the machine to activate machine-carried buttons or switches as well as wired control modules.
However, as in the case of television, hand-held remote control modules which control the operation of the television in terms of channel, volume and in some cases hue, density and contrast are easily lost or misplaced. Even if not lost, they may be in another location, requiring the viewers to retrieve them. As a result there can be considerable frustration on the part of television viewers when they wish to turn on or control the television. Moreover, with respect to computer workstations, control of the workstation with a hand-held remote control unit also suffers the same problems of loss of the remote control unit.
Additionally, and perhaps more importantly, providing a wide variety of buttons on a remote control module is often confusing to the user because of the multiple buttons. The large number of buttons was necessitated by the large number of functions to be controlled. The more functions, the more buttons which adds to the complexity of activating the various functions.
As a result there is a requirement for remoteless TV control which is instantly learnable in terms of instructing an individual as to how to operate the TV. The remoteless TV control must be easy to remember, easy to master, and reliable. Moreover, the penalty for false entry into a control mode must be made relatively small.
By way of background, the following articles describe machine control and the use of pattern and orientation analysis in the detection of objects within an optically scanned scene: D. H. Ballard and C. M. Brown, editors. Computer Vision. Prentice Hall, 1982; A. Blake and M. Isard. 3D position, attitude and shape input using video tracking of hands and lips. In Proceedings of SIGGRAPH 94, pages 185-192, 1994. In Computer Graphics, Annual Conference Series; T. J. Darrell and A. P. Pentland. Space-time gestures. In Proc. IEEE CVPR, pages 335-340, 1993; J. Davis and M. Shah. Gesture recognition. Technical Report CS-TR-93-11, University of Central Florida, Orlando, Fla. 32816, 1993; W. T. Freeman and E. H. Adelson. The design and use of steerable filters. IEEE Pat. Anal. Mach. Intell., 13(9):891-906, September 1991; M. Kass and A. P. Witkin. Analyzing oriented patterns. In Proc. Ninth IJCAI, pages 944-952, Los Angeles, Calif., August 1985; H. Knutsson and G. H. Granlund. Texture analysis using two-dimensional quadrature filters. In IEEE Computer Society Workshop on Computer Architecture for Pattern Analysis and Image Database Management, pages 206-213, 1983; J. M. Rehg and T. Kanade. Digiteyes: vision-based human hand tracking. Technical Report CMU-CS-93-220, Carnegie Mellon School of Computer Science, Pittsburgh, Pa. 15213, 1983; and J. Segen. Gest: a learning computer vision system that recognizes gestures. In Machine Learning IV. Morgan Kauffman, 1992. edited by Michalski et. al.
Summary of the Invention
In order to eliminate the problems of control of machines through the use of signing, in the present invention a remoteless system for the control of machines having displays includes hand gesture detection in which the detected hand gesture causes movement of an on-screen hand icon over an on-screen machine control icon, with the hand icon moving the machine control icon in accordance with sensed hand movements. In one embodiment only a single hand gesture is recognized, making the user's learning of many hand gestures unnecessary. Note that the single hand gesture is used to control the multiple functions associated with the respective on-screen machine control icons by merely moving the hand icon over the particular machine control icon desired. The result is a simple robust control system which can be easily learned and which can be used to control as many functions as there are on-screen icons.
In the general case, the subject remoteless TV control system which utilizes hand gesture recognition is simplified through the utilization of a small vocabulary of hand gestures which would otherwise be hard for the user to remember and hard for the computer to recognize. In one embodiment, the small vocabulary of hand gestures revolves around a single gesture which is relatively easily established by the user, namely an open hand toward the television, also known as the hello or "how" sign. The position of the hand in this gesture is sensed so that the movement of the hand moves a hand icon on the screen to permit placing of the hand icon over control icons such as channel or volume slider controls, or up-down arrows.
In one embodiment, false recognition of TV commands due to ordinary movements is solved through the utilization of a special gesture to turn on the system, with this gesture functioning as a trigger gesture for entering the TV control mode. The gesture, in one embodiment, is the aforementioned hello or "how" sign which involves minimal gesture memorization by the user, e.g. just putting a flat hand in the air. The flat hand is relatively easy for a machine to identify and track even in a cluttered visual scene, with the hand icon on the TV screen providing visual feedback to permit the user to control the TV merely by changing the position and or orientation of his hand.
The system is designed so that the penalty for false entry into control mode is small. For instance, the penalty in one embodiment is that the icons appear only at the bottom portion of the TV screen, so that very little of the screen is disrupted. As a result, upon false activation only the bottom of the screen is occupied which makes false entry relatively unobtrusive.
In order to enter the control mode a large area of what the camera sees is scanned for the presence of a hand in the "how" position. In one embodiment, scanning is accomplished by a normalized correlation of a set of one or more prototypical hand masks with the image. The correlation is based on the values of local orientation instead of image intensities to give more robust results, with the correlation vectors being unit vectors with given orientation angles based on origins at points on the hand.
In one embodiment, normalized correlation is used. As a result, the series of orientations resembles needles with various orientations about the hand. If the maximum correlation score exceeds a threshold value, then the trigger gesture is detected and the television enters control mode. In one embodiment, the position of the hand is taken to be the position of maximum correlation score.
Once entering into the control mode, the camera scans the scene for a hand which matches or correlates with the mask, with the correlations being based on correlating local orientations between the mask and the image. This type of system is robust against lighting changes and against varied and noisy backgrounds. In one embodiment, once a suitable correlation between the hand and the mask is recognized, the portion of the scene analyzed by the subject system decreases to the area of the hand so that faster hand tracking for the movement of the hand to move the hand icon is facilitated.
It has also been found that the effect of a cluttered background can be reduced through a background subtraction procedure described hereinafter.
Under some circumstances it is desirable to sense this single hand gesture in different angular orientations to provide different machine control functions and to allow the user to hold his hand at different orientations. In this case with the normalized correlation scores, different angular orientations of the hand can be sensed through the utilization of a number of masks having hands at different orientations. Thus different orientations of the open hand can be made to control different functions or result in different icon movement.
The utilization of hand gesture recognition through a single gesture and the utilization of a corresponding hand icon provides a visual feedback to eliminate the necessity for a large variety of hand gestures for different control functions. All that is necessary is that there be coincidence of the hand icon with a control icon on screen, with visual feedback from the screen indicating where to move one's hand in order to effectuate the desired control. Using multiple hand gestures for different functions is unnecessarily confusing, like providing a remote control device with a large number of buttons. The Subject System thus simplifies machine control by sensing only one gesture and providing multiple on-screen machine control icons for the multitude of different functions which it is desired to control.
In summary, by the movement of a hand or other body part a corresponding visible position indicator or cursor in the form of an icon is moved on-screen. This visible on-screen cursor allows the user to adjust his motions or actions to move the cursor in the desired way. Cursor movements then cause machine commands to be issued. Note that cursor movements over a machine control icon such as a slider can be made to cause movement of this icon which then results in the issuance of the requisite machine commands.
In one embodiment, a TV controlled by hand gestures includes detecting hand gestures and providing a hand icon on screen with the provision of icons representing TV controls such as volume, channel, color, density, etc. in which a television camera detects the hand in a noisy background through correlation techniques based on values of local orientation instead of image intensities. In order to trigger the system into operation, a trigger gesture such as the "how" sign is distinguished from the background through the normalized correlation of local image orientation angles with a hand template. Normalized correlation scores for each pixel are obtained, with correlations based on correlating local orientation between a mask defining a particular hand and the later acquired image of the hand. Once having detected a hand gesture, the field of view of the camera is reduced to slightly larger than the area at which the hand gesture was recognized to enable fast, robust tracking of hand movement for the control of the icons on screen. In one embodiment when the hand icon is over the icon representing the desired control, the desired control icon lights up and the desired control is effected, with the TV being returned to the viewing mode by closing the hand or otherwise discontinuing the display of the hand. In one embodiment the hand icon closes to confirm the closing of the user's hand, and the graphical control overlays containing the icons disappear from the TV screen at which time the television returns to the viewing mode.
Brief Description of the Drawings
These and other features of the Subject Invention will be better understood in connection with the Drawings taken in conjunction with the Detailed Description of which
FIG. 1 is a diagrammatic illustration of the control of a television through the utilization of hand signals, with a hand-icon utilized to control TV function icons on screen;
FIGS. 2A and 2B are diagrammatic representations of the television screen of FIG. 1 illustrating the positioning of the hand icon over a channel slider bar icon and the movement of the slider with the movement of the hand to effectuate channel change;
FIGS. 2C and 2D are photographs of an individual utilizing his hand in the "how" sign to capture the subject control system and to cause the on-screen hand icon to move in accordance with actual hand gestures;
FIG. 2E is a diagrammatic illustration of the television screen of FIG. 1 illustrating entry into the control mode through the detection of the "how" hand signal.
FIG. 2F is a diagrammatic illustration of the television screen of FIG. 1 illustrating the turning off of the control system with a closed fist hand gesture, with a closed fist icon being portrayed on screen;
FIG. 3 is a block diagram of the Subject Invention illustrating hand recognition, position recognition, the use of a hand template, and a processor for providing on-screen hand icons and machine control icons, while at the same time providing machine control through the detection of icon locational coincidence and movement;
FIG. 4A is a block diagram of a portion of the subject system indicating hand gesture recognition through normalized correlation and peak selection techniques;
FIG. 4B is a diagrammatic representation of the needle map associated with both the hand template and the image, illustrating the orientation representation;
FIG. 5 is a processing flow chart for the processing of raw video signals through image representation, background removal, correlation position detection, trigger gesture and tracking signal generation followed by on-screen controls and the remote control of a television, also illustrating template generation;
FIG. 6 is a diagrammatic and block illustration for a system of background removal, also illustrating actual photographs of the images for which the background removal is accomplished; and,
FIG. 7 is a diagrammatic representation of diagnostic screens or windows utilized in the analysis of the operation of the system of FIG. 3.
Detailed Description
Referring now to FIG. 1 in the hand gesture control of a machine, in one embodiment a television 10 having a screen 12 is provided with a camera 14 which scans a scene 16 through an angular view illustrated by dotted lines 18. In this scene is an individual 20 seeking to control television functions through the movement of his hand 22 in the direction of arrow 24.
The hand is detected as will be described from raw video of the scene. Upon detection of a particular hand gesture, in this case the "how" flat handed gesture, a hand icon 26 is portrayed on screen at an initial position convenient to the various control icons, 28-42. Also portrayed is a number of machine control icons, in this case a channel slider bar representation 28, a volume slider bar representation 30, an off button 32, a mute button 34, additional channel control arrows on buttons 36 and 38 and volume control arrows on buttons 40 and 42.
With the TV control system to be described, the movement of hand 22 in the direction of arrow 24 causes the on-screen hand icon 26 to move in the direction of arrow 44 such that the hand icon, when detected, mimics the hand motion. The visual feedback provided by the hand icon to individual 20 permits the individual to control the hand icon by moving his flattened hand in any direction he deems necessary in order to effectuate machine control.
As can be seen in FIG. 2A screen 50 is provided with the aforementioned machine control icons. In this figure the channel control icon 28 is depicted with the hand icon 26 being positioned over a slider 52, with the system to be described hereinafter detecting the coincidence of the hand icon with the slider.
As illustrated in FIG. 2B hand icon 26 having first established coincidence with slider 52 now moves slider 52 into position 52 to effectuate change of channel.
Referring now to FIG. 2C a picture of an individual providing the "how" sign to activate the Subject System and control the hand icon is presented within the camera viewing angle. After hand acquisition the individual moves his hand from the position shown in FIG. 2C to that shown in FIG. 2D. The movement of his hand towards his face then causes the on-screen hand icon to move in the direction of the individual's hand motion to provide the icon movement depicted in FIGS. 2A and 2B.
Referring now to FIG. 2E, as will be appreciated in order to capture the control of the television it is necessary to provide a triggering gesture, here the "how" sign. The fact of the machine entering into the control mode is indicated by the appearance of the hand icon 26 and the on-screen graphical control icons, 28-42.
Referring to FIG. 2F and screen 64, when it is desired to relinquish control of the television, a fist-like gesture, or any other gesture which is not a "how" sign, is recognized, with a fist hand icon 66 displayed on screen when the non-how sign gesture is acquired. Thus, it is possible to exit the control mode simply by making a fist.
Referring now to FIG. 3, in general a system to accomplish the aforementioned control includes a camera 70 and a hand recognition unit 72 to which is coupled a hand template or templates 74 corresponding to the user's hand. It is the purpose of the hand recognition unit to take raw camera data and to recognize a particular hand gesture exemplified by the hand template. Upon recognition of a particular hand gesture, its position is detected by a position recognition unit 76 which serves two purposes. First, upon detection of the particular hand gesture, its position in space is noted and a unit 78 is utilized to delimit the hand search region to the area around the detected hand position for robust and rapid hand gesture detection. Secondly, the position of the particular hand gesture is used to drive the particular on-screen hand icon 26 to an on-screen position depending on the position of the hand in space. For this purpose a processor 80 is utilized to process the hand recognition and position information and to drive a screen driver 82 in accordance with signals representing an on-screen hand icon 84 and a machine control icon 86, with the icons being previously stored and generated as illustrated at 88. Thus screen driver 82 under control of processor 80 provides for the on-screen icons and the movement thereof.
In order to control a machine, in this case a television, icon coincidence must be detected. Icon coincidence refers to positional coincidence of on-screen icons such as the hand and machine icons. Here the icon coincidence is detected by a detector 90. Upon detection of coincidence between a hand icon and a machine icon, hand icon movement is detected as illustrated at 92. Having identified a particular machine control by virtue of the aforementioned coincidence, hand icon movement is utilized to generate the appropriate machine control signals 94. Icon coincidence detection is also used by processor 80 to simultaneously effectuate movement of the moveable portion of on-screen machine icons. One of the first and principle requirements of the subject system is that a robust method be provided for gesture recognition such that a particular hand gesture can be picked out of a noisy environment and utilized to reliably control machine functions. As described above, one particularly powerful method of providing reliable gesture recognition is the utilization of correlation techniques based on values of local orientation. This is accomplished in hand recognition unit 72 of FIG. 3, with an intermediate result being the assignment of unit vectors having particular orientations for each portion of the hand which is detected. Here various points on the hand are provided as origins for the unit vector with the angular orientation of the unit vector corresponding to the dominant angular orientation of the image intensities of the hand at that point. The result as shown in FIG. 4B is a representation of the local orientations in the form of needle-like indicia 100.
More specifically, in operation, there are three basic functions which are accomplished by the system in order to effectuate hand gesture control. The first of these functions is to generate a template for the particular hand gesture to be recognized. In order to accomplish this initialization task an individual positions his hand in the field of view of the camera, with the location of the hand being marked out by a mouse or other device such that the field of view under consideration is limited to the hand. The system then proceeds in hand recognition unit 72 to process that region of the scene to capture the desired hand gesture for use as template 74. It is the purpose of the processing to generate a needle map corresponding to the desired hand gesture. In order to accomplish this, the X and Y derivatives of image intensity across the scanned scene are taken in the sense of taking an image intensity at one point and subtracting this intensity from the intensity of an adjacent point either immediately above or immediately to the side of this point. The result is the formation of an X derivative map and a Y derivative map, dI/dX and dI/dY. The next step is to generate a contrast image by adding (dI/dX).sup.2 and (dI/dY).sup.2 and taking the square root of the result. It will be appreciated that the contrast image is the strength of the image intensity gradient. The contrast image is then thresholded to remove noise by ignoring low contrast portions of the scene. Those portions which are above the contrast threshold are stored to generate the needle map as follows.
The needle map is provided by generating the X derivative of image intensity divided by the contrast and the Y derivative of image intensity by the contrast to yield cos (.THETA.) and sin (.THETA.) of a edge discontinuity in the scene, thereby to define the orientation of the aforementioned needles.
The resulting set of angles as a function of position can be displayed as a needle map such as that shown in FIG. 4B. The set of cos (.THETA.) and sin (.THETA.) values are then utilized as the template against which data corresponding to real time gestures is compared.
The second process which is performed by the Subject System is to detect the existence of a trigger gesture. This gesture is the aforementioned "how" sign in one embodiment or any easily recognized hand gesture.
Since in the previous step a hand template has been generated defining this gesture for the system, the system then searches the camera image for the occurrence of a needle map which is highly correlated to the hand template map. As to correlation, in one embodiment the degree of correlation is achieved via a normalized correlation technique in which the X derivative of image intensity divided by the contrast and the Y derivative of image intensity divided by the contrast are combined in a single, large vector. It is this single vector which is compared with a single vector generated from the hand template map to provide a degree of correlation between these two vectors. The normalized correlation between the two vectors A and B is generated in accordance with the following formula: ##EQU1##
It will be noted that it is important to form the correlation based on the cos (.THETA.) and sin (.THETA.) values rather than based on the orientation angle values, .THETA., themselves. 0 degrees will yield a low correlation with 359 degrees, even though the angles themselves are nearly colinear. Using a cos (.THETA.), sin (.THETA.) representation solves this problem. The vector (cos (0.degree.), sin (0.degree.)) will have a high correlation with the vector (cos (359.degree.), sin (359.degree.)).
It will be noted that the normalized correlation is calculated for regions of the image centered on every position of the image. However, it will be appreciated that the derivatives of image intensity with respect to X and Y are only valid for positions in the hand template where the contrast is above the predetermined threshold. This is also true for the incoming image. Thus, in one embodiment the correlation is only based on cos (.THETA.) and sin (.THETA.) measurements at positions where the contrast is above threshold for both the icoming image and the template. The next step provides a calculation for each position as to the percentage of the above contrast template positions which match up to above contrast image positions. Having derived this percentage for each position, if the percentage for a position is above a predetermined threshold, the correlation is valid for the position, whereas if below this threshold, the correlation is discarded. This obviously gives a step function configuration to the threshold curve. In practice however, a step function is not useful. A sigmoidal function is therefore substituted for the step function for smoothing purposes. The formula for sigmoidal function, sig, as a function of the percentage, p, is as follows: ##EQU2## where B, T, S, and M are sigmoid shape parameters. Typical values are T=1, B=0, M=0.4, and S=25. The correlation value used is then c*sig (frac), where c is the correlation value based on the positions which are above contrast threshold.
The result is a correlation map for the realtime image versus the template. This correlation map is useful for a number of functions. The first function is to ascertain if the maximum correlation at any point on the image is above a predetermined threshold. In general, it will be appreciated that throughout the image of the gesture the correlation varies assuming that it is not exactly a 100% correlation. There will therefore be points on the image which will exhibit a higher degree of correlation versus other points. It is this maximum correlation that determines whether a triggering gesture has occurred. Thus with this process one determines whether or not a triggering gesture has occurred and more importantly the position in the scene of the triggering gesture, namely that position associated with the maximum correlation.
After having found the trigger gesture, the system then enters the tracking mode. Prior to entering the tracking mode it will be appreciated that the television may be turned off. In one embodiment, the detection of the triggering gesture turns on the television. If the television is already on, the icon overlay is actuated upon detection of the triggering gesture, which is indicated with visual feedback by the provision of the hand icon on the screen.
Once having entered the tracking mode, the portion of the scene analyzed is limited to the area surrounding that region at which the triggering gesture was originally detected. The scene processed may therefore be only 50% of the total scene or as low as 10% of the scene, depending on system parameters. The system then looks for the position of the maximum correlation on a continuous basis so that the hand's movement is tracked in X and Y directions. The system also constantly updates the hand search portion of the system to be constantly centered upon the position of maximum correlation. It will be appreciated that the tracking mode is maintained as long as the maximum correlation value is above a predetermined threshold. If the maximum correlation value descends below this threshold, the system returns to its search for a trigger gesture, with the graphical overlays disappearing once having dropped below this threshold.
It will be appreciated that the detected position of maximum correlation drives the position of the on-screen hand icon, such that there is a predictable relationship between detected hand gesture position and hand icon position. In this way there is visual feedback to the user such that when the user moves his hand in a given direction, the on-screen hand icon moves in a similar manner. It will of course be appreciated that there is a scale factor between hand movement and the movement of the hand icon on the screen.
When the tracking mode is terminated through the falling of the maximum correlation value below the above mentioned threshold, in one embodiment a different hand icon is presented to the viewer, namely a closed fist indicating that hand gesture control has ceased.
In terms of machine control, as described above, when the hand control icon overlies the machine control icon a machine control function is actuated. This is accomplished via icon coincidence detector 90 which establishes coincidence when the coordinates of the hand icon and the machine icon are identical. As a practical matter, the machine control icon occupies a given area or region, with coincidence with the hand icon being detected when the hand icon position is within the limited machine icon area. When the machine icon is a button or switch, then the coincidence of the hand icon over the machine icon switch button activates that particular button after a predetermined period of time for continued coincidence. Alternatively, when the machine control icon has a moveable element such as a slider bar, then upon the occurrence of coincidence of the hand icon with the slider for a given time it is considered that this element has been captured. After capture of this element then movement of the hand icon effectuates a like movement of the slider such that the slider is made to move in the direction that the hand icon moves, namely the direction of the hand gesture.
In one embodiment, if the hand icon moves in the direction that the slider is designed to move, then the slider is allowed to move in accordance with the hand gesture. However, if the hand icon moves in a direction orthogonal to that of the allowed slider movement, then the slider is released and the control of the slider relinquished. Thus control of the slider is relinquished by moving the hand icon in a non-allowed direction.
A detailed implementation of this process is described in the program, written in C and C++, presented hereinafter.
By way of further explanation, referring now to FIG. 4, a robust system for hand gesture recognition is illustrated in which camera 70 has its output digitized by a video digitizer 100 which in turn is coupled to an image processing unit 102 that converts the pixel representation of the image into an orientation representation. The orientation representation is in the form of unit vectors, each having an angular orientation corresponding to an edge in the image, with the edge being a rapid light/dark discontinuity. In terms of detecting a predetermined hand gesture, pixels and the image are converted to needle map unit vector representation, with the unit vector needles running in the general direction of a hand image edge. For instance, if one considers a point about an inch down from the top of the index finger, and the index finger in pointing in the given direction, then the unit vector needle will carry the angular position of this portion of the finger. What is provided can be seen by the needle map shown in FIG. 4B in which a number of points at edges on the hand are provided with the appropriate unit vector needles.
The orientation representation, as can be seen, results in needle-like unit vectors, each emanating from one point on the image of the hand and running in general in the direction of the edge corresponding to that point. It has been found, that rather than utilizing raw image intensity, color, or any other parameter, a more robust hand gesture recognition system is accomplished through the conversion of pixels to orientation representations. It is this orientation representation of the hand gesture which is initially mapped into hand template 74, in which the template is initially formed from an output of processor 102 through the utilization of a hand template generator 104. It is the purpose of this generator, when the output of processor 102 is switched thereto via switch 106, to provide a template specific to the particular individual's hand in a particular hand gesture position. The result is a needle map which is stored at 108 as a template against which incoming needle maps are compared and correlated.
Having generated such a unit vector needle map, incoming video, when converted to the orientation representation, is compared by normalized correlation unit 110 to the needle map template which is stored. Correlation values across the image are generated, with a peak selection unit 112 providing an indication not only of the degree of correlation of the particular incoming hand gesture with the stored hand template, but also that region of the scene which contains this peak correlation is identified. A purpose of the peak selection unit is therefore to identify where in the scene the hand gesture exists as well as recognizing the fact of a predetermined hand gesture. Thus peak selection unit 112 assists in the position recognition associated with unit 76 of FIG. 3.
Graphics overlay display generation is accomplished as shown at 114 to combine the functions of processor 80 and screen driver 82 of FIG. 3. Moreover, the conversion of the peak location to television commands as illustrated at 116 combines the functions of icon coincidence detector 90, icon movement detector 92 and machine control signal generator 94 of FIG. 3.
Referring now to FIG. 5 as can be seen raw video in terms of red, green, blue 24 bit representation from video digitizer 120 is utilized in the image processor 102 of FIG. 4A. After the conversion to the orientation representation of the image, background is removed by a specialized processor 122 which, as illustrated in FIG. 6, a running average of the scene is summed with the current image to provide a next average, with the running average being compared with the current image to provide for background removal. Where the running average differs from the current image by more than some threshold value, then that region of the image is determined to be a non-background region and is processed normally. Where the running average differs from the current image by less than that threshold, those regions are determined to be background and are ignored for the purposes of correlation.
The normalized correlation of FIG. 4A is utilized in correlation position unit 124 which optionally utilizes a kalman filter 126 to steady over time the position of the recognized hand gesture. The correlation score is converted to the estimated variance needed by the kalman filter by another sigmoidal function
where V is the estimated variance of the hand position measurement, and typical parameter values are: T=1.0, B=0.5, S=70, and M=0.375.
The result of background removal and kalman filtering to provide for accurate correlation position is utilized in a trigger gesture recognition unit 128 to first detect the predetermined hand gesture which is utilized to trigger the system into the control mode. This is the aforementioned "how" sign which is generated through the scanning of the entire scene for the predetermined gesture. Once having detected the trigger gesture, the system enters into the tracking mode as illustrated at 130 such that the movement of the predetermined hand gesture can be tracked to determine the movement of the aforementioned hand icon.
Referring now to FIG. 7 a computer screen window 201 is illustrated which some diagnostic functions that explain the capture of the predetermined hand gesture for an initial trigger and tracking of hand position to move the on-screen hand icon. Here a window 202 includes a rendition of the image viewed by the camera (not shown in the figure), with box 204 indicating that part or region of the image which corresponds to the hand template. It will be appreciated that this window is generated during the tracking mode in which the system has already entered into its control phase, having detected a predetermined hand gesture. Box 204 therefore indicates the location within the scene of the predetermined hand gesture. It will be appreciated that as the hand gesture moves, box 204 moves in a corresponding fashion. Should the score indicating the degree of confidence of hand recognition drop below a predetermined threshold, then the area of search for the hand is increased beyond that indicated by box 204.
Referring now to window 206 it will be appreciated that the image shown in this window, namely image 208, is that of the representation of the hand gesture in terms of orientation vectors. It will also be appreciated that in addition to the orientation representation, this screen is also utilized to analyze contrast. If the contrast is below a certain threshold value, no orientation needles are drawn in that region. Window 206 therefore permits a visual inspection as to which orientation vectors are used and which are not.
Window 210 provides a diagnostic tool to indicate the degree to which background is removed. For instance, that which is depicted in white as opposed to black are the locations which are determined to contain foreground elements. This representation is shown at 212. In general this window depicts as white what the system is looking at, with all background objects such as chairs, pictures, tables, walls, etc., being eliminated from this view.
Window 214 is a diagnostic tool in which image 216 is a representation of the degree of correlation for each point in the image with the template to which it is matched. The whiter the image at a point, the higher the degree of correlation, with an image at window 114 providing the degree of correlation to the template. Additionally, in the hand acquisition mode, this image is quite large indicating that the computer is looking over a broad region of the image to determine the existence of a trigger signal. Thereafter image 216 is considerably smaller because after having acquired the hand gesture and the location in the scene at which the hand gesture is likely to occur, the search for the position of maximum correlation can be restricted to locations near the previously calculated position of maximum correlation. The reason that the image analyzed by the system is made smaller after initial hand gesture detection is to permit the system to more rapidly track the hand motion.
Referring now to window 220, this window enables visual analysis of the closeness of the needle map associated with the template to the needle map associated with the image. Here, the needle map of the detected hand is overlain with a needle map of the template. Thus there are two needle map representations, with the first needle map being that associated with the template and the second being that associated with the region of the image which has the maximum correlation with the template. This window is thus utilized to analyze whether the system is tracking a hand or some other object which has similar unit vector orientations. The system in one embodiment provides a diagnostic image in which those needles that are closely correlated with the template are in one color, whereas those needles not so closely correlated are in a second color.
Referring to window 230, a diagnostic bar graph or thermometer representation 232 is utilized to indicate the maximum correlation value for all of the correlations of the image in window 214 such that not only is the maximum correlation value detected, its value is depicted by the thermometer indicator 234 so that a visual representation of the overall degree of correlation is presented. In the bar graph depicted it will be appreciated that a first value 236 is required for initial detection of the triggering hand gesture, whereas a second value 238 is that below which the maximum correlation value cannot descend if the hand gesture is to remain verified.
While the above invention has been described in connection with hand gestures, it will be appreciated that gestures from any body part can be utilized for machine control. In general, any object which can be manipulated in free space can perform the function of the predetermined hand gesture, and as such systems which recognize predetermined objects in a scene, along with the movement thereof are within the scope of this invention.
A program listing in C and C++ which implements the above-described system is presented in the Appendix hereto.
Having above indicated several embodiments of the subject invention, it will occur to those skilled in the art that modifications and alternatives can be practiced within the spirit of the invention. It is accordingly intended to define the scope of the invention only as indicated in the following claims. ##SPC1##
Claims
We claim:
1. A system for control of a machine having a display screen and means for controlling said machine via actuating on-screen machine control icons, comprising: means for detecting the existence of and position in space of a predetermined hand gesture and for generating corresponding existence and position signals responsive thereto; means for providing an on-screen hand icon corresponding in on-screen position to the detected position of said predetermined hand gesture in space, such that changes in position of said predetermined hand gesture result in a corresponding change in position of said on-screen hand icon, whereby the position of said on-screen hand icon is controllable by movement of said predetermined hand gesture; and, means for detecting coincidence of said on-screen hand icon with an on-screen machine control icon as said hand icon made to move over said machine control icon by movement of said predetermined hand gesture and for generating corresponding machine control signal responsive to said detected coincidence wherein said means for detecting the existence and position of said predetermined hand gesture includes: means for providing a hand gesture template including orientation vectors corresponding to orientations of edges of said predetermined hand gesture corresponding to said predetermined hand gesture; means for optically scanning a region at which said predetermined hand gesture is expected to exist and for providing orientation vectors corresponding to orientations of edges of images within said scanned region; means for correlating the orientation vectors of said hand template with orientations vectors of the images within said scanned region to provide a correlation values; and, means for providing a trigger signal when said correlation value exceeds a predetermined threshold.
2. The system of claim 1, wherein said machine control icon represents a switch and wherein said corresponding machine control signal generating means includes means for generating a switch closure signal.
3. The system of claim 1, wherein said machine control icon includes a moveable element and wherein said coincidence detecting means includes means for detecting coincidence of said hand icon with said moveable element, thus to represent capture of said moveable element, and means responsive to the movement of said hand icon for effectuating corresponding movement of said moveable element and generation of the corresponding machine control signal.
4. The system of claim 3 wherein said machine control icon is a slider bar and wherein said moveable element is a slider.
5. The system of claim 3 wherein said moveable element moves in a predetermined direction and wherein said means for effectuating corresponding movement of said moveable element is actuated only by movement of said hand icon in said predetermined direction.
6. The system of claim 1 and further including means for terminating said machine control upon detection of non-coincidence of said hand icon with said machine control icon.
7. The system of claim 5 said further including means for terminating said machine control upon movement of said hand icon in a direction different from said predetermined direction.
8. The system of claim 1 wherein said means for detecting the existence and position of said predetermined hand gesture includes: a video camera directed towards a region in space at which said predetermined hand gesture is expected to exist, means coupled to said camera for analyzing the output of said camera for the presence of said predetermined hand gesture in the scene scanned by said vidio camera and for narrowing the scene scanned by said video camera to a region around the position of a detected hand gesture when said predetermined hand gesture is detected, thereby to minimize the effect of background noise in the further detection of said predetermined hand gesture once said predetermined hand gesture has been detected.
9. The system of claim 8 wherein said predetermined hand gesture is a flat hand presented with palm facing said camera.
10. The system of claim 1 wherein said predetermined hand gesture is the "how" sign.
11. The system of claim 1 wherein said predetermined hand gesture is the "how" sign, said predetermined gesture functioning as a trigger gesture, the detection of which establishes said machine control.
12. The system of claim 11, and further including means for detecting a second predetermined gesture and means responsive thereto for terminating said hand gesture machine control.
13. The system of claim 12 wherein said second predetermined hand gesture is a clenched fist.
14. The system of claim 1 and further including means for determining a maximum correlation value and the location thereof within said scanned region.
15. The system of claim 14 and further including means for delimiting the region scanned to that region in the vicinity of the position associated with said maximum correlation value.
16. The system of claim 14, wherein the position of said predetermined hand gesture is taken to be the position of said maximum correlation, thereby to facilitate hand gesture tracking.