Hand Gesture Control System

This invention relates to computer graphics and, more particularly, to a hand recognition computer control system.

It will be appreciated that the control of onscreen images in a computer environment is desirable in order to provide realtime interaction. One of the more difficult problems associated with the direction of character movement is when one seeks to optically detect hand gestures in real time. The time necessary to acquire or recognize the hand gesture, both in a cluttered environment and against a natural background, can exceed several seconds, a relatively long time for interactive applications.

During optical recognition of the hand gesture, it is oftentimes necessary to compare the digital images of the hand with a template so that a match can be made between the predetermined template and the hand gesture. In so doing, many multiplications must be performed resulting in a relatively large lag time for hand gesture recognition.

One such hand gesture recognition system is described in U.S. patent application Ser. No. 08/391,955 now U.S. Pat. No. 5,594,469 by William T.

Freeman and Craig D. Weissman, filed on Feb. 25, 1995 incorporated herein by reference. While the system described therein operates satisfactorily to uniquely determine particular hand gestures, there is nonetheless a need for a more simplified system which operates more quickly and accurately to detect hand gestures.

It is also a requirement, especially in the games industry, for an exceptionally inexpensive method for hand gesture recognition.

In order to accomplish hand gesture control in an efficient, time saving, and inexpensive manner, in the Subject System the hand is characterized by image moments which refer in the subject case to spatially weighted averages of image intensities. From these image moments can be found a corresponding rectangle which has the same set of image moments. The hand gesture is characterized through the orientation, position, length and width of the corresponding rectangle. This provides an exceptionally fast recognition system for a large variety of hand gestures without having to resort to complicated image processing.

For instance, for graphical object control in which onscreen characters are to be controlled by a participant or user, it is desirable to eliminate the necessity for a mouse or a joystick in favor a readily available item, namely one hand. The ability to be able to sense hand gestures for the control of the game eliminates the requirement for costly input apparatus. More importantly, with a joystick there are only two degrees of freedom which can be controlled; but with the hand there are multiple types of actions that can be expressed through articulation of the hand. As a result, multiple parameters such as orientation, horizontal and vertical position, and projected length and width can be sensed.

In order to provide game control or other onscreen graphical object control, in one embodiment hand gestures are sensed optically through the utilization of a camera with the image being converted into a digital representation such that the position of the image of the hand, its length and width, and its orientation are detected. This provides five different measurements of the hand gesture, namely horizontal and vertical position, projected length and width and orientation. Rather than sensing the exact 3-d configuration of the hand, it is a finding of this invention that computer control can be robustly provided through the above criteria.

In particular, the X projection and Y projection of the image are calculated along with the diagonal projection followed by the generation of image moments through a specialized calculation so as to derive a rectangle having a given position, length and width, and orientation, with the characteristics of the rectangle then providing a robust simplified means of detecting a particular hand gesture.

In one embodiment, a specialized trigger gesture, which is easily recognized, involves the providing of a circle with the thumb and the forefinger to provide a hole. This type of gesture is very easily recognized by the subject system and provides the system with a signal indicating that the user has initiated the trigger gesture.

This same hand gesture can be utilized to provide a trigger for any action which the computer is to execute, including, for instance, the firing of a gun, the initiation of a rocket blast off sequence, or in fact any other machine action.

In a further embodiment, not only are hand gestures easily detected through the utilization of the image moment calculation, additionally, body movement can be detected through the detection of the moments of the image and analysis of the five measurements derived from the equivalent rectangle. Noting that the center of the rectangle is the center of mass of the object, one can detect the center of mass and therefore its movement. In this instance, one is looking at only one component of the rectangle to provide onscreen motions of a character.

More particularly, simple position and size measurements of an object can be calculated by measuring image moments. This assumes that the object of interest, normally the hand, dominates the image and that the background is uniform or can be subtracted out. The following equations detail the weighted average that is utilized to calculate image moments. If I (x, y) is the image intensity at position x, y, then the image moments up to second order are: ##EQU1##

We can find the position, x.sub.c, y.sub.c, orientation .theta., and width l.sub.2 and length l.sub.1, of an equivalent rectangle which has the same moments as those measured in the image [1]. Those values give a measure of the hand's position, orientation, length and width. We have: ##EQU2##

Note that in Equations 1-5, the measurements corresponding to these equations are based on summations over the entire image, so that they will be robust against small changes in the image. The five abstracted parameters are independent of the overall contrast of the hand against the background. The calculation of Equations 2-5 are done just once per image, not once per pixel. The integrals of Equation 1 require five multiplies and six additions per pixel, the quadratic factors in x and y being precomputed.

Note, with respect to a camera field of view including one's arm because one edge of an arm will always be off the edge of the picture, the length of the hand, L.sub.1, is then coupled to the Y.sub.c position. Thus, the information provided by L.sub.1 is redundant with the information provided by Y.sub.c. Thus, one can use the L.sub.2 measurement for such camera field of view, but not the L.sub.1.

While the above has been described in terms of a generalized processor, an artificial retina chip such as described in K. Kyuma, E. Lange, J. Ohta, A. Hermanns, B. Banish, and M. Oita, Nature, Volume 372, Number 197, 1994 can be used to generate some image projections as opposed to calculating them with a general processor. These image projections can then be utilized by a microprocessor to calculate the image moments. When the artificial retina chip is coupled to a microprocessor, this provides for fast calculation of the image moments. Thus, by utilizing the two projections of the image as available from an artificial retina chip, one can calculate image moments without having to resort to the utilization of the double sums of the previous equations or resorting to microprocessor calculation of all three image projections. As a result, Equations 6-9 presented hereinafter show how 1-d integrals can be utilized in the computation of the image moments.

Let the vertical, horizontal, and diagonal projections be: ##EQU3## and

Then the image moments can be computed from those projections by [1]: ##EQU4##

These single sums will be faster to compute than double sums of Eq. (1).

The result is that hand gesture recognition or indeed body position recognition can be accomplished in milliseconds as opposed to several seconds.

In summary, a system is provided for rapidly recognizing hand gestures for the control of computer graphics, in which image moment calculations are utilized to determine an overall equivalent rectangle corresponding to hand position, orientation and approximate shape, with shape in one embodiment correlating to the length and width of the hand. In a further embodiment, a hole generated through the utilization of the touching of the forefinger with the thumb provides a special trigger gesture recognized through the corresponding hole in the binary representation of the hand. In a further embodiment, image moments are utilized to determine the center of mass of an image for controlling or directing onscreen images.

These and other features of the Subject Invention will be better understood taken in conjunction with the Detailed Description in conjunction with the Drawing of which:

FIG. 1 is a block diagram of an artificial retina chip embodiment of the subject invention in which hand gestures are detected and decoded for game control;

FIG. 2 is a flow chart illustrating the process of detecting hand gestures and providing game control signals for graphics generation;

FIGS. 3A-3E are photographs of the digitized pictures of selected hand gestures and rectangles corresponding to the gestures;

FIGS. 4A, 4B, and 4C are respective pictures of the digital image of selected hand gestures and the control of an onscreen graphical object, in this case a car, with the orientation of the car controlled by the orientation of the hand and the speed of the car controlled by the width of the hand;

FIG. 5 is a block diagram illustrating the detection of a special trigger gesture in the form of a hole for game control;

FIG. 6 is a series of photographs of the digital image of a hand generating the special hole trigger gesture, including the corresponding rectangle, with the detection of the trigger gesture indicated by a change in the fill pattern of the rectangle;

FIG. 7 is a diagrammatic representation of the image projections for a digital image of a hand, indicating by the circled text whether that projection can be calculated by the artificial retina chip (AR) or the microprocessor (uP); and,

FIGS. 8A-8E are a series of pictures and diagrams indicating the image of a human being as detected by an artificial retina chip, its corresponding rectangle, and the corresponding action that an onscreen image of the individual's foot is to represent upon detection of a particular characteristic of the detected image, namely, "center of mass" location.

Referring now to FIG. 1, in one embodiment, hand gesture recognition includes a system for scanning an area including a camera having a lens 10, an artificial retina chip 12, and a microprocessor 14 which is utilized to process the images to provide for the aforementioned image moment detection and control signals which are in turn applied to a game control unit 16 for driving a graphics generation unit 18 that provides the appropriate graphics on display 20.

As mentioned hereinbefore, it is the purpose of the subject invention to rapidly determine characteristics of a hand or other object in the field of view of lens 10 so as to be able to provide for the requisite game control, or in fact, any type of computer control display.

More particularly, and referring now to FIG. 2, that which is the output of microprocessor 14 is the position; orientation and dimensions of the equivalent rectangle. In order to provide for this output, as can be seen at 22, in one embodiment, artificial retina chip 12 calculates the x projection. The y projection of the image is calculated at 24. These two calculations can either be done separately or in the illustrated embodiment performed by processors carried by the artificial retina chip.

While these two projections are necessary for the calculation of image moments, microprocessor 14 is utilized to calculate the diagonal projection as illustrated at 26 so as to complete an image moment calculation at 28 in accordance with Equations 6 through 9. After having calculated image moments, the rectangle corresponding to these image moments is calculated at 30 such that the resulting rectangle, analyzed as to orientation, position, length and width, provides requisite signals for game control unit 16 to be able to drive the appropriate graphics generation unit 18 so as to provide the requisite on-screen display. It will be appreciated that the calculation of the rectangle as illustrated at 30 is accomplished by Equations 2-5.

What is therefore accomplished through the utilization of an artificial retina chip is a rapid means of calculating image moments through precalculation of x and y projections by the artificial retina chip, followed by a calculation of the diagonal projection in the microprocessor. It will be appreciated that the calculation of the diagonal projection is accomplished in accordance with Equation 8.

It will be appreciated that what is meant by an x projection and a y projection is merely the sum of all of the image intensities in the x direction and the y direction respectively. For instance, for the x projection, one merely adds up all of the pixels in every column, yielding one sum for every column. The same is done for every row such that with its orthogonal representation, the sums of image intensities may be utilized in the derivation of the corresponding rectangle. It will be appreciated that the sums along each of the orthogonal axes, plus the diagonal axis is used by Equation 9 to calculate the image moments.

Referring now to FIGS. 3A-3E, the resulting rectangles for the corresponding images are shown. For instance, in FIG. 3A, the image as shown at 40 is a side view of a hand at a given orientation as illustrated by a center line 42. It will be appreciated that the resulting rectangle 44 is oriented along the same center line, but has a width 46 corresponding to the width of the hand, and a length 48 corresponding to the length of the hand.

As can be seen, while images in themselves are rather complicated in nature, they can be characterized in a simple way through the rough size and shape of a corresponding rectangle. As can be seen in FIG. 3B, a side view of hand 50, having a center line 52 is characterized by rectangle 54, having center line 52, a length 56 and width 58. The image of a flat hand illustrated at FIG. 3C at 60, is characterized by a corresponding rectangle at 62 whereas a thumbing gesture, as illustrated by image 64, is characterized by a corresponding rectangle 66. Referring to FIG. 3E, the image of FIG. 3D is shifted to the right as illustrated 68, with a corresponding change in the position and orientation of the corresponding rectangle 70 being depicted.

Having described a relatively simple way of rapidly detecting hand gestures, it will be appreciated, as illustrated in FIGS. 4A, 4B, and 4C, that different hand gestures can be utilized to create different positions and motions of an on-screen image. For instance, the downwardly projecting hand 80 causes the car 82 to be oriented in a direction corresponding to hand 80, whereas hand position 84 causes car 82 to move in the direction indicated by the hand. However, a flat hand 86 causes car 82 to speed up in accordance with the width of the hand. What will be appreciated is not only can the direction of on-screen images be changed in accordance with hand gesture control, other characteristics of the image such as its velocity can also be changed through hand gesture detection.

Referring to FIG. 5, as part of hand gesture recognition in accordance with the subject invention, it is possible to detect a unique trigger gesture, which is then utilized to enter into a programming sequence or to provide some other computer control regime. In this case, an image is detected at 90, is binarized at 92, with holes in the binarized image being counted at 94 by a method such as described in [1] such that for game control, the number of holes is utilized via a control unit 96 to control screen display 98. The holes in this case are those generated through the touching of thumb with the index finger to form a circle. This is a highly recognizable image. In one embodiment, the trigger gesture can be used to control the shading of a particular rectangle. For instance, as illustrated in FIG. 6, with the thumb and the forefinger not touching, the equivalent rectangle 100 is portrayed as being opaque. Rather, when the trigger gesture is completed by the touching of the thumb with the forefinger as illustrated at 104, the resulting rectangle 106 may be shaded or colored differently. The difference in shading of the rectangles indicates the detection of this easily recognized trigger gesture.

Referring now to FIG. 7, given an image at 110 of a hand, the horizontal image projection, the vertical image projection, and the diagonal image projection are as illustrated respectively by curves 112, 114, and 116. The processing in order to generate these curves is, as mentioned before, in the artificial retina chip for both the horizontal and vertical projections, with the processing being in the microprocessor for the diagonal projection.

As mentioned hereinbefore, and referring now to FIGS. 8A-8E, it is possible for the subject system to recognize images other than hand. What will be seen from FIGS. 8A-8E is that a camera including lens 10 and artificial retina chip 12, is viewing a scene 200 in which an individual posed so as to present different aspects of the individual's body as shown by image 202. In a skate boarding situation, the individual shifts his weight to his right on a board in accordance with his body position as illustrated in FIG. 8A. The corresponding position of the individual is illustrated by rectangle 204. As illustrated to the right of this rectangle is a side view of the individual's leg 208 on skateboard 210, with the skateboard moving to the left (the individual's right) as illustrated by arrow 211.

Alternatively, as illustrated in FIG. 8B by the individual's position in image 212, the corresponding rectangle 214 indicates that the individual is standing straight up on the skateboard, with the individual's leg 208 as illustrated, such that the board moves straight in the direction 216.

As illustrated by image 220, the individual seeks to have his skateboard moved to his left by leaning to the left. This is indicated by the corresponding rectangle 222, in which the leg 208, as illustrated in FIG. 8C, leans to the individual's left (on-screen right) with the skateboard going on-screen right as illustrated at 224.

As illustrated by image 230, it is oftentimes desirable to hop the skateboard by jumping vertically. This is indicated the corresponding rectangle 232, which causes the on-screen image of the skateboard, along with the individual here illustrated at 240, to execute a jumping maneuver.

It will be appreciated that in FIGS. 8A, 8B, and 8C the position of the rectangle, x.sub.c and y.sub.c of equation 2, the so-called "center of mass" of the rectangle, is within a given distance from the center of cross hairs 242 and 244. However, when this center of mass rises vertically as illustrated by rectangle 232 outside a predetermined threshold distance established by the system, the subject system recognizes this position of the rectangle as being one indicating a jump. Thus, when the center of mass of rectangle 232 exceeds a predetermined vertical threshold distance from the cross hairs, the jump sequence is executed on-screen.

As can be seen by image 250, the individual is in a crouch position. This crouch position is reflected by the center of mass of rectangle 252 being below the intersection of the cross hairs by an amount which exceeds a predetermined threshold. This being the case, the image portrayed on-screen is that as illustrated at 254 to be a skateboarding individual 256 dipping down under a bridge-like structure 258.

What will appreciated is that the so-called "center of mass" of a rectangle, can be determined by Equation 2. This center of mass is then utilized to determine the motion of the on-screen image or, in fact, which on-screen image will be presented.

By center of mass of an image is meant the position in the image that would be the center of mass of an object which had a local mass density proportional to the image intensity at each pixel. The program for generating image moments and on-screen image control is now presented.

Having now described a few embodiments of the invention, and some modifications and variations thereto, it should be apparent to those skilled in the art that the foregoing is merely illustrative and not limiting, having been presented by the way of example only. Numerous modifications and other embodiments are within the scope of one of ordinary skill in the art and are contemplated as falling within the scope of the invention as limited only by the appended claims and equivalents thereto.