Method and apparatus for controlling a computing device with gestures

[0002] This invention relates generally to data input to devices and, more particularly, to finger-free operation of electronic devices.

[0003] Advances in electronics technology are leading to ever-smaller device designs with an ever-increasing variety of applications. Many “computing” devices, including handheld computers, Personal Digital Assistants (“PDA”), pagers and cellular phones, are now small enough to be easily carried or even wom.

[0004] Highly miniaturized computing devices, such as handheld computers and PDAs, have significant operational limitations as a consequence of their small size. In particular, the computer-human interface is highly constrained: the small size greatly restricts possible methods for data input. Traditional desktop computer-human interfaces, those that rely on keyboards and pointing devices, translate poorly to very small devices. Typically, PDAs lack a keyboard. Some handheld computer designs attempt to mimic desktop computer designs, however, only a miniature keyboard can be included.

[0005] In response to this problem, these computing devices generally provide for data input through a stylus and/or a limited number of buttons. Other miniature portable devices, such as pagers and cellular phones, typically rely on buttons for command input.

[0006] Because of the ease and familiarity of the graphical user interface (“GUI”) typically employed in the display of data in modern computers, for some applications it is desirable to maintain some features of a GUI in a miniature, computing device. The typical GUI relies on the computer user's input of a few command types, submitted via a pointing device and button clicks, to generate a response in the GUI. For example, pointing at a scroll bar and clicking can cause scrolling through a document or list; items in the list or in a graphical display can be selected by again pointing and clicking; and an item, such as an Internet address or a program, can be activated by pointing and double-clicking. As noted, however, command submission to the GUI in a miniature device cannot be accomplished with a “mouse” point and click approach.

[0007] Newer approaches to the computer-human command submission interface have the goal of improving the interface for computers of all sizes. One approach is to employ tilt and/or translation sensors to achieve an alternate means of data input. In “Inertial Proprioceptive Devices: Self-Motion-Sensing Toys and Tools” by C. Verplaetse, IBM Systems Journal, Vol. 35, pg. 639 (1996), the author describes devices that are able to sense and react to their own position and orientation through use of embedded inertial motion sensors. The author further describes commercially available accelerometers and gyroscopes for use as inertial motion sensors in portable computing devices.

[0008] An implementation of the position and orientation sensing approach to data input is described in “Design of Spatially Aware Graspable Displays”, by David Small and Hiroshi Ishii, CHI 97, pg. 367. The authors describe a large handheld or table positioned device that can be used to read a “newspaper” or other written document. Tilting of the device, in an intuitive manner, provides input commands for scrolling up or down, and left or right through the document: the graphical display responds to the angular orientation of the device. Translating the device closer to or further from the body of the user controls zooming of the document, in effect, scrolling in a third dimension. A button provides a “clutch” to lock and unlock the response of the display to user commands.

[0009] A much smaller prototype handheld device that also makes use of tilt for data input is described in “Tilting Operations for Small Screen Interfaces (Tech Note)” by Jun Rekimoto, UIST '96, pg. 167 (1996). This device employs a combination of: 1) tilting for input of scrolling commands to move through text or an image; and 2) button pushing for input of activating commands for navigation through the display of text or to view a portion of an image in greater detail.

[0010] The above described prototype data input methods still have the problem of requiring use of fingers for some types of command input, typically via buttons or a stylus. What is desired is a method for completely finger-free data input.

[0011] A method is provided for controlling a handheld computing device through the use of gesture commands. A gesture command corresponds to a pattern of movement of the handheld device that is measured as a function of time F(t), where t_sis a start time and tf is a finish time such that the interval of time is t_fminus t_s. The gesture command further corresponds to angular orientations of the handheld device.

[0012] The method comprises the steps of: measuring movement of the device over an interval of time with a motion sensor mounted to the device, the measuring producing a signal; analyzing the signal measured over the interval of time to determine a gesture command that corresponds to the movement of the device; and processing the gesture command to control the handheld computing device. To enable use of different gesture commands, the method includes a comparison of the signal to a catalog of gesture commands to determine a particular gesture command.

[0013] A great range of gesture commands is possible given the use of different axes and angular directions of rotation for a variety of different patterns of movement. Use of gesture commands can be combined with use of position commands to further extend the range of commands available for control of a computing device. In this manner, a computing device can be controlled without the use of buttons for command input.

[0014] The following detailed description makes apparent other purposes and advantages of the invention, with reference to the drawings in which:

[0015] FIG. 1 is a timing diagram of a series of tilt commands;

[0016] FIGS. 2a, b, c,and dare diagrams of example gesture commands in terms of angular orientation as a function of time;

[0017] FIG. 3 is a flow chart of the steps for utilizing gesture commands;

[0018] FIGS. 4a, b,and cdepict a GUI that displays an address database: 4ashows a vertical list of the alphabet; 4bshows a list of names; and 4cshows the detailed address information for one individual in the database.

[0019] FIG. 5 depicts an image display device that is controlled via gesture commands;

[0020] FIG. 6 is a flow diagram of the steps for speculative execution of a command; and

[0021] FIG. 7 depicts a wearable computer mounted to a superior dorsal aspect of a hand.

[0022] 1. A Method for use of Gesture Commands

[0023] A method is provided for finger-free command or data input through hand movement. The method can supplement position command input via hand orientation or position. The following three sections (1a,1b,and 1c) present: first, a description of gesture and position commands; second, a method for gesture command input; and third, examples of uses for gesture commands.

[0024] Note in the following that descriptions of hand movement and hand position are equivalent to descriptions of device movement and device position. It should be implicitly understood that the hand has attached to it a device that includes means for sensing movement and position, such as a motion sensor.

[0025] a. Description of commands

[0026] Commands that include hand movement are here termed gesture commands. In contrast, commands that are correlated to a particular angular orientation of a hand are here termed orientation or position commands. Position commands are given instantaneously, i.e. the position command given at any instant in time is equated to the position of the hand at that instant in time. Gesture commands are physically distinct from position commands: a gesture command requires an interval of time for submission by the hand. A gesture command comprises a time varying position expressed as a unique pattern of movement over an interval of time. A variety of gesture commands can be defined by use of different gesture movements, that is, different and repeatable patterns of movement expressed over an interval of time can be mapped to different commands.

[0027] Note the physically opposing yet complementary nature of position commands and gesture commands. A position command is submitted instantaneously: the position command at a given instant in time is simply a function of the position of the hand at that instant in time. In contrast, a gesture command requires an interval of time for submission: the gesture command is a function of a time varying position where the position varies during the interval of time. Position commands cannot be submitted in isolation. Motion through a range of positions is required to achieve a desired position of the hand. Thus, a series of position commands are submitted at the intermediate orientations that the hand passes through prior to the final desired position. In contrast, a gesture command can be submitted as a solitary event.

[0028] A position command can comprise, for example, a fixed angular orientation relative to a particular axis of rotation. A gesture command can comprise, for example, an angular orientation that varies over an interval of time relative to a particular axis of rotation. FIG. 1 graphically depicts an example of a series of position commands submitted during a time sequence. A position command in this example corresponds to the angular orientation, θ, about an axis of rotation. At each instant in time, the position command corresponds to the value of θ at that instant in time.

[0029] At the beginning of the time sequence in FIG. 1, the hand is oriented in a “neutral” position: the neutral position can be set at θ equal to zero or any preferred value. The neutral position is a rest position, that is, no command is submitted in the neutral position. The neutral position can be established in a number of ways. For a device that is receiving gesture commands, the position of the device at the time the device is turned on can be defined as the neutral position. Alternatively, the neutral position can be reset through submission of commands to the device.

[0030] As the time sequence progresses in FIG. 1, a series of position commands of increasing θ are submitted. The series of commands reach a plateau in the value of θ and eventually decrease in the value of θ as the hand returns to the neutral position. The fine scale undulations in the series of position commands represent the slight random movement or “noise” that would typically be exhibited by a hand.

[0031] FIGS. 2a, b, c,and dshow examples of four different, single gesture commands. Each of the example gesture commands is expressed over an interval of time commencing at ts and concluding at tf where the angular orientation about an axis of rotation varies during the interval of time. The example in FIG. 2ashows a gesture command that comprises a rotation in a positive angular direction followed immediately by a return approximately to the starting position. The example gesture command of FIG. 2bcomprises a pattern of movement similar to that in FIG. 2athough repeated twice in immediate succession. FIG. 2cshows a gesture command that is similar to the command that is depicted in FIG. 2athough possessing a brief delay prior to returning to the starting angular orientation. Lastly, FIG. 2dshows a gesture command that commences with a movement in a negative angular direction. Note that the gesture commands of FIG. 2aand FIG. 2dhave patterns of movement that are mirror images of each other. More generally, any pattern of movement has a related mirror image pattern of movement. Different patterns of movement, such as the four described above, can be correlated to different gesture commands. A computing device that makes use of gesture commands can employ different gesture commands to obtain different responses of the computing device.

[0032] The above-described position and gesture commands can be further generalized. Position commands can correspond to angular orientations relative to any of three orthogonal axes. Gesture commands can correspond to a pattern of movement relative to one of the three orthogonal axes. More complex patterns of movement can include rotational movements about two or three of the three orthogonal axes. Hence, a great variety of gesture commands are possible.

[0033] b. Method for gesture command submission

[0034] FIG. 3 depicts steps 100 for input of gesture commands to a computing device. In step 110, movement of the computing device is measured over an interval of time. A motion sensor attached to the device measures the movement and generates a signal that corresponds to the movement. Analysis of the signal in step 120 determines if a gesture command was submitted during the interval of time.

[0035] If the computing device makes use of more than one gesture command, the particular gesture command that was submitted is determined in step 121. This determination is accomplished by comparison of the gesture that was submitted during the interval of time with a catalog of gesture commands. A match between the submitted gesture and a gesture command in the catalog serves to identify the submitted gesture command. Allowance is made for the minor variations in movement that are typically expected for repeated hand movements, i.e. a perfect match is not required to determine the gesture command.

[0036] In step 140, the gesture command is processed to control the computing device. The actions of steps 110 through 140 can be repeated to permit submission of a series of gesture commands.

[0037] A series of position commands can be submitted along with a series of gesture commands. A computing device can employ a single motion sensor to measure all commands or multiple motion sensors can be employed.

[0038] c. Examples of the use of gesture commands

[0039] Different position and gesture commands can be employed to initiate different types of responses in a computing device. For example, position and gesture commands can be used, respectively, to initiate select and activate responses in a computer's GUI. A GUI that provides access to an address database serves as an example of a very simple implementation. FIGS. 4a, b,and cdepict the appearance of the GUI in a computer display screen at three different moments in time. FIG. 4ashows the GUI displaying the alphabet in a vertical list. A highlighted “selection” band highlights the letter “E”. The computer user scrolls through the alphabet until the desired letter is highlighted. During scrolling, the selection band can first move to the top or bottom of the list followed by scrolling of the list. Alternatively, the selection band can reside at a fixed location in the GUI with the text doing all the scrolling.

[0040] In response to a series of position commands, the list scrolls vertically. Preferably, as indicated, the scrolling position commands would be a function of angular orientation relative to the pitch rotation axis. The direction of scrolling, i.e. up or down, is determined by the angular orientation of a position command, i.e. whether the angular orientation is positive or negative. The neutral position, e.g. θ equal to zero, is determined as previously described.

[0041] In one alternative, the magnitude of e determines the scrolling rate, as the table below shows.

[0042] To prevent a response to small movements, including noise, angular orientations of very small magnitude, θ<θ₁, are ignored. Angular orientations of a magnitude within a first angular range, θ₁<θ<θ₂, produce slow, stepwise scrolling, to ease selection of a particular letter. Angular orientations of a magnitude within a second angular range, θ₂<θ<θ₃, produce continuous scrolling with the rate of scrolling increasing with an increasing magnitude of the angular orientation.

[0043] Once the selection band highlights the desired letter, a gesture command, preferably about the roll axis, as indicated in FIG. 4a,serves to activate that letter. In the example of FIG. 4b,gesturing for the letter “S” shifts one level downward in the database hierarchy: the display then presents a vertical list of individuals whose last names begin with the letter “S”. Scrolling through the name list proceeds, as above, via position commands until the desired individual's name is found and placed at the selection bar. A second gesture command activates the name, with the same sort of activation response as described above: a shift downward one level in the database hierarchy. The details of the individual's personal information are then displayed as depicted in FIG. 4c.

[0044] Many variants in the GUI's response are possible through the use of different gesture commands. For example, a clockwise fanning pattern of movement and a mirror image counterclockwise fanning pattern of movement can provide down-shift and up-shift activation commands. Gestures about a is second axis, e.g. the pitch axis, can provide commands to lock and unlock the GUI, that is, responses by the GUI can be temporarily shutoff. In an alternative where two-dimensional lists of items are displayed, position commands relative to a second axis, preferably the roll axis, can be used to initiate horizontal scrolling of the lists. Activation of items through gesture commands can be accomplished as described for one-dimensional lists.

[0045] Alternatively, the various motion commands can provide a large range of responses in more sophisticated GUIs than those described above. GUIs with pull-down menus are accommodated with a similar command scheme. Scrolling in two-dimensions is again initiated through position commands about two axes until a desired menu item is highlighted. A gesture command then serves to activate or “open” the highlighted menu item. A highlighted selection bar within the pull-down menu can then be scrolled to a desired menu item and the item activated with a gesture command.

[0046] An example of an image display device 700 that makes use of position and gesture commands is depicted in FIG. 5. The display device 700 is handheld and includes motion sensors 740 for receipt of position and gesture commands. The display device can include a button 730 as an on/off switch. The angular orientation of the display device 700 can be changed about the roll axis 751 and the pitch axis 752 for submission of position and gesture commands.

[0047] The display device 700 is used to observe images that are stored in a database. Scrolling position commands are used to scroll from image to image displayed on a display screen 710 until an image of interest is found. A gesture command then activates the displayed image for more detailed viewing. Additional gesture commands serve to “zoom” in to observe the selected image in greater detail or to zoom out to observe a greater portion of the image or the entire image. Position commands, again, serve to scroll the image across the display screen 710.

[0048] The above described method for input of gesture commands can be applied in the operation of a variety of computing devices, in particular, wearable versions of computing devices. Such devices include watches, televisions, radios, cellular phones, pagers, and two-way radios. For example, a cellular phone could be turned on and off and phone numbers selected and dialed, all without use of buttons. Further, gesture commands can be generalized to include translational movements in addition to rotational movements.

[0049] 2. Speculative Execution of Commands

[0050] As depicted in FIGS. 2a, b, c,and d,submission of a gesture command is initiated at time t_s. Complete submission of the gesture command, however, requires a finite interval of time Δt equal to t_fminus t_s. Therefore, the existence and identity of the command cannot be determined until an interval of time of length Δt after submission of the command begins. Confusion in command processing can occur during the interval of time. For example, a portion of a gesture command can be interpreted to be a series of position commands and acted upon as such prior to completion of the gesture command.

[0051] FIG. 6 depicts the steps 200 in a method for speculative execution of commands. The method serves to produce an expected response in a computing device in response to submission of a gesture command. The method is also applicable to any command types that require an interval of time for submission.

[0052] At step 210, device movement is measured via a motion sensor attached to the device. A signal that corresponds to the movement is produced. At step 220, the signal that was produced during the interval of time is analyzed to determine if it corresponds to a gesture command. If a gesture command is determined at step 220, step 230 then occurs: the gesture command is processed as if it had been processed at time t_s.

[0053] Position commands can be utilized in parallel with gesture commands. In step 241, the signal is analyzed to determine a series of position commands. At step 242, each position command is processed upon determination. If a gesture command exists during the interval of time, the various orientations present in the gesture command may falsely be determined, at step 241, as position commands. The false position commands will then inappropriately be processed at step 242. The confusion is corrected at step 230. By processing the gesture command as if it had been processed at the beginning of the interval of time, the effects of the false position commands are rejected. That is, the gesture command controls the device as of the orientation that existed at the time t_s.

[0054] Steps 211 and 231 as shown in FIG. 6 indicate one implementation of the above method. At step 211, the state of the device is recorded. Recording is done continuously. At step 231, recorded state data of the device is provided for the processing of the gesture command performed at step 230. State data that was recorded at time ts provides a reference for the appropriate processing of the gesture command in step 230.

[0055] A simple example of the recording of state data can be given with reference to FIG. 4a.A series of position commands would cause scrolling of the alphabet that appears in the GUI displayed in FIG. 4a.As the display scrolls, the state of the display would be recorded over time. That is, the letter that was highlighted at a particular point in time in the past could be determined by recalling the recorded data for that particular point in time. After submission of a gesture command, the gesture command could then be correctly applied to the letter that was highlighted at the beginning of the gesture command by recalling the state data for that time.

[0056] 3. Button Free Wearable Computer

[0057] FIG. 7 depicts a wearable computer 300 mounted on a superior dorsal aspect of a hand. The wearable computer 300 includes a motion sensor 330. The wearable computer 300 is handheld in the sense that it is mounted to the hand. With this mount location, convenient wrist and forearm rotational movements can be used to move the computer 300 about three orthogonal axes of rotation. Thus, a variety of position and gesture commands can be easily submitted to the wearable computer 300. A range of command input is available, through use of gesture commands, without recourse to buttons or other options that would require use of fingers.

[0058] The computer 300 can include a display screen 310 for display of data such as the GUI described above in Section 1 part c. Alternatively, the display screen 310 can display the image database described above in Section 1 Part c. Given the superior dorsal aspect location of the display screen 310, the display screen 310 is readily visible and conveniently located. The wearer can observe the display screen 310 with the wearer's hand held in a comfortable position and without interference from typical clothing. The fingers of the hand are free for other activities, such as holding a pen. Buttons 320 can be included in the computer 300 to provide alternative options for button entry of commands, e.g. on and off commands.

[0059] In one embodiment, as depicted in FIG. 7, the components of the wearable computer 300 are housed in an enclosure that possesses an approximately triangular shape with rounded vertices to conform to the superior dorsal aspect portion of the hand. The underside of the enclosure can further be shaped to conform to the surface of the superior dorsal aspect portion of the hand. Soft or flexible material can be employed in the enclosure to increase the comfort of the user. Further, the material can be textured or colored to disguise the presence of the computer 300 on the user's hand.

[0060] Alternative locations for the wearable computer can be envisioned. Placement on a foot would still permit movement about three axes as would placement on a head. Thus, an individual with diminished use of arms or arms and legs would have other options for interaction with a computing device through body movements. Placement at other locations on a hand would still permit movement about all three axes though without the convenience specific to the superior dorsal aspect location.

[0061] The mount that positions the wearable computer 300 at the superior dorsal aspect location can include various features, such as: straps; an adhesive; suction cups; a ring; a bracelet; or the computer 300 can be built into a glove.

[0062] Alternatively, components of the wearable computer 300 can be placed at different locations on or off of the body. For example, the motion sensor 330 can be mounted to the superior dorsal aspect of the hand while the display screen 310 is mounted elsewhere on the body or is placed off of the body.

[0063] Rather than a computer, the wearable device alternatively can be one of a variety of computing devices, e.g. a cellular phone, a pager, a two-way radio, or a multifunction wristwatch. If included, the display screen 300 can display data relevant to the particular computing device.

[0064] The above embodiments are merely illustrative. Those skilled in the art may make various modifications and changes that will embody the principles of the invention and fall within the scope of the invention.