User input for a computer

[0002] This invention relates to user input for a computer.

[0003] In a video-enhanced interactive computer program, for example, video sequences are displayed to a viewer as part of the experience of using the program. The selection of video sequences to be played is made by the program based on actions taken by the user. The user actions may be taken using a mouse or joy stick. To enhance the interactive experience of the user, it has been proposed to enable the user to indicate actions by manipulating a hand-held wand that has no cable connection to the computer. The wand has a light that can be turned on and off by the user and the computer has a digital camera and software that can detect the turning on and off of the light from the output of the digital camera. When the user turns on the light at a moment that corresponds to an intended action, the computer can change its choice of the next video sequence to be displayed or perform any of a variety of other actions. By continually receiving the user's actions in the form of light signals from the wand and altering the selection of video sequences played back to the user, the computer program is able to provide a highly interactive experience for the user.

[0004] An embodiment of the invention enables, for example, motion of a wand to be detected automatically by accelerometers, which makes the user's interaction with a video-enhanced interactive computer program more intuitive.

[0005] In general, in the invention, information is determined about acceleration of a user manipulated input device (using, for example, an accelerometer). A wireless communication element (e.g., a lamp) is controlled in response to the acceleration information. A computer presentation of video information to the user is controlled in response to signals of the wireless communication element.

[0006] FIG. 1 is a block diagram of a wand and a computer. Accordingly to an embodiment of the present invention.

[0007] FIGS. 2 and 3 are flow charts according to an embodiment of the present invention.

[0008] FIG. 4 is a back diagram of another embodiment of the present invention.

[0009] As seen in FIG. 1, a hand-held wand 10 used as an input device has a hand-sized housing 12 with a lamp 14 (a radiation source) at one end. A digital camera 16 is positioned to capture an image that includes light from lamp 14. Image data from the digital camera or an analog camera with an A to D converter is delivered to a computer 20. The computer is also connected to a video display 22. Video display 22 is visible to a person who is using the wand 10.

[0010] The computer includes a microprocessor 28 that executes an operating system 30 and application software 32 (stored in memory 33), such as a video-enhanced interactive drum playing program that provides appropriate video sequences of a drum being played. The video sequences selected for play depend on motions of the wand as it is manipulated by the user's hand to simulate the beating of a drum. A wide variety of application software could respond to the wand motions in a variety of other ways. For example, the computer could play a range of musical sounds or could speak (using speech synthesis) or move an object on the display.

[0011] The wand contains a set of accelerometers 40 that detect acceleration of the wand in any of three orthogonal axes. Acceleration signals are sent to a controller 42 (e.g., a microprocessor or special-purpose logic) where they are processed to determine instants or periods when the lamp should be flashed or continuously lit. A program 44 could also derive velocity and position information from the acceleration data. The program uses the instantaneous acceleration information (and could also use the time-dependent acceleration information and velocity and position information) to effect a pattern of lamp illumination that is appropriate to enable the user to indicate intended actions as part of his interaction with the application software being used.

[0012] The lighting pattern is determined automatically by the program in the wand without the user needing to do anything other than move the wand in a way that is relevant to the functions of the application program.

[0013] For example, in the case of the drum playing program, the program in the wand determines when a rapid deceleration of the wand occurs at the end of a downward stroke, indicating that the user is intending to “strike” a surface of a drum being displayed. The lamp would be flashed automatically. The flash is captured in images delivered from the digital camera to the computer. Image processing software 50 in the computer analyzes the images to detect the occurrence of the flash of light. In response to the flash, semantic inference software 52 also running on the computer infers that the flash meant a user drum beat and would generate a trigger to the application software. The application software plays a video sequence showing the drum head being struck and a drum beating sound is produced.

[0014] In a simple case, each drum beat of the wand by the user triggers a single flash, and the drum program displays a single beat video sequence with a corresponding single drum beat sound. In a more complicated case, the positions of the lamp flashes in a succession of images captured by the digital camera are analyzed to infer drumming of two or more different drum heads, cymbals, or other percussion instruments. The video sequence and sound for a given beat depend on which instrument the user is striking.

[0015] Referring to FIG. 2, the program at the wand begins 69 by computing and storing acceleration, velocity, and position information 70 based on acceleration readings from the accelerometers. Sequences of readings could be stored over time. Next, the program compares the stored information with instantaneous thresholds or time sequences of values to determine whether the lamp should be lit 72. In the simple drum beating example, the threshold could represent an amount of deceleration and a simple comparison could be done with the deceleration value received from the accelerometers. Next the lamp is either lit 74 or the processing iterates, as appropriate.

[0016] As seen in FIG. 3, at the computer, the processing of information received from the wand begins 79 with the image processing software determining and storing the times of occurrence (and, for more complex schemes, the positions and durations) of the sequences of lamp flashes 80. Next, the semantic inference software uses stored rules, thresholds, and matching sequences to infer 82, for a given application, the intended user meaning of the information received from the wand. The intended meaning would be conveyed to the application software 84. The application software would then control the selection of video sequences and sounds 86 to be performed for the user. In the simple case of the drum beating, the semantic software could simply translate the receipt of a light flash to a display of the video sequence for a drum beat.

[0017] The wand may include a manual switch 60 that would enable a user to trigger the controller to light the lamp at desired times for non-automatic control of the interface.

[0018] The manner in which the control program in the wand determines the illumination pattern for the lamp may depend on the functions which the wand is to perform for a given application. For example, the control program could effect different illumination patterns for a drumming program, an orchestra conducting program, and a ping pong game. The choice of programs could be made using a selection switch 21 controlled by the user.

[0019] Similarly the semantic inference engine could use different algorithms to infer user intended actions as appropriate for different application software. The wand includes a battery 61 for powering the electronics and lamp. The computer includes storage 53, I/O systems 55, and other conventional components, not shown.

[0020] Other embodiments are within the scope of the following claims.

[0021] For example, the interactive device could be in other forms than a wand including any useful or fantasy object or a representation of any useful or fantasy object, such as a steering wheel, a baton, a doll, a bat, a ball, a drum stick, or a piano keyboard to name only a small number. The device need not be small enough to be hand held. Other forms of radiation could be used to convey information from the wand to the computer, including infrared, sound, or radio frequency waves.

[0022] Special purpose logic could be used in the wand instead of a microprocessor.

[0023] The wand need not use accelerometers to detect acceleration or deceleration. For example, as seen in FIG. 4, the lamp 88 could be constantly illuminated and hidden from the camera by a mechanical shutter 90. The shutter is attached to the housing 92 of the wand by a spring, holds on weight 96, and is free to move upward and downward 98. The spring and weight are chosen so that in normal use, sharp deceleration of the wand motion causes the shutter to open and shut with a similar effect to the embodiment of FIG. 1 and without the need for as much electronic circuitry.

[0024] The wand could be arranged to flash your acceleration rather than deceleration.