Method and apparatus for sensing body gesture, posture and movement

[0001] 1. Field of the Invention

[0002] The present invention relates to a method and apparatus for sensing the gesture, posture and movement (GPM) of the head, limbs, limb parts and other parts of the body.

[0003] 2. Background Information

[0004] Current techniques for tracking, or establishing, the gesture, posture or movement of a body or body part can be summarized into two categories. The first category comprises those systems that utilize external reference sources such as magnetic waves reflected off the body to detect the position or movement of body parts. The second category comprises those devices that use sensors on the body; such as gyroscopes, accelerometers, potentiometers, hand-gloves and jackets; for detecting the position and dynamic motion of a body part.

[0005] The problem with known systems is that they require some external reference to the body, and/or require a large and awkward attachment to the body part in order to perform their task.

[0006] It is an object of the present invention to provide a method and apparatus for determining the gesture, posture and movement of a body part of a person that is small and causes less discomfort to the person.

[0007] It is yet a further object of the present invention to provide a method and apparatus that overcomes or ameliorates problems with known systems, or that at least provides the public with a useful alternative.

[0008] According to a first aspect of the invention there is provided a method for indirectly assessing the gesture, posture or movement of a body part of a person including:

• • [0009] applying an ultrasound transmitter and receiver to a body part to be monitored,
  • [0010] using the transmitter to transmit an ultrasound signal into the body part and receiving a reflected and/or scattered ultrasound signal at the receiver,
  • [0011] manipulating the reflected and/or scattered ultrasound signal to obtain parameter data, and
  • [0012] comparing the parameter data to reference information to obtain gesture or posture information for the body part.

[0013] According to a second aspect of the invention there is provided a method for indirectly assessing the gesture, posture or movement of a body part of a person including:

• • [0014] applying an ultrasound transducer or transducer array to a body part to be monitored,
  • [0015] using a transmitter to transmit an ultrasound signal into the body part and receiving a reflected and/or scattered ultrasound signal at the receiver via the ultrasound transducer or transducer array,
  • [0016] manipulating the reflected and/or scattered ultrasound signal to obtain parameter data, and comparing the parameter data to reference information to obtain gesture or posture information for the body part.

[0017] Preferably, the method includes analysing changes in gesture or posture to determine movement information for the body part.

[0018] Preferably, the ultrasound signal is transmitted into muscle tissue used to manipulate the body part.

[0019] Preferably, the ultrasound signal is transmitted into tendon tissue linking muscle tissue and the body part.

[0020] Preferably, parameter information is one or more of amplitude, phase, flight-time, frequency spectrum and waveform pattern of the reflected signal.

[0021] Preferably, the method includes an initial training step for obtaining reference information, the training step including:

• • [0022] using the transmitter to transmit one or more training signals into the body part and receiving reflected and/or scattered training signals at the receiver via the ultrasound transducer or transducer array,
  • [0023] manipulating the reflected and/or scattered training signals to obtain training parameter data, wherein the training parameter data includes one or more of amplitude, phase, flight-time, frequency spectrum and waveform pattern of the reflected training signals, and
  • [0024] storing the training parameter data as reference information.

[0025] According to a third aspect of the invention there is provided a method for indirectly assessing the gesture, posture or movement of a body part of a person including:

• • [0026] applying an ultrasound transducer or transducer array to a body part to be monitored,
  • [0027] using a transmitter to transmit ultrasound signals into the body part and receiving reflected and/or scattered ultrasound signals at the receiver,
  • [0028] manipulating the reflected and/or scattered ultrasound signals to obtain a plurality of parameter data, and
  • [0029] comparing changes in the parameter data to determine movement information for the body part.

[0030] According to a forth aspect of the invention there is provided apparatus for indirectly assessing the gesture, posture or movement of a body part of a person including:

• • [0031] an ultrasound transducer or transducer array for attaching to a body part to be monitored,
  • [0032] an ultrasound transmitter and receiver for transmitting an ultrasound signal into the body part and receiving a reflected and/or scattered ultrasound signal,
  • [0033] a signal processor for manipulating the reflected ultrasound signal to obtain parameter information, and
  • [0034] a processor for storing reference information and comparing the parameter information to said reference information to determine gesture, posture or movement information for the body part.

[0035] Preferably, the apparatus further includes a data collector for receiving and storing the reflected and/or scattered signal and/or parameter information.

[0036] Preferably, the ultrasound transmitter is adapted and arranged to transmit the ultrasound signal into muscle tissue, and the receiver is adapted and arranged to receive the reflected and/or scattered ultrasound signals for the muscle tissue via the ultrasound transducer or transducer array.

[0037] Preferably, at least the processor is implemented on a computer, the computer including a display to graphically illustrate the gesture, posture or movement information for the body part.

[0038] Preferably, the ultrasound transducer, transmitter and receiver, signal processor, and the processor are integrated into a compact unit that can be worn on the body part.

[0039] Preferably, the ultrasound transducer, transmitter and receiver, signal processor, and the processor are integrated into a compact unit that can be embedded in the body part.

[0040] Preferably, the ultrasound transducer is integrated with other types of sensors for monitoring the gesture, posture, and movement of the body part.

[0041] Preferably, the ultrasound transducer has a flat shape.

[0042] Preferably, the ultrasound transducer is attached on the body surface by adhesive materials on the transducer surface.

[0043] Preferably, the ultrasound transducer is embedded in the inner surfaces of a prosthesis socket.

[0044] Preferably, the ultrasound transducer array is arranged on a flexible substance.

[0045] According to a fifth aspect of the invention there is provided a method for indirectly assessing the gesture, posture or movement of a body part of a person including:

• • [0046] applying an ultrasound transmitter and receiver to a body part to be monitored,
  • [0047] applying a tissue stimulator on the body part to manipulate muscle and/or tendon tissue of the body part,
  • [0048] using the transmitter to transmit a detecting ultrasound signal into the body part and receiving a reflected and/or scattered ultrasound signal at the receiver,
  • [0049] processing the reflected and/or scattered ultrasound signal to obtain the stiffness information of the muscle and/or tendon tissue, and
  • [0050] comparing the stiffness information to reference information to obtain gesture or posture information for the body part.

[0051] According to a sixth aspect of the invention there is provided apparatus for indirectly assessing the gesture, posture or movement of a body part of a person including:

• • [0052] an ultrasound transducer or transducer array for attaching to a body part to be monitored,
  • [0053] an ultrasound transmitter and receiver for transmitting a detecting ultrasound signal into the body part and receiving a reflected and/or scattered ultrasound signal,
  • [0054] a tissue stimulator for manipulating muscle and/or tendon tissues of the body part,
  • [0055] a signal processor for processing the reflected ultrasound signal to obtain the tissue stiffness information, and
  • [0056] a processor for storing reference information and comparing the tissue stiffness information to said reference information to determine gesture, posture or movement information for the body part.

[0057] Preferably, the tissue stimulator is a compressor for loading and unloading the muscle and/or tendon tissues in a static way.

[0058] Preferably, the tissue stimulator is a low frequency vibrator for vibrating the muscle and/or tendon tissues.

[0059] Preferably, the tissue stimulator is a second ultrasound transmitter for transmitting a stimulating ultrasound beam into the muscle and/or tendon tissues.

[0060] Further aspects of the invention will become apparent from the following description, which is given by way of example only.

[0064] Referring to FIG. 1, single or multiple ultrasound transducers 1 are attached on the skin surface 8 to transmit ultrasound waves 6 into the soft tissues 9 of a part of the body. One transducer 1 can be used for transmitting and/or receiving ultrasound signals 6,7. Multiple transducers 1 can be used to transmit or receive ultrasound signals 6,7 simultaneously. The ultrasound signals 6 are reflected and/or scattered from the various tissue interfaces and inside various tissues 9. These reflected and/or scattered ultrasound signals 7 are received, digitized, and processed to obtain various parameters of the signal. The various parameters are related to the dimension, position, angle and/or strength of the muscle, tendon or other tissues and are further related to the gesture and/or posture of the body part associated with the soft tissue. Different gestures and postures will result in different ultrasound reflected and/or scattered signals 7. Thus signal parameters are compared to a table of reference information to determine the gesture and/or posture of the body part.

[0065] The ultrasound signals are continuously recorded and analyzed to detect the changes in the soft tissues, particularly the muscles. The movement of the body parts can then be determined by analysing the information of the soft tissue changes, that is to say the changes in gesture and posture. The changes of muscles, tendons or other soft tissues can happen perpendicularly to the skin surface, or along the skin surface, which can be detected using multiple transducers or a transducer array aligned along the direction of the changes.

[0066] The parameters of ultrasound signals for detecting those differences may include amplitude, phase, flight-time, frequency spectrum, waveform pattern and their combinations. Various known signal processing techniques can be used to extract the parameter information.

[0067] The preferred embodiment of an apparatus for use in the invention includes an ultrasound transducer 1, an ultrasound transmitter and receiver 2, and digital signal processing module 3. These interface with a computer 4, which comprises a data collection module and a processor for storing reference information and comparing signal parameters to determine gesture and posture. The processor also analyses changes in gesture and posture to determine movement of the body part.

[0068] Alternative embodiments of the invention include two or more transducers located perpendicular to and/or along the skin surface.

[0069] In the preferred embodiment the computer 4 includes a 2D or 3D computer simulation program which illustrates movement of the body part graphically. In an alternative embodiment, the computer 4 includes a program to translate the detected gesture, posture and movement of the fingers, hand and/or arm into the action of keystrokes or computer mouse command including mouse movement, button clicks, and wheel rolling. This facilitates a virtual computer input device or human-machine interface.

[0070] In an alternative embodiment the interface 5 between digital signal processing module 3 and computer 4 is wireless. The transducer 1, ultrasound transmitter and receiver 2, and digital signal processing module 3 are incorporated into a compact unit that can be worn by a person. In yet a further embodiment the compact unit may include a data collection module for storing received signals and/or parameter data. This would enable the wearer to go about their daily activities with the stored data being accessible at a later time for analysis of gesture, posture and movement. This is beneficial for medical analysis and rehabilitation, such as movement and gait analysis.

[0071] In another alternative embodiment, the ultrasound transmitter and receiver 2, digital signal processing module 3, and computer 4 are incorporated into a compact unit that can be worn by a person. Such a compact unit can further include transducers to become an all-in-one unit. The transducers can be arranged on the back of a watch-like compact unit, which can be worn on the limb. The compact unit has a communication port that can transfer data with another computer, which can be used during training or downloading stored gesture, posture and movement information for further analysis. In the compact unit, the ultrasound signals can be processed in real-time to obtain the gesture, posture and movement information of the body part. This real-time information can be used for the prosthesis control, control and monitoring for functional electrical stimulation, computer control, robotic control, and capturing gesture language.

[0072] Initially, in order to determine the reference information training is required. The various patterns of reflected and scattered ultrasound signal parameters are obtained for different gestures and postures. Movement associated with changes in gesture and posture are analysed and the information is stored in a look-up table. The look-up table can be a simple list of values or a set of coefficients of a sophisticated system modelling like an artificial neural network. During training the apparatus is connected to computer 4 which guides the wearer through various gestures, postures and movements. As knowledge of the correlation between the ultrasound signals from the soft tissues and the corresponding gestures, postures and movements are accumulated for various persons it is envisaged that the training process for individual wearers may be ultimately eliminated.

[0073] The training can be performed with a local computer or with a remote computer via networks like the Internet. In the remote mode, the local computer or similar device that connects to the network only provides a data link between the ultrasound detector and the remote computer, and a terminal for local data input and output. The remote computer is a central server that can work with many ultrasound detectors simultaneously. The central server controls the training and performs all the data analysis for extracting reference information. Internet training can reduce the cost for the software maintenance and upgrading.

[0074] Flat ultrasound transducers are coupled onto the soft tissue using liquid gels or silicon gels on the skin 8 surface. Film-like transducers can also be used. Adhesive substances or tapes like that used in electromyography (EMG) electrodes is used to attach the transducer onto the skin surface. Multiple transducers, transmitters and receivers can be used to monitor the gesture, posture, and movement of different body parts simultaneously. The flat ultrasound transducers are arranged on a flexible substance. This makes the transducer array easy to attach on body parts with various curvatures. The transducers are arranged as one or two dimensional arrays.

[0075] The ultrasound transducer can also be integrated with other types of sensors for monitoring the gesture, posture, and movement of a body part, such as EMG electrodes, gyroscope, and accelerometer. Corresponding signal conditioners for the sensors can be used to collected corresponding signals. These signals can be analyzed together with the ultrasound signals obtained by the ultrasound transducer to provide more precise detection of the gesture, posture, and movement of the corresponding body part.

[0076] The ultrasound transducers or a compact unit including the transducer, transmitter, receiver, signal collector, and signal analyser can also be embedded into the body or the prosthesis socket of an artificial body part. Wired or wireless links between the body embedded transducers or the compact unit and the outside control unit can be used to transfer electrical energy, signal, and data. This is particularly useful for people with artificial limbs. The transducers or the compact unit can be embedded into the body during or after the amputation surgery. The method and apparatus can be used to obtain information to mechanically manipulate/move artificial limbs in response to gestures and postures of the remained body parts. The body embedded transducer or compact unit can also be used with body embedded functional electrical stimulation system for controlling or monitoring the gesture, posture, or movement of a body part. For the controlling purpose, the ultrasound transducer will be arranged on the muscle that is used to control the electrical stimulation. For the monitoring purpose, the transducer will be arranged on the muscle to be stimulated.

[0077] A tissue stimulator 17 can be used to manipulate the soft tissues including the muscle and tendon of the body part being monitored by the detecting ultrasound signal of the ultrasound system. The tissue stimulator can be a compressor for loading and unloading the tissue in a static way, a vibrator for vibrating the tissue in a low frequency, or a second ultrasound transmitter for transmitting a stimulating ultrasound beam into the tissue.

[0078] In an embodiment using a compressor as tissue stimulator 17 the tissue is compressed during the continuous collection of the detecting ultrasound signals. The reflected and/or scattered ultrasound signals collected before and after compression is applied are processed to determine the deformation of the tissue due to the compression. The amount of tissue deformation relates to the stiffness and other mechanical properties of the tissue. Thus, tissue stiffness information can be obtained using this compressing process.

[0079] In an embodiment using a vibrator as tissue stimulator 17 the tissue is vibrated during the continuous collection of the detecting ultrasound signals. The reflected and/or scattered ultrasound signals collected during the tissue vibration are processed to obtain the propagation speed of the vibration in the tissue, which relates to the tissue stiffness and other mechanical properties. Thus, tissue stiffness information can be obtained using this vibrating process.

[0080] In an embodiment using a second ultrasound transmitter as tissue stimulator 17 the tissue is stimulated and subsequently the detecting ultrasound beam is disturbed. How the reflected and/or scattered ultrasound signals of the detecting ultrasound beam are disturbed by the stimulating ultrasound beam, for example the change of amplitude and central frequency, relates to the tissue stiffness and other mechanical properties. Thus, the tissue stiffness information can be obtained using this stimulating process.

[0081] The stiffness of the muscle and tendon relates to the degree of the contraction and stretching, respectively. Since the muscle contraction and tendon stretching further relates to the posture or gesture of the body part, the tissue stiffness information can be used to estimate the posture or gesture using reference information stored during a training process.

[0082] FIG. 2 illustrates some typical locations of transducers and compact receiving units on parts of the body for detecting gesture, posture and movement of respective body parts. Ultrasound transducers 10 are arranged on various parties of the body. A wearable receiving unit 11 is attached on the waist and connects to the transducers 10 that are attached on the neck, chest, shoulder, upper arm, and upper leg. The postures and movements of the head, trunk, upper arm, forearm, and lower leg can be monitored, respectively.

[0083] A watch-like receiving unit 12 is worn on the wrist and connects to transducers 10 that are attached on the forearm and upper arm. The gestures and movements of the upper arm, wrist and fingers can be monitored.

[0084] A second wearable receiving unit 13 is attached on the lower leg and connects to the transducers 10 that are attached on the knee, lower leg, and foot. The postures and movements of the knee joint, lower leg, foot, and toes can be monitored.

[0085] A compact receiving unit 14 integrated with a transducer is attached on the upper leg to monitor the posture and movement of the lower leg.

[0086] A second watch-like receiving unit 15 is attached on the wrist and connects to an array of transducers 16 that are attached on the forearm. The gestures and movements of the wrist and individual fingers can be monitored.

[0087] FIG. 2 only shows examples of the arrangement of transducers and receiving units on the front of the body, they can also be attached on the sides and back of the body.

[0088] A system according to the invention is useful for posture, movement and gait analysis in biomedical engineering and sports. It may also find application in robot controls, virtual reality systems, human-machine interfaces, game controllers and for capturing life-like movements for animated characters.

[0089] Where in the foregoing description reference has been made to integers or elements having known equivalents then such are included as if individually set forth herein.

[0090] Embodiments of the invention have been described, however it is understood that variations, improvements or modifications can take place without departure from the spirit of the invention or scope of the appended claims.