Method and apparatus for monitoring human activity pattern

[0001] This application claims the benefit of Korean Patent Application No. 10-2005-0003635, filed on Jan. 14, 2005, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.

[0002] 1. Field of the Invention

[0003] The present invention relates to a method and apparatus for monitoring human activity, and more particularly, to a method and apparatus for monitoring a human activity pattern to provide information on the amount of physical activity of a user by monitoring the caloric consumption of the user during daily activities.

[0004] 2. Description of the Related Art

[0005] In order to maintain the healthy life of an individual, there is a need to continuously measure the amount of daily activity and caloric consumption without limiting the daily activities.

[0006] Among the technologies for monitoring the amount of daily activity are those disclosed in WO 96-30080 and U.S. Pat. No. 6,165,143. These patents disclose technologies for finding the activity pattern of an individual by using a variety of sensors, and measuring the amount of physical activity. However, these conventional technologies have restrictions such that in order to find the activity pattern of an individual, the direction and the location of a sensor must be fixed.

[0007] For example, in the WO 96-30080, a sensor is implanted in the heart, and the direction and location of the sensor are required to be fixed, and in U.S. Pat. No. 6,165,143 sensors are required to be attached at the waist, the upper leg, and the frontal points of knee joints.

[0008] The present invention provides a method and apparatus for monitoring a human activity pattern in which by using a 3-axis acceleration sensor and a terrestrial magnetism sensor, movement in the direction of gravity and movement in the horizontal direction by a user are separated. Further, by using the signal characteristics with respect to the locations of the sensor, the attached locations of the sensor can be recognized regardless of the directions of the sensor, and the activity pattern of the user can be determined.

[0009] According to an aspect of the present invention, there is provided a method for monitoring a human activity pattern including: sensing an inertia movement signal according to a movement of a user using a sensor unit attached to the user; detecting a direction of the sensor unit from acceleration; by using the inertia movement signal and direction, detecting a wearing location of the sensor unit; and determining the activity pattern of the user from the inertia movement signal by reflecting the wearing location.

[0010] According to another aspect of the present invention, there is provided an apparatus for monitoring a human activity pattern including: a sensor unit attached to a user, which senses an inertia movement signal according to a movement of the user; and a data processing unit which detects an acceleration signal and a direction signal of the sensor unit by using the inertia movement signal, detects a wearing location of the sensor unit by using the inertia movement signal and the direction, and determines the activity pattern of the user from the inertia movement signal by reflecting the wearing location.

[0011] According to still another aspect of the present invention, there is provided a computer readable recording medium having embodied thereon a computer program for executing the method for monitoring an activity pattern.

[0026] The present invention will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown.

[0027] Referring to FIG. 1, an apparatus for monitoring a human activity pattern includes a sensor unit 10 and a data processing unit 11.

[0028] Also, the apparatus for monitoring a human activity pattern may further include an interface unit 12 for providing results processed in the data processing unit to a user, or receiving required inputs from the user, and a mobile terminal 13 which operates in the same manner as the interface unit 12 does, but is wirelessly connected. In addition, according to another exemplary embodiment, the apparatus for monitoring a human activity pattern may be implemented as a separate apparatus, or may be embedded in the mobile terminal 13. In the latter case, the interface unit 12 can be implemented as a display panel or a keypad located on the mobile terminal 13.

[0029] The mobile terminal, may be capable of wireless communication, and may be a personal digital assistant (PDA), a portable computer, or a mobile phone. The mobile terminal may communicate in a short range wireless communication scheme such as Bluetooth, and/or may communicate through a cable such as a USB port or RS232C.

[0030] The sensor unit 10 includes a 3-axis acceleration sensor 101 for measuring an inertia movement, more specifically, acceleration in x, y, and z directions, and may further include a terrestrial magnetism sensor 102 or gyroscope (not shown) for detecting the orientation of the sensor unit 10 with respect to a planar surface parallel to the sensor unit 10. Also, in order to sense the orientation of the sensor unit 10 with respect to the planar surface, a tilt sensor for measuring the tilt from a reference vertical axis may further be included.

[0031] The data processing unit 11 processes an acceleration signal output from the acceleration sensor 101 to measure an acceleration value in relation to vibration in the 3-axis directions or an external acceleration value such as gravity, and processes the direction signal output from the terrestrial magnetism sensor 102. While detailed descriptions pertaining to the signal processed by the data processing unit 11 are limited to the acceleration signal, one of ordinary skill will appreciate that in other exemplary embodiments the signal can be extended more broadly to the inertia movement signal.

[0032] Also, the data processing unit 11 transforms the measured acceleration values and direction signal in the body frame of the sensor unit 10 into the ones of the space fixed coordinates.

[0033] FIG. 2 is a flowchart of the operations performed by a method for monitoring a human activity pattern according to the present invention.

[0034] First, the data processing unit 11 detects the direction of the sensor unit 10 from the DC component of the acceleration sensor 101, and after compensating the acceleration of the AC component output from the acceleration sensor 101 for the direction of the sensor unit 10, outputs the compensated result in operation 24.

[0035] FIG. 3 is a flowchart of the operations performed in an exemplary process for detecting direction.

[0036] First, a yaw angle ψ is detected by using the terrestrial magnetism sensor 102 in operation 30. The yaw angle is not necessarily needed for detecting the direction of the sensor unit 10, but is a useful component. FIG. 4A illustrates the principle of measuring a yaw angle by using the terrestrial magnetism sensor 102. Referring to FIG. 4A, when the terrestrial magnetism sensor 102 is tilted with respect to the planar surface 40, the yaw angle ψ indicates the angle that the terrestrial magnetism sensor 102 sweeps the planar surface 40 from the reference line 41 indicating the E-direction of the planar surface 40. When ĝ denotes the gravity acceleration and {right arrow over (z)} denotes the vector of the orientation of the terrestrial magnetism sensor 102, a vector {right arrow over (z)}_// obtained by projecting {right arrow over (z)} onto the planar surface 40 and the yaw angle ψ can be obtained through the following equation 1:

[0037] $\begin{array}{cc}{\stackrel{->}{z}}_{//}=\stackrel{->}{z}-\text{(}\stackrel{->}{z}·\hat{g}\text{)}\hat{g}\cos \left(\Psi \right)=\frac{{\stackrel{->}{z}}_{//}·\hat{E}}{\sqrt{{\stackrel{->}{z}}_{//}·{\stackrel{->}{z}}_{//}}}& \left(1\right)\end{array}$

[0038] The pitch angle θ and the roll angle Φ can be obtained from the DC component of an acceleration signal output from the acceleration sensor 101 or the tilt sensor in operation 31. FIG. 4B illustrates the principle of obtaining a pitch angle and a roll angle by using the DC component of acceleration.

[0039] Referring to FIG. 4B, the pitch angle θ indicates an angle from the reference line 42 to the Y-axis of the acceleration sensor 101, and the roll angle Φ indicates an angle from the reference line 41 to the X-axis of the acceleration sensor 101. The pitch angle θ and roll angle Φ can be obtained by the following equation 2:

[0040] $\begin{array}{cc}\cos \left(\frac{\pi }{2}-\varphi \right)=\frac{\stackrel{->}{x}·\hat{g}}{\sqrt{\stackrel{->}{x}·\stackrel{->}{x}}}\cos \left(\frac{\pi }{2}-\theta \right)=\frac{\stackrel{->}{y}·\hat{g}}{\sqrt{\stackrel{->}{y}·\stackrel{->}{y}}}& \left(2\right)\end{array}$

[0041] If the yaw angle, pitch angle, and roll angle are obtained as shown in FIGS. 4A and 4B, a rotational transform matrix with respect to the yaw angle, pitch angle, and roll angle is obtained in operation 32. The rotational transform matrix may be obtained with respect to only the pitch angle and roll angle, or with respect to the yaw angle, pitch angle, and roll angle. The rotational transform matrix is multiplied by the AC component of the acceleration value output from the acceleration sensor 101. Thus, acceleration components in x, y, and z direction in the body frame of the acceleration sensor 101 are transformed into acceleration values in the space fixed coordinates in operation 33. Consequently, the acceleration values output from the data processing unit 11 are compensated for the direction of the sensor unit 10 to be output. At this time, if the acceleration is compensated for by using the rotational transform matrix containing the yaw angle, pitch angle, and roll angle, more accurate compensation can be performed than when using the rotational transform matrix containing only the pitch angle and roll angle,

[0042] Using the acceleration value in the space fixed coordinates, the wearing location of the sensor unit 10 is detected in operation 21. The wearing location can be detected by a kinematics approach to human walking and pendulum modeling.

[0043] The kinematics approach focuses on the fact that when a person moves, a trajectory of a signal output from the sensor unit 10 varies depending on the wearing location. The pendulum modeling regards the sensor as attached to the human body as a pendulum, and models the movement trace of the sensor as shown in FIG. 5, to determine the characteristics of a signal which differ depending on the wearing location. That is, when the waist or the body is regarded as a fixed point, and the sensor unit 10 is located on the arm, hand, or leg, or in a pocket or handbag, the movement of the sensor unit 10 is modeled as a single or as a double pendulum movement.

[0044] Referring to FIG. 5, reference number 50 indicates the body of the fixed point, and reference number 51 indicates the sensor unit 10 modeled as the single pendulum when the sensor unit 10 is held in the hand or the pocket. Reference number 52 indicates the sensor unit 10 modeled as the second pendulum connected to the first pendulum while the arm is modeled as the first pendulum when the sensor unit 10 is put in the handbag.

[0045] If (x₁, y₁) denotes the location of the first pendulum 51, and (x₂, y₂) denotes the location of the second pendulum 52, then when the user moves, it can be regarded that the fixed point 50 moves horizontally at a speed of v. At this time, the location of each pendulum 51 and 52 can be obtained by the following equation 3:
x₁=vt+l₁ sin θ₁
y₁=−l₁ cos θ₁
x₂=vt+l₁ sin θ₁+l₂ sin θ₂
y₂=−l₁ cos θ₁−l₂ cos θ₂  (3)

[0046] Here, l₁ denotes the distance between the fixed point 50 and the first pendulum 51, and l₂ denotes the distance between the first pendulum 51 and the second pendulum 52.

[0047] When the movement trajectory is modeled as a single pendulum, the acceleration signals in the gravity and horizontal directions show a phase diagram in the form of a circle as shown in FIG. 6A. At this time, with respect to the radius lψ² of the circle, it is determined whether the sensor unit 10 is on the arm or leg, or in the pocket. That is, by referring to the distance between the fixed point 50 to the first pendulum 51 as the distance from the reference point of the body to the wearing location of the sensor unit 10, the location of the sensor unit 10 is determined from the radius of the circle.

[0048] At this time, the data processing unit can store in advance the distances, input through the interface unit 12, from the reference point to all the wearing locations at which the sensor unit 10 can be located, such as the arm, leg, pocket, and hand.

[0049] When the movement trajectory is modeled as a double pendulum, the acceleration signals in the gravity and horizontal directions show a phase diagram that is asymmetrical with respect to the acceleration axis for the gravity direction component as shown in FIG. 6B.

[0050] Accordingly, from the phase diagram of the acceleration in the gravity direction and the horizontal direction, it can be determined whether the movement trajectory is modeled as a single pendulum or a double pendulum, and the wearing location of the sensor unit 10 can be also determined. That is, it can be determined to which part, such as the hand or leg, the sensor unit 10 is attached, or whether the sensor unit 10 is carried in a handbag apart from the human body.

[0051] If the wearing location of the sensor unit 10 is detected, the wearing mode is determined at that location by using acceleration values on the space fixed coordinates in operation 22 (FIG. 2). Here, the wearing mode indicates an activity pattern such as walking, running or cycling. The determination is made by referring to the frequency and intensity of the acceleration signal with respect to the wearing location. This is because even though activity patterns may be identical, the acceleration signals vary according to the wearing location of the sensor unit 10. That is, the detected acceleration signals of the sensor unit 10 held in the hand and put in the pocket may be different. Also, preferably, the data processing unit stores acceleration ranges for each activity pattern with respect to the wearing location in order to determine the wearing mode.

[0052] When the activity being performed, or wearing mode, is determined, the presence or absence of periodicity in an acceleration signal is determined. Periodicity is determined because the signal of walking, running, or cycling shows periodicity in the gravity direction or in the horizontal direction according to the wearing location of the sensor unit 10.

[0053] More specifically, the determination of the activity pattern can be performed by calculating the dynamic parameters of the gravity direction component and horizontal direction component of an acceleration signal.

[0054] FIGS. 7A and 7B illustrate the frequency distribution of the components of an acceleration signal in the gravity direction and the horizontal direction, respectively, with respect to signal intensity according to the pattern of activity. In a case wherein the movement speed of a leg is measured, the sensor unit 10 is attached on the thigh. Referring to FIGS. 7A and 7B, each unique frequency and intensity area is divided in the gravity direction and horizontal direction, in FIGS. 7A and 7B, respectively, for each type of activity. Similar distribution plots or phase diagrams can be made by replacing the intensity axis with other dynamic parameters such as mean, median, peak, standard deviation, skew, or kurtosis of acceleration for each direction, and a correlation coefficient between each pair of accelerations can be used to classify the physical activity more specifically.

[0055] In case there is overlap of more than two activities for given dynamic parameters, a sum of conditional probabilities of dynamic parameters given that an activity occurs will determine the activity pattern such as

[0056] $\begin{array}{cc}{\Sigma }_i=\sum _{j}p\left({\eta }_i❘{\xi }_j\right)p\left({\xi }_j\right)=\sum _{j}p\left({\eta }_i❘{\xi }_j\right)\mathrm{wherein},{\eta }_i\text{:}\mathrm{pattern}\left(e.g,\mathrm{rest},\mathrm{walk},\mathrm{jog},\mathrm{run}\right),{\xi }_j\text{:}\mathrm{dynamic}\mathrm{parameter}\left(e.g,{\sigma }_x,\mathrm{etc}\right),& \left(4\right)\end{array}$

[0057] According to equation (4), the activity can be classified by finding the maximum Σ_i.

[0058] FIG. 10 illustrates an example of conditional probabilities of activity pattern. In FIG. 10, the horizontal axis is of standard deviation of ζ. As shown in FIG. 10, each of the activity patterns is distinguished from each other.

[0059] The activity pattern and its duration can be provided to the user through the mobile terminal 13. Thus, the user can learn which activity pattern was performed, when it was performed, and for how long.

[0060] If the activity pattern is walking or running, the data processing unit 11 again detects the current wearing location of the sensor unit 10. This is to determine whether the wearing location of the sensor unit 10 is changed during the activity. For example, if the location of the sensor unit 10 is changed from the user's hand in the pocket, the acceleration and direction detected by the sensor unit 10 also change, and therefore the operation 21 is performed again to detect the wearing location.

[0061] If the activity pattern is determined, the analysis of the determined activity pattern can be performed in operation 23. The analysis of the activity pattern includes calculation of calories consumed by the activity pattern, the number of steps, and the moving distance. In addition, if the gravity direction component of the acceleration value sharply changes while the change in the horizontal direction component is negligible, it is determined that the user has fallen over, and an alarm can be sent through the mobile terminal 13. If it is determined from personal information that the user is advanced in age, an emergency center can be informed of the fall by the mobile terminal 13.

[0062] As an example of the analysis of the activity pattern, the process for measuring the consumed calories will now be explained in more detail.

[0063] FIG. 8A illustrates values output from the acceleration sensor 101 when speed is increased over time. Reference number 80 indicates the speed gradually increasing over time, and reference numbers 81 and 82 show acceleration sensed by different acceleration sensors. Referring to FIG. 8A, the speed of 0.7 km/h or more is regarded as that of running, and it can be seen that the values output from the acceleration sensor 101 change abruptly from those output when the speed is 0.6 km/h. FIG. 8B illustrates the caloric consumption of 24 users, measured with respect to the amount of physical activity. Referring to FIG. 8B, it can be seen that the caloric consumption amount for walking is clearly distinguished from that for running. Also, it can be seen that even in the area for running or walking, the measured amounts have a wide distribution. This distribution occurs because the physical condition of users varies. Accordingly, in an exemplary embodiment of the present invention, the consumed calories are measured with reference to the personal information of the user. The personal information includes at least one of the sex, age, height, and weight of the user. The caloric consumption has a linear relation with respect to the amount of physical activity measured by the acceleration sensor 101, as described by the following equation 4:

[0064] $\begin{array}{cc}\mathrm{Calorie}=b\times VM+cVM=\sqrt{\sum _{i=x,y,z}\int a_{i}dt}& \left(5\right)\end{array}$

[0065] Here, b and c are constants and a_i is an acceleration signal.

[0066] In the equation 4, constants b and c are determined according to the personal information of an individual, and in the present invention, are obtained by applying a known multiple regression analysis method.

[0067] As another example of the activity mode, measuring the number of steps will now be explained. Generally, the number of steps is measured by counting the number of times the gravity direction component of the acceleration exceeds a certain value. The number of steps is inclined to be over-counted when the user walks fast while inclined to be under-counted when the use walks slowly. Also, shock noises such as random shocks can be measured incorrectly as steps.

[0068] Accordingly, in the present exemplary embodiment, according to the changing range of the gravity direction component of the acceleration, the measuring time and threshold value are adjusted, and after measuring the steps, a locking period is set so that the shock noises are not measured.

[0069] FIGS. 9A and 9B illustrate the gravity direction components of the acceleration when a user moves at speeds of 3.0 km/h and 8.5 km/h, respectively. Reference numbers 90 and 92 each indicate a time for beginning to count the number of steps in FIGS. 9A and 9B, respectively. While reference numbers 91 and 93 indicate threshold speed levels counted by steps, in FIGS. 9A and 9B, respectively.

[0070] Referring to FIGS. 9A and 9B, in order to measure the number of steps, it is preferable that with the increasing activity intensity, the counting time interval is shorter and the threshold value is higher. If the number of steps is measured, the moving distance can also be calculated. According to sports medicine, the length of a step of an ordinary person is (height-100 cm), so if the length is multiplied by the number of steps, the moving distance can be calculated.

[0071] If the analysis of the activity pattern is performed, the analysis result can be provided to the user through the mobile terminal 13 in operation 24. The result includes current caloric consumption, number of steps, and/or moving distance.

[0072] The present invention can also be embodied as computer readable code on a computer readable recording medium. The computer readable recording medium is any data storage device that can store data which can be thereafter read by a computer system. Examples of the computer readable recording medium include read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and carrier waves (such as data transmission through the internet). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. Also, functional programs, code, and code segments for accomplishing the present invention can be easily construed by programmers skilled in the art to which the present invention pertains.

[0073] According to the present invention, by detecting the location on which the sensor is attached, and monitoring the activity pattern of the user with reference to the detected location, the activity pattern of the user can be monitored without limiting the wearing location of the sensor.

[0074] Also, by measuring the activity pattern, the elapsed time, the caloric consumption of the activity, the number of steps, or the moving distance, information on the amount of physical activity can be provided to the user.

[0075] Furthermore, if the user falls over, then if necessary, the mobile terminal can notify an emergency center.

[0076] While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. The exemplary embodiments should be considered in a descriptive sense only and not for purposes of limitation. Therefore, the scope of the invention is defined not by the detailed description of the invention but by the appended claims, and all differences within their scope will be construed as being included in the present invention.