US 7,728,839 B2Grant

Discriminative motion modeling for human motion tracking

Issue Date:2010-06-01

•25 Claims

•8 Drawing Sheets

Abstract

A system and method recognizes and tracks human motion from different motion classes. In a learning stage, a discriminative model is learned to project motion data from a high dimensional space to a low dimensional space while enforcing discriminance between motions of different motion classes in the low dimensional space. Additionally, low dimensional data may be clustered into motion segments and motion dynamics learned for each motion segment. In a tracking stage, a representation of human motion is received comprising at least one class of motion. The tracker recognizes and tracks the motion based on the learned discriminative model and the learned dynamics.

Metadata

Assignee

Honda Motor Co., Ltd.

Inventors

Ming-Hsuan Yang
Zhimin Fan

Application Information

Application Number:US 11/553,374

Filing Date:2006-10-26

Priority Date:2005-10-28

Art Unit:2628

Classifications

IPC:

G06T7/20

Patent Drawings (8 sheets)

Description

Related Applications

[0001] This application claims priority from U.S. provisional application No. 60/731,158 entitled “Discriminative Motion Modeling For Human Motion Tracking” which is incorporated by reference herein in its entirety.

Field of the Invention

[0002] The invention relates generally to computer vision, and more specifically, to tracking human motion.

Background of the Invention

[0003] Many applications in computer vision involve motion analysis and modeling, such as motion tracking and action recognition. Most conventional methods for motion modeling are largely limited to simple motions. A comprehensive analytical model for complex motions, such as biological motion or human motion, is a challenging problem. One of the difficulties in motion modeling stems from the high dimensionality of the complex motion, which demands great descriptive power from the model itself. Without any constraint, it is very difficult, if not impossible, to model arbitrary motions. Fortunately, in practice, the motions of interest are more or less constrained due to physical or biological reasons. Although these constraints can be highly nonlinear, they largely reduce the intrinsic complexity of the motion. For example, human motions cannot be arbitrary but must be confined by anthropologically feasible joint angles, e.g., the upper arm and the lower arm cannot move independently.

[0004] Thus, one issue in motion tracking is to characterize and take advantage of these constraints. Since it is generally difficult to explicitly describe motion constraints, a plausible alternative is to learn them from training data. Human motion, although complex, resides in a space whose dimensionality is significantly lower than its joint angle space. Thus, dimensionality reduction is a significant step of learning to help reduce the problem complexity and build a motion model.

[0005] Many conventional techniques are available for dimensionality reduction in human motion tracking. One conventional technique is to reduce the dimensionality using ISOMAPS and learn a Gaussian mixture model in the low-dimensional space as described in Tenenbaum, J. B., et al., A Global Geometric Framework For Nonlinear Dimensionality Reduction, Science, 2000, vol. 290 pp. 2319-2323 which is incorporated by reference herein in its entirety. Another conventional technique is to use Laplacian eigenmaps for dimensionality reduction, and employ continuity interpolation when modeling dynamics as described in Sminchisescu, C., and A. Jepson, Generative Modeling for Continuous Non-Linearity Embedded Visual Inference, ICML, 2004 which is incorporated by reference herein in its entirety. In yet another conventional technique, K-means clustering is used to partition the state space first, and then Principal Component Analysis (PCA) is used to reduce the dimensionality.

[0006] These conventional methods are suitable when the motion is short, uniform and continuous, but are inappropriate for recognizing and tracking different motion patterns. These techniques may introduce confusion among different motion classes due to the compactness in the low-dimensional space and prevent accurate tracking.

[0007] There have been several previous attempts to deal with training data comprising multiple classes of motion. For example, a transition probability matrix may be learned as described in Wang, Q., et al., Learning Object Intrinsic Structure for Robust Visual Tracking, Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2003, pp. 227-233 and North, B., et al.,Learning and Classification of Complex Dynamics,IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, pp. 1016-1034 which are both incorporated by reference herein in their entirety. An alternative to the transition matrix is to apply the training algorithm repeatedly for each individual motion. A problem that confronts these two methods is that it is possible that the different motions to be tracked may have some similar human poses, and these poses may be even closer when the dimensionality of the data is reduced. When the tracker is approaching these confusion areas caused by these similar poses, the tracker can be distracted since there is no discriminance enforced among those possible motion patterns. When motion segments with different characteristics are intermingled together, the accuracy of motion modeling may deteriorate.

[0008] Due to the problems above, the preservation of differences between motion patterns in the training set is a significant property when tracking multiple classes of motion. Therefore, it is also preferable to maintain the discriminance between motions in the lower dimensional space. Traditional discriminative models such as Linear Discriminative Analysis (LDA) are inappropriate for this problem because the motions to be modeled are generally non-linear and non-Gaussian.

[0009] What is needed as a system for discriminative motion modeling that can recognize and track a variety of human motion patterns in a reduced dimensionality space.

Summary of the Invention

[0010] A system and method is provided to efficiently recognize and track a set of stylized human motions. A representation of human motion is received having at least one motion from a motion class. A motion comprises a sequence of pose states residing in a high dimensional space. A discriminative model is received, for example, from an offline learning stage to project the sequences of pose states from the high dimensional space to a low dimensional space. The discriminative model enforces discriminance between the different classes of motion in the low dimensional space. The tracker can accurately and efficiently track motions from different motion classes based on the learned discriminative model. This framework is general and does not require restrictions on the selection of the stylized motions to be tracked.

[0011] In one embodiment, in order to preserve the discriminance in the low-dimensional space, the discriminative model projects data points that are near neighbors in the original space so that they are still the near neighbors in the low-dimensional space. Also, the neighbor data points in the original space from different motion classes are separated as far as possible in the low-dimensional space.

[0012] In on embodiment of the learning stage, the model of the human body is represented as a skeleton and body parts. A pose state vector represents the pose of the skeleton and each motion type is represented by a sequence of such vectors. Vectors in the high dimensional space are projected onto a low-dimensional space by a dimensionality reduction algorithm. A clustering algorithm is applied to the low-dimensional data to generate a set of motion segments, and an autoregressive process (ARP) is learned for each motion segment.

[0013] In one embodiment of the tracking stage, the initialized skeleton is projected onto the low-dimensional space. The nearest neighbor(s) of the current skeleton is then computed, and the corresponding possible motion type(s) of these neighbors are determined. The discriminative motion model helps the tracker to be selective with these nearest neighbors, especially when multiple motion classes are tracked. The learned dynamics are then used to predict the future skeleton pose in the low-dimensional space and can be reconstructed to the original space. Finally, the body parts hypotheses and the predicted skeleton determine the tracking result interactively.

Brief Description of the Drawings

[0014] FIG. 1 is a computer system in accordance with one embodiment of the present invention.

[0015] FIG. 2 is an input/output diagram in accordance with one embodiment of the present invention.

[0016] FIG. 3 is a flow diagram illustrating an offline learning stage in accordance with one embodiment of the present invention.

[0017] FIG. 4 is a human body model in accordance with one embodiment of the present invention.

[0018] FIG. 5 is a flow diagram illustrating a method for applying a discriminative model in accordance with one embodiment of the present invention.

[0019] FIG. 6 is a flow diagram illustrating a tracking stage in accordance with one embodiment of the present invention.

[0020] FIG. 7 is a flow diagram illustrating a method for generating pose predictions in accordance with one embodiment of the present invention.

Detailed Description of the Invention

[0021] A preferred embodiment of the present invention is now described with reference to the figures where like reference numbers indicate identical or functionally similar elements. Also in the figures, the left most digit of each reference number corresponds to the figure in which the reference number is first used.

[0022] Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.

[0023] Some portions of the detailed description that follows are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times, to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices, without loss of generality.

[0024] However, all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or “determining” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0025] Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by a variety of operating systems.

[0026] The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

[0027] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references below to specific languages are provided for disclosure of enablement and best mode of the present invention.

[0028] In addition, the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the claims.

[0029] FIG. 1 is a computer system according to one embodiment of the present invention. The computer system 100 comprises an input device 102, a memory 104, a processor 106, an output device 108, and an image processor 110. The input device 102 is coupled to a network 120, a database 130, and an image capture unit 140. The output device 108 is coupled to a database 150, a network 160, and a display 170. In other embodiments, the input device is connected to only one or two of a network 120, a database 130, and an image capture unit 140. In yet another embodiment, the input device may be connected to any device configured to input data to the computer system. Similarly, in some embodiments, the output device may be connected to one or more of a database 150, network 160, display 170 or any other device cable of receiving outputted data. In another embodiment, the computer system comprises one or more of a processor 106, an image processor 110, or other specialized processor.

[0030] FIG. 2 is an input/output diagram according to one embodiment of the present invention. In an offline learning stage 220 a discriminative model 230 is generated based on motion capture data 210. A tracking stage 250 receives the discriminative model 230 and human motion data 240 (such as a video) comprising at least one motion to be tracked. Based on the learned discriminative model 230, the tracking stage outputs tracking data 260.

[0031] The motion capture data 210 comprises data of at least one motion class. For example, a motion class may be walking, clapping, tossing, balancing, or swinging. Motion capture data 210 may be acquired by a variety of conventional techniques. In one embodiment, a subject wears a special suit with markers and performs motions captured by a video camera or similar device. Further, motion capture data 210 may be two dimensional or three dimensional. Motion capture data 210 may be acquired from multiple subjects performing the same classes of motions. This provides the statistical data used to generate the deterministic model 230.

[0032] In the learning stage, motion capture data 210 is received 302 by the computer system 100. In one embodiment, the motion capture data 210 may be received by a image capture unit 140 interfaced to an input device 102. In other embodiments, the motion capture data 210 may be received by the input device 102 from a database 130 or through a network 120.

[0033] Motion capture data is processed 304 to extract pose states. Processing may be performed by a processor 106, image processor 110, or other specialized processor. An example of the pose state representation is illustrated in FIG. 4. In this example embodiment, a human body 400 includes a pose state comprising skeleton joints 404. The pose state is represented by a vector, x. For example, x may represent the relative coordinates of skeleton joints 404. In another embodiment, the pose state vector, x, may instead represent other characteristics that describe the pose state. For example, the pose state vector, x, may represent angles, positions, velocities, or accelerations of joints, limbs, or other body parts or points of interest used to describe the pose state. A motion is composed of a time sequence of pose states, and is represented by a sequence of vectors X=[x₁x₂. . . x_m]. Each pose state vector, x_i, may be assigned a motion label y_i. The motion label y_irepresents the class of motion to which the pose state vector, x_i, belongs.

[0034] The sequence of vectors, X, representing a motion resides in a high dimensional space, denoted by HεRⁿwhere R is the space of real numbers and n is the dimensionality of the high dimensional space. For example, if 13 skeleton joints 404 are used to represent the pose (as in FIG. 4), each pose state vector, x, would have n=26 dimensions, representing the concatenated two dimensional image coordinates of the 13 skeleton joints 404. In another embodiment, the pose state vector, x, may comprise three dimensional image coordinates. Processing data in this high dimensional space is computationally expensive and inefficient, making real-time tracking difficult or impossible. Therefore, it is desirable to project the high-dimensional data to a low-dimensional space denoted by LεR^lwhere l is the dimensionality of the low dimensional space. This projection becomes possible given that human motion is naturally constrained due to physical or biological reasons. For example, the upper arm and lower arm cannot move independently and there are a limited number of anthropologically feasible joint angles.

[0035] Turning back to FIG. 3, a discriminative model is learned 306 to project the high-dimensional data, X, to a low dimensional space. The discriminative model obtains low-dimensional state vectors Z=[z₁z₂. . . z_m], where the dimensionality, l, of each z_iis less than the dimensionality, n, of each x_i. In reducing the dimensionality of the data, the discriminative model also enforces discriminance between different classes of motion. By enforcing discriminance, the model projects data such that samples which are near neighbors in the high dimensional space and of the same motion class remain near neighbors in the low-dimensional space. Samples which are near neighbors but from different motion classes are separated as far as possible in the low-dimensional space.

[0036] For example, a pose state vector x_aof the motion class “clap” may appear very similar to a pose state vector x_bof the motion class, “swing”. The discriminative model will project the data such that the low dimensional state vectors z_aand z_bare far apart in the low dimensional space. In contrast, if a pose state vectors x_aand x_care both of the motion class “clap” and appear very similar in the high dimensional space, the discriminative model will project the data such that z_aand z_cremain similar in the low dimensional space. The discriminative model is thus configured to avoid confusion areas between different classes of motions and allows for accurate tracking of multiple motion classes.

[0037] The discriminance between the data from different motion classes is achieved by increasing the inter-class separability, which is the sum of the distances between each data point and their neighboring data points that are from different classes, and simultaneously reducing the intra-class variety, which is the sum of the distances between each point and their neighboring points that are in the same class. The advantage of contriving these two measurements is that in contrast to many conventional techniques, it does not rely on the assumption that the data follows Gaussian distribution.

[0038] In one embodiment, Local Discriminant Embedding (LDE) (or similar Marginal Fisher Analysis) provides the discriminative model 230 to project data from the high dimensional space to the low dimensional space. LDE and Marginal Fisher Analysis are described in more detail in Chen, H., et al., Local Discriminant Embedding and Its Variants, Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2000, pp. 126-133; and Yan, et al., Graph Embedding: A General Framework for Dimensionality Reduction, Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 830-837, which are both incorporated by reference herein in their entirety. Additional details may be found in He, X., Locality Preserving Projections, NIPS, 2003 which is incorporated by reference herein in its entirety.

[0039] FIG. 5 illustrates one embodiment of a method for learning 306 the discriminative model. For the motion data set X=[x₁x₂. . . x_m]εH, a projection matrix V generates the low dimensional data by Z=V^TX=[z₁z₂. . . z_m]εL. The class label for each x_iis y_i.

[0040] The intra-class variety is computed 510 as
^i,jiεN_k₁⁺(j) or jεN_k₁⁺(i)

[0041]

\begin{matrix} \begin{matrix} S_{w} = \sum_{i, j}^{} \sum_{i \in N_{k_{1}}^{+} (j) or j \in N_{k_{1}}^{+} (i)}^{} { W_{x_{i}} - W_{x_{j}} }^{2} \\ = 2 W^{T} X (D^{w} - W^{w}) X^{T} W \end{matrix} W_{ij}^{w} = 1 if j \in N_{k_{1}}^{+} (i) or i \in N_{k_{1}}^{+} (j); 0, else & (1) \end{matrix}

[0042] where N_k₁⁺(i) denotes the set of k₁nearest neighbors of point x_i, which are in the same class of class y_i. D^wis a diagonal matrix with entries d_ii^w=Σ_jW_ij^w

[0043] The inter-class separability is computed 520 as
^i,jiεN_k₂⁻(j) or jεN_k₂⁻(i)

[0044]

\begin{matrix} \begin{matrix} S_{b} = \sum_{i, j}^{} \sum_{i \in N_{k_{2}}^{-} or j \in N_{k_{2}}^{-} (i)}^{} { W_{x_{i}} - W_{x_{j}} }^{2} \\ = 2 W^{T} X (D^{b} - W^{b}) X^{T} W \end{matrix} W_{i . j}^{b} = 1 if j \in N_{k_{2}}^{-} (i) or i \in N_{k_{2}}^{-} (j); 0, else & (2) \end{matrix}

[0045] where N_k₂⁻(i) denotes the set of k₂nearest neighbors of point x_i, which are in classes other than y_i. D^bis a diagonal matrix with entries d_ii^b=Σ_jW_ij^b

[0046] Then, the projection matrix, V, can be obtained 530 by solving the following generalized eigenvector problem,
X(D^b−W^b)X^Tv=λX(D^w−W^w)X^Tv (3)

[0047] Eigenvectors v₁, v₂, . . . , v_lare chosen corresponding to the l largest eigenvalues. The mapping from the high dimensional space to the low dimensional space can then be computed 540 by z_i=V^Tx_i, where V=[V₁, V₂, . . . , V_l].

[0048] A dynamic human motion model can further improve motion tracking. Motion can be practically modeled as an autoregressive process by recognizing that the movement in a short period tends to be continuous. This model is based in part on techniques described in Wang and North referenced previously; and Agarwal, A. and B. Triggs, Tracking Articulated Motion Using a Mixture of Autoregressive Models, ECCV, 2004; and Li, Y., et al., Motion Texture: A Two-Level Statistical Model for Character Motion Synthesis, SIGGRAPH, 2002 which are both incorporated by reference herein in their entirety.

[0049] A clustering algorithm is applied 308 to separate the data of each motion pattern in the low dimensional space into clusters. In one embodiment, a K-means algorithm provides the clustering algorithm. To ensure that each cluster is formed by contiguous sequences, the time stamp t_iof each sample point z_iis also considered. Thus, in one embodiment, similar low dimensional state vectors, z, that have very different time stamps, t, are clustered separately. For example, if low dimensional state vectors z_aand z_bare very similar and are very close together in time, it is highly likely the state vectors are part of the same motion segment and should be clustered together. However, if the similar state vectors are far apart in time, it is less likely they are part of the same motion segment and may instead represent similar poses from different motion segments. Thus, the clustering algorithm encourages temporally contiguous sample vectors to be clustered into one segment.

[0050] Motion dynamics are learned 310 for each motion segment to help guide the tracking. Motion dynamics may comprise two-dimensional or three dimensional motions. In one embodiment, an auto-regressive process (ARP) is applied to learn the motion patterns of each segment by

[0051]

\begin{matrix} z_{t} = \sum_{k = 1}^{K} A_{k} z_{t - k} + d + Bn, & (4) \end{matrix}

[0052] where z_tεR^lis the state vector in the low-dimensional space, A_kεR^l×l; k=1, . . . , K is the state transition matrices for K clusters, d is the offset vector, BεR^l×l, u_t˜N(0, Q) is the noise vector. The parameters {A_k, k=1, . . . , K, d, B} can be learned by Maximum Likelihood Estimation (MLE) as described in North, et al., referenced above. Those motion segments, whose resulting prediction error by ARP is large, are considered to be erratic movements and are pruned away. The number of clusters in the clustering algorithm, K, is chosen as the one from a range, (for example, [1, 6]]) which can produce the minimum ARP prediction error of all clusters.

[0053] The tracking stage 250 is configured to track a set of stylized motion patterns based on the learned model. In tracking, the human body can be represented by a human body model, for example, as illustrated in FIG. 4. In one embodiment, the data comprising the human body model is a combination of two subsets of parameters: pose state (represented by, for example, skeleton joints 402) and body parts 404. This decomposable model helps to distribute the computation into two subsets, each with reduced dimensionality. These two subsets jointly and interactively determine the human configuration, i.e., the skeleton joints 402 represents the abstract “style” of motions, which accounts for the general and global movement of human. The body parts 404 are the “content” of the motion, which are different for different people wearing different clothes.

[0054] The body parts 404 are represented by a set of body part descriptors. In one embodiment, rectangles associated with limbs, torso, and head, for example, represent each body part. In another embodiment, different shapes are used to represent the body parts. Each body part 404 has the parameters of center point, height, width, scale and orientation. The body parts 404 may then be manipulated with operations such as translation, rotation, scaling, and so on.

[0055] In one embodiment, various types of histograms are used as the body part descriptors. For example a color histogram, a gradient orientation histogram, and a color distance histogram may be used. A color histogram is a conventional histogram that represents an image by counts of the colors of each pixel in the image. A gradient orientation histogram is a representation that provides edge information in an image. Both the orientation and the magnitude of the gradient are computed for each pixel within the body part. Then the orientations are quantized into a predetermined number of bins and the magnitude of each gradient is added into its corresponding bin.

[0056] A color distance histogram represents the spatial distribution of color pixels. The color distance histogram describes how the color pixels are distributed within the body part, which can alleviate confusion between candidates which have similar color histogram but very different spatial distribution of color pixels. In computing the color distance histogram, the intensity range of the pixels is quantized into a predetermined number of bins. For each pixel, the pixel distance to the center of the body part is computed and the distance is added to the bin corresponding to the intensity of that pixel. Thus, both the intensity and the spatial structure of the color pixels are considered.

[0057] In the tracking stage, 250, an image is received from the human motion data 240 (for example, a video) comprising the motion to be tracked. In one embodiment, no background subtracting is performed on the received image in contrast to many conventional techniques. The current skeleton vector, x_c, is projected 604 to the low dimensional space by z_c=V^Tx_cwhere V is the projection matrix generated in the learning stage 220. Pose predictions are then generated 608 to predict a future pose state of the skeleton. FIG. 7 illustrates one embodiment of a method for generating 608 pose predictions. The P nearest neighbors of z_care determined 702 where each of the P nearest neighbors come from different motion segments. In one embodiment, P is a predetermined quantity. The motion type of each of the P nearest neighbors are then determined 704. The learned motion dynamics of the P different motion segments are applied 706 to form the pose predictions for each nearest neighbor. Thus, predictions are generated 608 based on learned motion dynamics for different motion segments. In one embodiment, pose predictions are generated using the auto-regressive process of Eq. 4 described previously.

[0058] Given the skeleton configuration in the low-dimensional space, L, the corresponding skeleton is reconstructed 610 in the original space, H for the predicted poses. Here, the basic assumption is that, the local geometry is consistent in both the original space and the low-dimensional space. Neighborhood reconstruction is applied to reconstruct a predicted pose, x_p,in the original space from the predicted pose, z_p,in the low dimensional space in a manner similar to that described in Roweis, S. and L. K. Saul, Nonlinear Dimensionality Reduction By Locally Linear Embedding, Science, 2000, vol. 290, pp. 2323-2326 which is incorporated be reference herein in its entirety.

[0059] In reconstructing x_pfrom z_p, the nearest neighbors of z_pare first determined. Next, the linear coefficients that best reconstruct z_pfrom the selected neighborhood are computed by a least square method. The neighborhood of z_pin L has known corresponding states in H. x_pis then reconstructed by linearly combining these states with the calculated coefficients.

[0060] In order to determine the optimal or near optimal combination of the body parts and the predicted skeleton, the body parts 404 must be fitted to the skeleton 402. Each of the body parts are fitted onto each of the possible predicted skeletons S_p, for p=1, . . . , P. One way to represent the topology of the body parts is by a graph G=(V, E), where the vertices V={v₁, . . . , v_Q} correspond to the Q body parts (in FIG. 4, for example, Q=10). An edge e(v_i, v_j) is assigned to each connected parts v_iand v_jsuch as the head and the torso, for example. The optimal fitting can be formulated as minimizing a cost function over the graph G, the parameters of which include the configuration of the skeleton S_pand the configurations of all the Q body parts B_i, i=1, . . . , Q.

[0061] For each body part B_i, the associated cost function m(Bi) can be written as follows,
m(B_i)=q(B_i,I)+d(B_i,S_p) (5)

[0062] where, q(B_i, I) measures how well the body part matches the image I when positioned as B_i, and d(B_i, S_p) measures the extent to which the body part B_ideviates from the skeleton S_p. There is also a cost over each edge e(v_i, v_j)ε E, which is the distance between those adjacent body parts, i.e., d(B_i, B_j). The optimal solution is computed by H*={S*, B_i*, i=1, . . . , Q}, such that

[0063]

\begin{matrix} H^{*} = \arg \min_{B} (\sum_{e (v_{i}, v_{j}) \in E} d (B_{i}, B_{j}) + \sum_{v_{i} \in V} m (B_{i})) & (6) \end{matrix}

[0064] Conventionally, solving the above optimization problem {S*, B_i*, i=1, . . . , Q} needs O(QⁿF) time, where Q is the number of the body parts, n is the possible configurations of each body part, and F is the number of all possible skeletons. If there are no constraints on the configurations of the skeleton then F=13^f, where 13 is the number of the skeleton points, f is the number of possible locations of each skeleton point. Solving these equations these equations using conventional techniques brings overwhelming computational cost.

[0065] In contrast to conventional methods, computation is significantly reduced by restricting the number of the possible skeletons to only P predictions based on nearest neighbors as described previously. The discriminative motion modeling is well adapted to make this local neighborhood search because the model reduces confusion areas caused by the multiple motion classes in the training data. Computation may be further reduced by employing dynamic programming to make the body part fitting polynomial rather than exponential in time. This technique is based in part on the techniques used in Felzenszwalb, P. F., Efficient Matching of Pictorial Structures, Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2000, pp. 66-73 which is incorporated by reference herein in its entirety.

[0066] For any vertex v_jwith no children (i.e., any leaf of the tree), its best configuration can be computed as a function of its parent v_i,

[0067]

\begin{matrix} B_{j}^{*} (B_{i}) = \arg \min_{B_{j}} (d (B_{i}, B_{j}) + m (B_{j})) & (7) \end{matrix}

[0068] For any vertex v_jthat has both parent v_iand children, the best location of all its children can be known given the configuration of B_j. Let C_Bjdenote the best cost of all its children. Then the best configuration of v_jis,

[0069]

\begin{matrix} B_{j}^{*} (B_{i}) = \underset{B_{j}}{\arg \min} (d (B_{i}, B_{j}) + m (B_{j}) + C_{B_{j}}) & (8) \end{matrix}

[0070] For the root vertex v_r, its best configuration is found as,

[0071]

\begin{matrix} B_{r}^{*} = \underset{B_{r}}{\arg \min} (m (B_{r}) + C_{B_{r}}) & (9) \end{matrix}

[0072] That is, fitting from the leaf parts can start, such as the lower arms and legs, then connects to their parents, i.e., the upper arms and legs, and finally reach to the root part, i.e., the head. Each non-leaf node records its best children given its own configuration. After that, the best configurations of each body part can be determined by tracing down from the root part to the leaf parts. The computation here is polynomial, O(Qn²).

[0073] While particular embodiments and applications of the present invention have been illustrated and described herein, it is to be understood that the invention is not limited to the precise construction and components disclosed herein and that various modifications, changes, and variations may be made in the arrangement, operation, and details of the methods and apparatuses of the present invention without departing from the spirit and scope of the invention as it is defined in the appended claims.

Claims

What is claimed is:

1. A method for recognizing and tracking human motion comprising steps of:

receiving, by an input device, a plurality of learned motion segments representing different learned motions within a motion class, wherein each learned motion segment comprises a plurality of state vectors and each state vector comprises a time stamp, and wherein one of the learned motion segments comprises temporally contiguous state vectors clustered together in a low-dimensional space based on the time stamps;

receiving, by the input device, a representation of human motion having at least one motion from the motion class, the at least one motion comprising a sequence of pose states represented in a high dimensional space;

processing the received representation according to computer-executable instructions stored in a memory that cause a processor to execute steps of:

projecting the sequences of pose states from the high dimensional space to the low dimensional space according to a discriminative model that when applied to the sequence of pose states increases the inter-class separability between pose states of different motion classes and decreases the intra-class separability between pose states of a same motion-class;

determining an integer P nearest neighbors of a first projected pose state in the low dimensional space, the P nearest neighbors from P different learned motion segments;

determining P pose predictions for the P different learned motion segments; and

determining the pose prediction that best matches a current frame of the representation of human motion; and

storing the determined pose prediction to a memory.

2. The method of claim 1 wherein determining the pose prediction that best matches a current frame of the representation of human motion comprises steps of:

reconstructing at least one pose prediction in the high dimensional space based on the discriminative model; and

determining an optimal matching of the at least one pose prediction in the high dimensional space to a current frame of the representation of human motion.

3. The method of claim 2 wherein determining an optimal matching comprises steps of:

representing the current frame by a human body model comprising coordinates of joints and body parts having shapes associated with limbs, torso and head;

representing each pose prediction in the high dimensional space by the human body model; and

selecting the pose prediction that optimally matches to the current frame based on the human body model.

4. The method of claim 3 wherein the human body model comprises body part descriptors including one or more of a color histogram, a gradient orientation histogram, and a color distance histogram.

5. The method of claim 1 wherein determining the P pose predictions comprises steps of:

determining a motion type of each of the nearest neighbors; and

applying a dynamic model to each of the nearest neighbors based on the motion type, the dynamic model learned in a learning stage.

6. The method of claim 1 wherein the discriminative model is received from a learning stage and wherein the learning stage is prior to said receiving steps, the learning stage comprising steps of:

receiving motion capture data from a motion capture source, the motion capture data comprising a first motion from a first motion class and a second motion from a second motion class that is different from the first motion class;

processing the motion capture data to extract a first sequence of pose states representing the first motion and a second sequence of pose states representing the second motion;

learning the discriminative model configured to project the first and second sequence of pose states to a low dimensional space and enforce discriminance between the first and second motion classes in the low dimensional space;

applying a clustering algorithm to cluster the temporally contiguous state vectors into the learned motion segments in the low dimensional space; and

learning a dynamic model for each motion segment to generate motion predictions in the low dimensional space.

7. The method of claim 6 wherein the clustering algorithm includes a k-means clustering algorithm.

8. The method of claim 6 wherein learning the discriminative model includes applying a Local Discriminant Embedding (LDE) model.

9. The method of claim 6 wherein learning a discriminative model comprises steps of:

computing an intra-class variety representing the sum of the distances between data points that are in the same motion class;

computing the inter-class separability representing the sum of the distances between data points that are in different motion classes;

obtaining a projection matrix configured to reduce the intra-class variety and increase the inter-class separability; and

projecting the motion capture data from the high dimensional space to the low dimensional space based on the projection matrix.

10. The method of claim 1 wherein the pose state comprises a vector of skeleton joint coordinates.

11. The method of claim 1 wherein the at least one motion is tracked without background subtraction.

12. The method of claim 1 wherein tracking the at least one motion comprises tracking the at least one motion in three dimensions.

13. A system for recognizing and tracking human motion comprising:

an input device for receiving a representation of human motion having at least one motion from a motion class, the at least one motion comprising a sequence of pose states represented in a high dimensional space, and for receiving a plurality of learned motion segments representing different learned motions within the motion class, wherein each learned motion segment comprises a plurality of state vectors and each state vector comprises a time stamp, and wherein one of the learned motion segments comprises temporally contiguous state vectors clustered together in a low-dimensional space based on the time stamps;

a processor adapted to project the sequences of pose states from the high dimensional space to the low dimensional space according to a discriminative model that when applied to the sequence of pose states, increases the inter-class separability between pose states of different motion classes and decreases the intra-class separability between pose states of a same motion class, determining an integer P nearest neighbors of a first projected pose state in the low dimensional space, the P nearest neighbors from P different learned motion segments, determining P pose predictions for the P different learned motion segments, and determining the pose prediction that best matches a current frame of the representation of human motion; and

a memory adapted to store the determined pose state.

14. A computer program product, comprising a computer readable medium storing computer executable code for recognizing and tracking human motion, the computer executable code when executed causing a processor to perform steps of:

receiving a plurality of learned motion segments representing different learned motions within a motion class, wherein each learned motion segment comprises a plurality of state vectors and each state vector comprises a time stamp, and wherein one of the learned motion segments comprises temporally contiguous state vectors clustered together in a low-dimensional space based on the time stamps;

receiving a representation of human motion having at least one motion from the motion class, the at least one motion comprising a sequence of pose states represented in a high dimensional space;

determining an integer P nearest neighbors of a first projected pose state in the low dimensional space, the P nearest neighbors from P different learned motion segments;

determining P pose predictions for the P different learned motion segments; and

determining the pose prediction that best matches a current frame of the representation of human motion; and

storing the determined pose prediction to a memory.

15. The computer program product of claim 14 wherein determining the pose prediction that best matches a current frame of the representation of human motion comprises steps of:

reconstructing at least one pose prediction in the high dimensional space based on the discriminative model; and

determining an optimal matching of the at least one pose prediction in the high dimensional space to a current frame of the representation of human motion.

16. The computer program product of claim 15 wherein determining an optimal matching comprises steps of:

representing the current frame by a human body model comprising coordinates of joints and body parts having shapes associated with limbs, torso and head;

representing each pose prediction in the high dimensional space by the human body model; and

selecting the pose prediction that optimally matches to the current frame based on the human body model.

17. The computer program product of claim 16 wherein the human body model comprises body part descriptors including one or more of a color histogram, a gradient orientation histogram, and a color distance histogram.

18. The computer program product of claim 14, wherein determining the P pose predictions comprises steps of:

determining a motion type of each of the nearest neighbors; and

applying a dynamic model to each of the nearest neighbors based on the motion type, the dynamic model learned in a learning stage.

19. The computer program product of claim 14 wherein the discriminative model is received from a learning stage and wherein the learning stage is prior to said receiving steps, the learning stage comprising steps of:

processing the motion capture data to extract a first sequence of pose states representing the first motion and a second sequence of pose states representing the second motion;

applying a clustering algorithm to the cluster temporally contiguous state vectors into the learned motion segments in the low dimensional space; and

learning a dynamic model for each motion segment to generate motion predictions in the low dimensional space.

20. The computer program product of claim 19 wherein the clustering algorithm includes a k-means clustering algorithm.

21. The computer program product of claim 19 wherein learning the discriminative model includes applying a Local Discriminant Embedding (LDE) model.

22. The computer program product of claim 19 wherein learning a discriminative model comprises steps of:

computing an intra-class variety representing the sum of the distances between data points that are in the same motion class;

computing the inter-class separability representing the sum of the distances between data points that are in different motion classes;

obtaining a projection matrix configured to reduce the intra-class variety and increase the inter-class separability; and projecting the motion capture data from the high dimensional space to the low dimensional space based on the projection matrix.

23. The computer program product of claim 14 wherein the pose state comprises a vector of skeleton joint coordinates.

24. The computer program product of claim 14 wherein the at least one motion is tracked without background subtraction.

25. The computer program product of claim 14 wherein tracking the at least one motion comprises tracking the at least one motion in three dimensions.

Patent Citations (10)

Patent	Date	Inventor	Cited By
US5263098(A)	1993-11-01	Horikami	Examiner
US5353132(A)	1994-10-01	Katsuma	Examiner
US5943435(A)	1999-08-01	Gaborski	Examiner
US6295367(B1)	2001-09-01	Crabtree et al.	Applicant
US6483940(B1)	2002-11-01	Wang	Examiner
US6591146(B1)	2003-07-01	Pavlovic et al.	Examiner
US6778705(B2)	2004-08-01	Gutta et al.	Examiner
US6947042(B2)	2005-09-01	Brand	Examiner
US6985172(B1)	2006-01-01	Rigney et al.	Examiner
US7092566(B2)	2006-08-01	Krumm	Examiner

Non-Patent Literature (33)

Belhumeur et al. Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. IEEE Transactions on Pattern Analysis and Machine Intelligence. vol. 19. Issue 7. Jul. 1997.Examiner
Yan et al. Discriminant Analysis on Embedded Manifold. European Conference on Computer Vision. 2004.Examiner
Yang et al. Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. vol. 26. Issue 1. Jan. 2004.Examiner
Raja et al. Segmentation and Tracking Using Color Mixture Models. Proceedings of the Third Asian Conference on Computer Vision. vol. 1. 1998.Examiner
Dalal et al. Histograms of Oriented Gradients for Human Detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition. vol. 1. Jun. 2005.Examiner
Urtasun et al. Priors for People Tracking from Small Training Sets. Proceedings of the Tenth IEEE International Conference on Computer Vision. Oct. 15-21, 2005.Examiner
Morris et al. Singularity Analysis for Articulated Object Tracking. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Jun. 23-25, 1998.Examiner
Ju et al. Cardboard People: A Parameterized Model of Articulated Image Motion. Proceedings of the Second International Conference on Automatic Face and Gesture Recognition. Oct. 14-16, 1996.Examiner
Brand et al. Style Machines. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. 2000.Examiner
Mikolajczyk et al. A Performance Evaluation of Local Descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence. vol. 27. Issue 10. Oct. 2005.Examiner
Agarwal et al. Tracking Articulated Motion using a Mixture of Autoregressive Models. European Conference on Computer Vision. May 2004.Examiner
Mowbray et al. Automatic Gait Recognition via Fourier Descriptors of Deformable Objects. Audio Visual Biometric Person Authentication. 2003.Examiner
Magee et al. Building Class Sensitive Models for Tracking Applications. Proceedings of the British Machine Vision Conference. 1999.Examiner
Dhillon et al. Class Visualization of High-Dimensional Data with Applications. Computational Statistics & Data Analysis. vol. 41. Issue 1. 2002.Examiner
Cao et al. Expressive Speech-Driven Facial Animation. ACM Transactions on Graphics. vol. 24. Issue 4. Oct. 2005.Examiner
Yam et al. Gait Recognition by Walking and Running: A Model-Based Approach. Asian Conference on Computer Vision. Jan. 2002.Examiner
Bowden. Learning Statistical Models of Human Motion. Computer Vision and Pattern Recognition. 2000.Examiner
Tanco et al. Realistic Synthesis of Novel Human Movements from a Database of Motion Capture Examples. Proceedings of the Workshop on Human Motion. 2000.Examiner
Agarwal, A., et al., “Tracking Articulated Motion Using a Mixture of Autoregressive Models,” Proceedings of the 8th European Conference on Computer Vision, 2004, 12 pages.Applicant
Chen, H., et al., “Local Discriminant Embedding and Its Variants,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2000, pp. 126-133.Applicant
Felzenszwalb, P. F., et al., “Efficient Matching of Pictorial Structures,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2000, pp. 66-73.Applicant
He, X., et al., “Locality Preserving Projections,” Proceedings, Neural Information Processing Systems (NIPS), 2003, 8 pages.Applicant
Li, Y., et al., “Motion Texture: A Two-Level Statistical Model for Character Motion Synthesis,” ACM Computer Graphics (SIGGRAPH), 2002, pp. 465-472.Applicant
North, B., et al., “Learning and Classification of Complex Dynamics,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Sep. 2000, pp. 1016-1034, vol. 22, No. 9.Applicant
Roweis, S. T., et al., “Nonlinear Dimensionality Reduction by Locally Linear Embedding,” Science, Dec. 22, 2000, pp. 2323-2326, vol. 290.Applicant
Sminchisescu, C., et al., “Generative Modeling for Continuous Non-Linearly Embedded Visual Inference,” Proceedings of the 21st International Conference on Machine Learning (ICML), 2004, 8 pages.Applicant
Tenenbaum, J. B., et al., “A Global Geometric Framework for Nonlinear Dimensionality Reduction,” Science, Dec. 22, 2000, pp. 2319-2323, vol. 290.Applicant
Wang, Q., “Learning Object Intrinsic Structure for Robust Visual Tracking,” Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'03), 2003, pp. 227-233.Applicant
Yan, S., et al., “Graph Embedding: A General Framework for Dimensionality Reduction,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 830-837.Applicant
Sminchisescu, C., et al., “Estimating Articulated Human Motion with Covariance Scaled Sampling,” International Journal of Robotics Research, 2003, pp. 371-393, vol. 22, No. 6., [online] [retrieved on Nov. 27, 2006] Retrieved from the Internet: <URL: http://www.cs.toronto.edu/˜crismin/PAPERS/css—ijrr03.pdf>.Applicant
Sminchisescu, C., et al. “Learning to Reconstruct 3D Human Motion from Bayesian Mixtures of Experts. A Probabilistic Discriminative Approach,” Technical Report, University of Toronto, Oct. 2004, pp. 1-28, CSRG-502, [online] [retrieved on Nov. 27, 2006] Retrieved from the Internet: <URL: http://www.cs.toronto.edu/˜crismin/PAPERS/csrg502.pdf>.Applicant
Zhao, T., et al., “3D Tracking of Human Locomotion: A Tracking as Recognition Approach,” University of Southern California, Institute for Robotics and Intelligent Systems, pp. 1-6, Los Angeles, CA., [online] [retrieved on Nov. 27, 2006] Retrieved from the Internet: <URL: http://iris.usc.edu/Outlines/papers/2002/zhao-icpr02.pdf>.Applicant
PCT International Search Report and Written Opinion, PCT/US06/42088, Nov. 3, 2008, 14 Pages.Applicant

Abstract

Description

Related Applications

Field of the Invention

[0002] The invention relates generally to computer vision, and more specifically, to tracking human motion.

Background of the Invention

[0009] What is needed as a system for discriminative motion modeling that can recognize and track a variety of human motion patterns in a reduced dimensionality space.

Summary of the Invention

Brief Description of the Drawings

[0014] FIG. 1 is a computer system in accordance with one embodiment of the present invention.

[0015] FIG. 2 is an input/output diagram in accordance with one embodiment of the present invention.

[0016] FIG. 3 is a flow diagram illustrating an offline learning stage in accordance with one embodiment of the present invention.

[0017] FIG. 4 is a human body model in accordance with one embodiment of the present invention.

[0018] FIG. 5 is a flow diagram illustrating a method for applying a discriminative model in accordance with one embodiment of the present invention.

[0019] FIG. 6 is a flow diagram illustrating a tracking stage in accordance with one embodiment of the present invention.

[0020] FIG. 7 is a flow diagram illustrating a method for generating pose predictions in accordance with one embodiment of the present invention.

Detailed Description of the Invention

[0040] The intra-class variety is computed 510 as
^i,jiεN_k₁⁺(j) or jεN_k₁⁺(i)

[0041]

\begin{matrix} \begin{matrix} S_{w} = \sum_{i, j}^{} \sum_{i \in N_{k_{1}}^{+} (j) or j \in N_{k_{1}}^{+} (i)}^{} { W_{x_{i}} - W_{x_{j}} }^{2} \\ = 2 W^{T} X (D^{w} - W^{w}) X^{T} W \end{matrix} W_{ij}^{w} = 1 if j \in N_{k_{1}}^{+} (i) or i \in N_{k_{1}}^{+} (j); 0, else & (1) \end{matrix}

[0042] where N_k₁⁺(i) denotes the set of k₁nearest neighbors of point x_i, which are in the same class of class y_i. D^wis a diagonal matrix with entries d_ii^w=Σ_jW_ij^w

[0043] The inter-class separability is computed 520 as
^i,jiεN_k₂⁻(j) or jεN_k₂⁻(i)

[0044]

\begin{matrix} \begin{matrix} S_{b} = \sum_{i, j}^{} \sum_{i \in N_{k_{2}}^{-} or j \in N_{k_{2}}^{-} (i)}^{} { W_{x_{i}} - W_{x_{j}} }^{2} \\ = 2 W^{T} X (D^{b} - W^{b}) X^{T} W \end{matrix} W_{i . j}^{b} = 1 if j \in N_{k_{2}}^{-} (i) or i \in N_{k_{2}}^{-} (j); 0, else & (2) \end{matrix}

[0045] where N_k₂⁻(i) denotes the set of k₂nearest neighbors of point x_i, which are in classes other than y_i. D^bis a diagonal matrix with entries d_ii^b=Σ_jW_ij^b

[0046] Then, the projection matrix, V, can be obtained 530 by solving the following generalized eigenvector problem,
X(D^b−W^b)X^Tv=λX(D^w−W^w)X^Tv (3)

[0051]

\begin{matrix} z_{t} = \sum_{k = 1}^{K} A_{k} z_{t - k} + d + Bn, & (4) \end{matrix}

[0061] For each body part B_i, the associated cost function m(Bi) can be written as follows,
m(B_i)=q(B_i,I)+d(B_i,S_p) (5)

[0063]

\begin{matrix} H^{*} = \arg \min_{B} (\sum_{e (v_{i}, v_{j}) \in E} d (B_{i}, B_{j}) + \sum_{v_{i} \in V} m (B_{i})) & (6) \end{matrix}

[0066] For any vertex v_jwith no children (i.e., any leaf of the tree), its best configuration can be computed as a function of its parent v_i,

[0067]

\begin{matrix} B_{j}^{*} (B_{i}) = \arg \min_{B_{j}} (d (B_{i}, B_{j}) + m (B_{j})) & (7) \end{matrix}

[0069]

\begin{matrix} B_{j}^{*} (B_{i}) = \underset{B_{j}}{\arg \min} (d (B_{i}, B_{j}) + m (B_{j}) + C_{B_{j}}) & (8) \end{matrix}

[0070] For the root vertex v_r, its best configuration is found as,

[0071]

\begin{matrix} B_{r}^{*} = \underset{B_{r}}{\arg \min} (m (B_{r}) + C_{B_{r}}) & (9) \end{matrix}