3D object tracking method and apparatus

[0001] This application claims the benefit of Korean Patent Application No. 2010-0094134, filed on Sep. 29, 2010 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference.

[0002] 1. Field

[0003] Embodiments relate to a method of tracking movement of a 3D object through a vision system, such as a camera.

[0004] 2. Description of the Related Art

[0005] 3D model-based tracking is a process of generating a 3D model of an object to be tracked and performing edge-based tracking of an image of the model. That is, the model is projected on the image using 3D rigid body transform of the previous position of the model using characteristics of a camera, and the edge of the image is searched in the vertical direction of a space of the edge of the 3D model. Then, a 3D rigid body transform matrix in which a distance error between the edge of the image and the model is minimized is calculated. Such a process is applied to each frame, thereby tracking the 3D model.

[0006] As described above, in order to calculate the 3D model rigid body transform matrix of the 3D model, the 3D model is projected on the image based on the previous position of the 3D model, and then a plurality of points is sampled from the edge. Points where the sampled points meet the edge of the image meet in the vertical direction of the space of the 3D model are obtained, and then the 3D rigid body transform matrix where the sum total of distances to these points is minimized is calculated. Here, when the points are not correctly matched to the edge of the 3D model, the 3D rigid body transform matrix is incorrectly calculated, and thus precise tracking of the object is not achieved.

[0007] Therefore, it is an aspect of one or more embodiments to provide a 3D object tracking method and apparatus in which a model of an object to be tracked is divided into a plurality of polygonal planes and the object is tracked using texture data of the respective planes and geometric data between the respective planes to enable more precise tracking.

[0008] Additional aspects of embodiments will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the embodiments.

[0009] In accordance with an aspect of one or more embodiments, a 3D object tracking method includes modeling an object to be tracked to generate a plurality of planes, and tracking the plurality of planes, respectively.

[0010] The modeling of the object may include selecting points from among the plurality of planes, respectively, and calculating projective invariants using the selected points.

[0011] The tracking of the plurality of the planes may be achieved through model-based tacking.

[0012] In the tracking of the plurality of planes, when a tracking failure rate is below a designated level, virtual planes of planes, tracking of which has failed, may be generated, and the object may be tracked using the virtual planes.

[0013] In the tracking of the plurality of planes, when the tracking failure rate is more than the designated level, a tracking failure warning may be generated and the tracking may be stopped.

[0014] The designated level may be 50%.

[0015] In accordance with another aspect of one or more embodiments, a 3D object tracking apparatus includes an imaging device to capture an image of an object to be tracked, and a control unit to perform modeling of the object to be tracked to generate a plurality of planes, and to track the object by tracking the plurality of planes, respectively.

[0016] In order to perform the modeling of the object, the control unit may select points from among the plurality of planes, respectively and calculate projective invariants using the selected points.

[0017] The tracking of the plurality of the planes may be achieved through model-based tacking.

[0018] In the tracking of the plurality of planes, when a tracking failure rate is below a designated level, the control unit may generate virtual planes of planes, tracking of which has failed, and track the object using the virtual planes

[0019] In the tracking of the plurality of planes, when the tracking failure rate is more than the designated level, the control unit may generate a tracking failure warning and stop the tracking.

[0020] The designated level may be 50%.

[0026] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout.

[0027] FIG. 1 is a view exemplarily illustrating an image-based robot operating system to utilize a visual servoing technique. As shown in FIG. 1, at least one Charge-Coupled Device (CCD) camera 110 (i.e., an imaging device) is attached to the tip of an industrial robot hand 100, and thus recognizes and controls movement of the robot hand 100 according to a result of analysis of an image acquired based on the movement of the robot hand 100. Here, non-described reference numeral 10 represents an object to be handled by the robot hand 100.

[0028] A control unit 102 receives an image from the camera 110, tracks the object, and generates and provides a control command to control the movement of the robot hand 100 to the robot hand 100 based on a result of the tracking, thereby allowing the robot hand 100 to assume a desired precise motion in regard to the object 10. That is, the control unit 102 performs modeling of an object (a 3D model) to be tracked to generate a plurality of planes, such rectangular planes. Further, the control unit 102 selects points within the respective modeled planes, and calculates projective invariants at two positions by tracking these points. Through such a method, a model consisting of planes and points present on the planes is generated. When the modeling is completed, the control unit 102 performs tracking of the object 10. For this purpose, the control unit 102 tracks the plural planes of the generated model, respectively. While tracking the plural planes, the control unit 102 checks a geometric constraint between the respective planes. That is, the respective planes form a regular relation in a space according to the shape of the object, and, for example, in case of a rectangular parallelepiped, the geometric constraint may be a relation of an angle of 90 degrees between the respective planes. While tracking the object, when a tracking failure rate is more than a designated level, a tracking failure warning is generated and the tracking is stopped. On the other hand, when the tracking failure rate is below the designated level, it is judged that accuracy in tracking of the object is of a reliable level and virtual planes of planes, tracking of which has failed, are generated. Here, the virtual planes of the planes, tracking of which has failed, are generated with respect to a plane having the highest accuracy in tracking, i.e., having the minimum sum total of distance errors. In order to generate the virtual planes, the geometric constraint between the respective planes is used. Separately from tracking of the respective planes, the selected points are tracked. When the generation of the virtual planes of the planes, tracking of which has failed, is completed, positions of all the planes including the generated virtual planes are optimized. The optimization is achieved by calculating 3D rigid body transform matrices of all the planes and minimizing the sum total of distance errors using the Newton's method. When the optimization is completed, an optimized position of the overall object with respect to an estimated plane is calculated. Here, the position is calculated only using the planes having projective invariants more than a designated level, while the planes having projective invariants below the designated level are excluded. When the tracking is completed, the tracking is terminated.

[0029] According an embodiment, control unit 102 includes a computer to perform operations.

[0030] FIG. 2 is view illustrating projective invariants in an image-based robot operating system in accordance with an embodiment. As shown in FIG. 2, a method of calculating a projective invariant descriptor I from an arbitrary 2D shape is carried out by extracting a 2D outline from the image and calculating the projective invariant descriptor I from coordinates of points on the outline. Expressions below represent calculation of projective invariant descriptors from two 2D images, respectively.

[0031] $I_1=\frac{\det \left(P_5,P_2,P_3\right)\det \left(P_4,P_1,P_2\right)}{\det \left(P_4,P_2,P_3\right)\det \left(P_5,P_1,P_2\right)}$ $I_2=\frac{\det \left(P_5,P_3,P_1\right)\det \left(P_4,P_1,P_2\right)}{\det \left(P_4,P_3,P_1\right)\det \left(P_5,P_1,P_2\right)}$

[0032] FIG. 3 is a flow chart illustrating a 3D object tracking method in accordance with an embodiment. In FIG. 3, 302 represents a modeling process of the object (the 3D model) to be tracked, and 310 represents a tracking process of the object.

[0033] First, modeling of the object (the 3D model) to be tracked to generate a plurality of planes, such as rectangular planes, is performed (operation 304). For this purpose, two or more images of the corresponding object at different positions where calibration is carried out are obtained. Further, points within the respective modeled planes are selected (operation 306), and projective invariants at two positions are calculated by tracking the points (operation 308). Through the above method, a model consisting of planes and points present on the planes is generated.

[0034] Reference numeral 310 represents the tracking process of the object. During the tracking process (operation 310), the respective planes of the model generated in operation 302 are tracked (operation 312).

[0035] While tracking the respective planes, a geometric constraint between the respective planes is checked (operation 314). The respective planes form a regular relation in a space according to the shape of the object, and, for example, in case of a rectangular parallelepiped, the geometric constraint may be a relation of an angle of 90 degrees between the respective planes.

[0036] While tracking the object, when a tracking failure rate is more than 50% (yes in operation 316), a tracking failure warning is generated (operation 318), and the tracking is stopped. That is, if the number of tracked planes is 2n and tracking of n or more planes fails, it is judged that the tracking failure rate is more than 50%. In this case, accuracy in tracking of the object is excessively low and the tracking of the object is deemed to have failed, and thus the tracking is stopped.

[0037] On the other hand, if the tracking failure rate is below 50% (no in operation 316), it is judged that accuracy in tracking of the object is of a reliable level and virtual planes of planes, tracking of which has failed, are generated (operation 320). The virtual planes of the planes, tracking of which has failed, are generated with respect to a plane having the highest accuracy in tracking, i.e., having the minimum sum total of distance errors. In order to generate the virtual planes, the geometric constraint between the respective planes is used.

[0038] Separately from the tracking of the respective planes in operation 312, the points selected in operation 306 are tracked (operation 322).

[0039] When the generation of the virtual planes of the planes, tracking of which has failed, is completed, positions of all the planes including the generated virtual planes are optimized (operation 324). The optimization is achieved by calculating 3D rigid body transform matrices of all the planes and minimizing the sum total of distance errors using Newton's method.

[0040] When the optimization is completed, an optimized position of the overall object with respect to an estimated plane is calculated (operation 326). Here, the position is calculated using only the planes having projective invariants more than a designated level, while the planes having projective invariants below the designated level are excluded.

[0041] When the tracking is completed, the tracking is terminated (yes in operation 328. If the tracking is not completed (no in operation 328), the tracking process from operation 312 to operation 326 is repeated.

[0042] FIG. 4 is a view illustrating generation of a virtual plane of the object in accordance with an embodiment. As shown in FIG. 4, in order to generate a virtual plane 404 of a plane 402, tracking of which has failed, the virtual plane 404 of the plane 402, tracking of which has failed, is generated with respect to a plane having the highest accuracy in tracking, i.e., having the minimum sum total of distances of the edges. In order to generate the virtual planes, the geometric constraint between the respective planes is used.

[0043] As is apparent from the above description, in a 3D object tracking method and apparatus in accordance with an embodiment, a model of an object to be tracked is divided into a plurality of polygonal planes and the object is tracked using texture data of the respective planes and geometric data between the respective planes to enable more precise tracking.

[0044] Embodiments can be implemented using computing hardware (computing apparatus) and/or software, such as (in a non-limiting example) any computer that can store, retrieve, process and/or output data and/or communicate with other computers. The results produced may be displayable on a display device. A program/software implementing the embodiments may be recorded on non-transitory computer-readable media comprising computer-readable recording media. Examples of the computer-readable recording media include a magnetic recording apparatus, an optical disk, a magneto-optical disk, and/or a semiconductor memory (for example, RAM, ROM, etc.). Examples of the magnetic recording apparatus include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape (MT). Examples of the optical disk include a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc-Read Only Memory), and a CD-R (Recordable)/RW.

[0045] Further, according to an aspect of the embodiments, any combinations of the described features, functions and/or operations can be provided.

[0046] Although a few embodiments have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the claims and their equivalents.