Method of retrieving moving pictures using joint points based on pose information

[0002] 1. Field of the Invention

[0003] The present invention relates to a method of retrieving moving pictures using joint points based on pose information for allowing a user to retrieve pictures including a desired person object or the pose of the person object from still pictures or moving pictures.

[0004] 2. Description of the Prior Art

[0005] A conventional information retrieval method employs mainly a keyword based retrieval method using texts, as can be seen from web retrieval in the internet. However, as video and audio compression technology advances, it is necessary that the user can retrieve his/her desired information among large-capacity multimedia information.

[0006] The conventional information retrieval method can be called one of keyword retrieval methods using note to multimedia data based on a keyword. Also there has been proposed another retrival method using basic picture characteristics such as color, material, etc. However, the keyword retrieval method has a drawback that an available language is limited. Also the method has limitations in reflecting the contents of real multimedia data because it uses only the basic picture characteristic. In particular, since the objects such as persons are important content objects appearing in various genres such as movies, sports, etc, it is difficult for user to perform the retrival based on the contents of a person for which he/her desires to retrive from the person object, using the conventional technology.

[0007] It is an object of the present invention to solve the problems involved in the prior art, and to provide a method of retrieving moving pictures using joint points based on pose information, which can extract and represent joint points on joint point models from pictures existing in a desired major person the user desires among larger-capacity moving pictures/still pictures which are objects for retrieval, and retrives the person object using the extracted joint points.

[0008] To achieve the above object, the method of retrieving moving pictures using joint points based on pose information according to the present invention comprises the steps of extracting and expressing joint points for the joint point model against pictures where a major person exists from moving pictures or still pictures which are the object of retrieval by means of joint points model based indexing apparatus, and retrieving the person object according to the joint points extracted by said joint point model based indexing apparatus.

[0009] The present invention can be applied to all the other objects having joint points characteristic, for example animals such as hoarse, lion, etc., as well as person or personified objects.

[0010] The above object, and other features and advantages of the present invention will become more apparent by describing the preferred embodiment thereof with reference to the accompanying drawings, in which:

[0011] FIG. 1 shows one embodiment of joint points model for a person object.

[0012] FIG. 2 is a structural view of joint points based indexing apparatus.

[0013] FIG. 3 shows an example of characteristic point extraction for a person object.

[0014] FIG. 4 is a structural view of joint points model based retrieval system.

[0015] FIG. 5 shows an example of retrieval by means of query input.

[0016] Similar reference characters refer to similar parts in the several views of the drawings.

[0017] Now, the present invention will be explained in detail by reference to the accompanying drawings.

[0018] FIG. 1 shows one embodiment of joint points model for a person object. In the drawing, the major joint points for a person object are a total of 15 where they are joint points from 1(1-1) to 15(1-15). The above joint points are not limited to 15 but more than 15 joint points can be modeled for more detailed expression of poses.

[0019] FIG. 2 is a structural view of joint points based indexing apparatus. Assuming that moving pictures/still pictures data to become the object of retrieval are stored at a storage, it performs joint points extraction and contents based indexing process.

[0020] In FIG. 2, a reference numeral 2-1 is a moving picture/still picture data and indexing information storage, 2-2 is a scene change and key-frame detector, 2-3 is a user interface for joint point extraction, 2-4 is joint points model based joint point extraction and descriptor and 2-5 is joint points model based shot analysis and contents based video indexer.

[0021] Moving pictures from the moving pictures/still pictures data and indexing information storage 2-1 are input to the scene change and key-frame detector 2-2, which then divides the moving pictures into each time axis with shot by scene switching and then determines representative pictures corresponding to each shot. The shot is defined as a collection of pictures having a series of continuity which are arranged along the time axis within the moving pictures, in which respective shots are divided each other by the scene switching. The representative picture may be defined as a representative picture capable of representing the shots within one shot, in which more than one picture may exist per each shot. The reason for performing screen switching and representative picture detection against the moving pictures is that it requires a heavy work for a user to extract joint points using joint points model for all the pictures existing in a major person, in case of performing the joint point extraction of the present invention in a semi-manual way or in a manual way. Also, that is because that it can significantly reduce the amount of work necessary for extracting the joint point if the moving sequences are divided into shots and only joint point for the major person are extracted only against the representative pictures. In case of still picture, it is directly input into the joint point model based joint point extraction and descriptor 2-4 without passing through the scene change and key-frame detector 2-2.

[0022] The input to the joint point model based joint point extraction and descriptor 2-4 is a detected representative picture in case of moving pictures, but it is still pictures in case of still pictures. Furthermore, if there exists a person object within the input picture, joint points can be extracted through the user interface for joint point extraction 2-3 in case of manual/semi-manual detection. In case of automatic detection, the joint point is automatically extracted without intervention of the user.

[0023] FIG. 3 shows an example of characteristic point extraction for a person object.

[0024] In FIG. 3, there are shown the joint points extracted for two person objects, where the two extracted points each represented as a group of ‘+’ symbols and a group of ‘x’ symbols show the joint points each representing one person object, which are extracted based on the joint point model shown in FIG. 1.

[0025] If there exist person objects, the extracted joint point information corresponding to the representative pictures transmitted from the scene change and key-frame detector 2-2 and from the joint point model based joint point extraction and representer are input to the joint point model based shot analysis and contents based video indexer 2-5. The joint point model based shot analysis and contents based video indexer 2-5 performs the contents based indexing process which performs analysis of the contents within the shots from the input pictures and the corresponding joint points or of the relationship between the shots. The contents analysis into the shots includes pose tracking, etc. within the shots of the person object using color information extraction and representation for one person object on the basis of the joint points and pose information on the basis of the joint points. The analysis of the relationship between the shots may include an interaction of dialogue screen between persons, which commonly occur among moving picture data. Through this joint point based shot analysis and indexing, since an interact relationship between a series of shots is found in case of moving picture, a content analysis one step higher than the content based shot is made possible.

[0026] The indexing information output from the joint point model based shot analysis and contents based video indexer 2-5 is stored at a suitable position within the moving picture/still picture data and indexing information storage along with multimedia data and then is used for retrieval based on these indexing information upon content based retrieval.

[0027] FIG. 4 is a structural view of joint points model based retrieval system, which allows a user to retrieve he/his desired information using the above extracted and stored indexing information.

[0028] In FIG. 4, a reference numeral 4-1 is joint points model based user query and retrieval result interface, 4-2 is joint points model based query joint point descriptor 4-3 is an similarity degree value analysis and picture extractor, 4-4 is a similarity degree measuring means between query/person object and 4-5 is a moving picture/still picture and indexing information storage.

[0029] The joint point model based user query and retrieval result interface 4-1 functions to provide a query input interface for allowing an initial user to find a desired person object. If the query by the joint point model is input from the joint point model based user query and retrieval result interface 4-1 to the joint point model based query joint point descriptor 4-2, the joint point model based query joint point descriptor 4-2 generates the joint point model based representation from the query input by the user and then transmits it to the similarity degree measuring means between query/person object. The similarity degree measuring means between query/person object 4-4 functions to perform an similarity degree measurement on various characteristics between the indexing information stored at the storage and the person objects input by the query. At this time, the values of the similarity degree per each measured characteristic each is sent to the similarity degree value analysis and picture extractor 4-3. The similarity degree value analysis and picture extractor 4-3 receives from the moving picture/still picture and indexing information storage 4-5 representative pictures or still pictures being high at analogy degree in total into which a weighed value per each similarity degree is considered.

[0030] The total similarity degree indicates a synthetic similarity degree which is calculated from the analogy degree per each characteristic, by comparing the value represented by means of the joint point model per each characteristic for representative pictures or still pictures within each moving picture calculated from the similarity degree measuring means between query/person object 4-4 with the value represented by the joint point model for the query, considering the weighted value per characteristic specified by the user. The picture, being high in total similarity degree, is the picture what the user wants to find and which is similar to the query input, and it is transmitted to the user through the user interface of the joint point model based user query and retrieval result interface.

[0031] The characteristic similarity degree by means of the joint point model may includes a scale similarity degree of a person object, a position similarity degree, a pose similarity degree, a color information similarity degree, an in-time pose similarity degree within shot, a construction similarity degree between shot, etc.

[0032] Now, the above description will be further explained by reference to the accompanying drawings.

[0033] The joint point model based indexing apparatus shown in FIG. 2 extracts and stores joint points for a person object from moving pictures or still pictures which is the object of retrieval. At this time, the joint point model may include 15 joint point models as shown in FIG. 1. The joint point model in FIG. 1 may be expanded to represent a more detailed representation of a person object. In FIG. 2, if a moving picture is input from the 2-1 block, the 2-2 block detects screen switchings, divides them by shot unit and then extracts representative pictures from them. The representative pictures may be more than one picture, and the scene switching detection and the representative picture extraction technology may be used a conventional one. If a still picture is input from the moving picture/still picture data and indexing information storage 2-1, it is directly input to the joint point model based joint point extraction and descriptor 2-4. The joint point model based joint point extraction and descriptor 2-4 extracts and represents the joint point for the input picture on the basis of the joint point model, in case that a person object exits.

[0034] The user interface for joint point extraction 2-3 is a user interface block for extracting joint points for a person object in a manual way or a semi-manual way. If the joint point is automatically extracted, the user interface for joint point extraction 2-3 does not an additional function.

[0035] FIG. 3 shows an example of joint points which are extracted from the joint point model based joint point extraction and descriptor 2-4 through the user interface for joint point extraction 2-3. In FIG. 3, there exist two person objects within a picture, where points each corresponding to the joint points, respectively, are extracted on the basis of the joint point model for each person object. At this time, the points should be extracted according to the number per each joint on the basis of the joint point model shown in FIG. 2. The joint point model based joint point extraction and descriptor 2-4 in FIG. 2 functions to output the contents representing the joint points to the joint point model based shot analysis and contents based video indexer 2-5 according to the scheme representing the extracted joint points. The following mathematical Equation 1 is one example of joint point representation scheme which is expressed using a position vector for each of the joint points.

[0036] HPD₁={JP_i=(x_i,y_i)1≦i≦15},   [Equation 1]

[0037] where (x_i,y_i) is i-th joint point position vector,

[0038] (x_i,y_i)=(−1,−1), if i-th joint point does not exists.

[0039] The joint point model based shot analysis and contents based video indexer 2-5 analyzes additional contents of a picture from the contents representing the joint points which are transmitted from the joint point model based joint point extraction and descriptor 2-4. The additional picture contents analysis may include a color information, a pose information, a quality information, an occurrence frequency of a same person object and an interaction analysis between person objects on the basis of the joint points, and a semantic collection of a shot through the interaction analysis between person objects, etc. These information is indexed along with the information representing the joint points for the joint point model based person object and then stored at the moving picture/still picture data and indexing information storage 2-1.

[0040] When the indexing for the retrieval object data is completed by the joint point model based indexing apparatus in FIG. 2, the user can retrieve he/his desired person object using the joint point model based retriver in FIG. 4. Next, the user inputs the query joint point and its related information using the joint point model based user query and retrival result interface 4-1.

[0041] FIG. 5 shows one example of the query input and retrieval.

[0042] A query input window 5-1 is provided through the joint point model based user query and retrieval result interface 4-1 in FIG. 4, which allows the user to sketch the person object 5-2 what he/she wants to retrieve through the input window and then to input the joint points 5-3 on the basis of the joint point model. It also provides the retrieval result screen 5-4 to the user through the joint point model based user query and retrival result interface 4-1 in FIG. 4.

[0043] The detailed description for the retrieval procedure will be now explained.

[0044] The retrieval procedure for the person object is performed in the joint point model based retrieval apparatus in FIG. 4. At this time, the moving picture/still picture and indexing information storage 4-5 performs a same function to the moving picture/still picture data and indexing information storage 2-1 used in the joint point model based indexing apparatus of FIG. 2, and also stores moving pictures and still pictures being the object of retrival and joint points model based indexing information corresponding them.

[0045] The joint point model based query joint point descriptor 4-2 represents the joint points input through the query interface of the joint point model based user query and retrieval result interface 4-1, one example of which is represent as equation 1. At this time, the user can provide additional information to the joint point through the query interface. For example, they may include color information, pose informations, etc. and these information are input with the joint points concerned.

[0046] The similarity degree measuring means between query/person object 4-4 receives an indexing information from the moving picture/still picture and indexing information storage 4-5 and also joint points based query information from the joint point model based query joint point descriptor 4-2 to performs a similarity degree measurement. There may be a lot of similarity degree for the person object, which may include, for example position, pose, scale, etc.

[0047] One example of measuring the similarity degree for the position is as follows. In the present invention, the scale based on the distance of the second joint point from the query joint points and the joint points is the object of retrieval. The used scale may be expressed into the following equation 2. $\begin{array}{cc}\mathrm{Position}\mathrm{measure}=\frac{\max \mathrm{distance}-\Delta {\mathrm{JP}}_2}{\max \mathrm{distance}},\mathrm{where}\max \mathrm{distance}=\sqrt{{\mathrm{width}}^2+{\mathrm{height}}^2}\Delta {\mathrm{JP}}_2=\sqrt{{\left(x_{\mathrm{q2}}-x_{\mathrm{d2}}\right)}^2+{\left(y_{\mathrm{q2}}-y_{\mathrm{d2}}\right)}^2}& \left[\mathrm{Equation}2\right]\end{array}$

[0048] In equation 2, the width represents the number of pixels in a horizontal direction within a frame, the height represents the number of pixels in a vertical direction within a frame, (x_q2, Y_q2) represents the 2nd number among the query input joint points, i.e., JP₂, and (x_d2, y_d2) represents a position vector corresponding the 2nd number among the joint points of a major person within the retrieval object frame. The retrieval results may be obtained by selecting higher position_measure by means of similarity degree measure by equation 2.

[0049] Meanwhile, one example of similarity degree measurement method for the pose is as follows.

[0050] The pose similarity degree is a barometer necessary to measure how the pose between the query input by the user and the person within the frame being the object of retrieval is similar. In the present invention, it first obtains angle distribution based on the 2nd joint point to the joint point for the retrieval object person and then uses as a barometer the difference between the 2nd joint point and the 2nd joint point's reference angle distribution of the query input joint point. The used barometer can be expressed into the following equation 3. $\begin{array}{cc}\begin{array}{c}{\theta }_i=\frac{180}{\pi }A\mathrm{Tan}\left({\mathrm{JP}}_i-{\mathrm{JP}}_2\right)\\ =\frac{180}{\pi }A\mathrm{Tan}\left(\left(y_i-y_2\right)/\left(x_i-x_2\right)\right)\mathrm{if}{x}_i,y_i=-1\end{array}& \left[\mathrm{Equation}3\right]\end{array}$

[0051] In equation 3, the italic alphabet i is the value which is calculated from the angle of the 2nd joint point JP₂reference in the i-th joint point Jpi. This angle value of each joint point is calculated for all the existing joint points among the total 14 joint points except for the 2nd joint point.

[0052] In a same manner, after the angle is also calculated for the joint points input by query, the difference in the angle between the query joint point and the joint point being the object of retrieval can be obtained as equation 4. $\begin{array}{cc}{\mathrm{\Delta \theta }}_i=[\begin{array}{cc}\left|{\theta }_{\mathrm{di}}-{\theta }_{\mathrm{qi}}\right|& \mathrm{if}\left|{\theta }_{\mathrm{di}}-{\theta }_{\mathrm{qi}}\right|\le 180\\ 360-\left|{\theta }_{\mathrm{di}}-{\theta }_{\mathrm{qi}}\right|& \mathrm{if}\left|{\theta }_{\mathrm{di}}-{\theta }_{\mathrm{qi}}\right|>180\\ 0& \mathrm{if}{\mathrm{JP}}_{\mathrm{qi}},{\mathrm{JP}}_{\mathrm{di}}=\left(-1,-1\right)\end{array}& \left[\mathrm{Equation}4\right]\end{array}$

[0053] In equation 4, the italic alphabet represents the value of difference in the angle between the query joint point corresponding to the i-th joint point and the joint point of the retrieval object data. The difference in the angle will be 180° in maximum, but if there does not exist a corresponding joint point it will have the value of 0. The pose_measure between the query using the angle difference component and the retrieval object can be calculated as the following equation 5. $\begin{array}{cc}\mathrm{Pose}\mathrm{measure}=\frac{1}{M}\sum _{i*\left[1,3,\dots 15\right]}\frac{\left(\max \mathrm{angle}-{\mathrm{\Delta \theta }}_i\right)}{\max \mathrm{angle}},\mathrm{where}\max \mathrm{angle}=180M\mathrm{is}\mathrm{number}\mathrm{of}\mathrm{joint}\mathrm{points},{\mathrm{JP}}_{\mathrm{qi}}\mathrm{or}{\mathrm{JP}}_{\mathrm{di}}=\left(-1,-1\right)& \left[\mathrm{Equation}5\right]\end{array}$

[0054] In equation 5, M means the total number of the joint points where the query joint point or the joint point being the object of retrieval is commonly existing among the 14 joint points except for JP₂. In equation 5, the pose_measure can be calculated as the sum of the values in which the angle difference value of the joint points between the query calculated in equation 1 and the data being the object of retrieval is normalized.

[0055] The scale_measure within the screen of a person is used for the purpose of considering that the scale of the person object will be different if it has a same position and pose. The condition such as regardlessness of the scale and also the scale_measure can be input by not only considering the position and the pose but also considering the scale_measure with the query joint point. In other words, it measures the distance within the frame between JP₁and JP₂, for each of the query joint point and the joint point being the object of retrieval as a barometer and calculates the scale_measure as equation 6. $\begin{array}{cc}{\mathrm{JP}}_{1.2}=\sqrt{{\left(x_2-x_1\right)}^2+{\left(y_2-y_1\right)}^2}\mathrm{Scale}\mathrm{measure}=[\begin{array}{cc}{\mathrm{JP}}_{\mathrm{q1}\mathrm{.2}}/{\mathrm{JP}}_{\mathrm{d1}\mathrm{.2}},& \mathrm{if}{\mathrm{JP}}_{\mathrm{d1}\mathrm{.2}}\ge {\mathrm{JP}}_{\mathrm{q1}\mathrm{.2}}\\ {\mathrm{JP}}_{\mathrm{d1}\mathrm{.2}}/{\mathrm{JP}}_{\mathrm{q1}\mathrm{.2}},& \mathrm{if}{\mathrm{JP}}_{\mathrm{d1}\mathrm{.2}}<{\mathrm{JP}}_{\mathrm{q1}\mathrm{.2}}\end{array}& \left[\mathrm{Equation}6\right]\end{array}$

[0056] Each of the above similarity degrees is a barometer for position, pose and scale which represent the characteristic of a person object and retrieves a synthetic measure value as a basis by adding a weighted value to the similarity degree for each of the characteristics. The total similarity degree being a synthetic similarity is determined by equation 7.

[0057] Similarity=α·Position_measure+β·Pose_measure+γ·Scale_measure  [Equation 7]

[0058] In order to find a person object most similar to the query input, the present invention calculates the query joint points and each of the joint points of all the person object being the object of retrieval from the total similarity in equation 7, finds a representative frame in which a person object having the highest similarity exists or the pose of shot, and then outputs the result to the user.

[0059] The joint point model based user query and retrieval result interface 4-1 in FIG. 4 allows the user to control the weighted values α, β, γ for each of the position, pose and scale similarities.

[0060] Upon performing the joint point model based retrieval, if it performs the indexing in consideration of color information, pose information, relative position information between person objects, etc. in addition to the position, pose and scale, the total similarity degree can be calculated in consideration of the similarity degree for the corresponding characteristic in addition to the position, pose, scale similarity components in equation 7.

[0061] As described above, the retrieval method based on the joint point model according to the present invention is disclosed which can be used to find a desired scene on the basis of shape or pose of a person object from video data such as moving pictures, still pictures, etc. This joint point model based on the retrieval method will be especially effective when implementing a multimedia database by which a user can retrieve the contents concerned with a desired person object within a large capacity of a video data which is increased with a multimedia compression technology advanced in time. Furthermore, when using joint points model, the method according to the present invention allows various shapes of user query inputs by indexing various fundamental characteristic elements based on the joint point, for example color information, pose information, quality information, etc.

[0062] While the present invention has been described and illustrated herein with reference to the preferred embodiment thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.