Method, medium, and apparatus predicting direct mode motion of a multi-angle moving picture

[0001] This application claims the benefit of Korean Patent Application No. 10-2004-0061132, filed on Aug. 3, 2004, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.

[0002] 1. Field of the Invention

[0003] Embodiments of the present invention relate to moving picture coding and/or decoding, and more particularly, to a method, medium, and apparatus predicting direct mode motion of a multi-angle moving picture on a temporal and spatial basis.

[0004] 2. Description of the Related Art

[0005] MPEG-4 and H.264 moving picture coding has been used to code a moving picture taken by a single camera, with such standards focusing in large part on improvements in coding efficiency. However, in real communications or virtual reality, a single scene may be taken using a plurality of cameras, in order to form a moving picture from a plurality of angles, and thus a method, medium, and apparatus coding and decoding such a moving picture has become necessary.

[0006] MPEG (Moving picture Experts Group), which is a group of standards used for moving picture coding and decoding, has attempted to establish a standard for 3-dimensional audio/video coding by conducting a search for a standardization by organizing 3-D AV coding AHG (Ad-hoc Group) relating to 3-dimensional video coding and transmission. Thus, presently, a multi-angle 3-dimensional moving picture coding has not yet been standardized.

[0007] Embodiments of the present invention set forth a method, medium, and apparatus calculating a motion vector and a reference block for each block of a B picture included in a multi-angle moving picture.

[0008] To achieve the above and/or other aspects and advantages, embodiments of the present invention may include a method of predicting direct mode motion of a block in a multi-angle moving picture, including locating a block to which a co-located block, of the block for which coding is sought, makes reference with a motion vector, calculating a forward motion vector and/or a backward motion vector for the block for which coding is sought by making reference to a picture of an adjacent camera that exists in a direction parallel to a direction corresponding to the motion vector, when the reference picture is not taken by a same camera as a picture of the block for which coding is sought, and calculating forward and backward reference blocks of the block for which coding is sought by applying weighting to the calculated motion vectors.

[0009] The block to which the co-located block makes reference with the motion vector may be temporally previous to the co-located block and the block for which coding is sought. In addition, the co-located block may be temporally subsequent to the block for which coding is sought.

[0010] The calculating of the forward motion vector and/or the backward motion vector may be performed, when the reference picture is taken by the same camera as the picture for the block for which coding is sought, by predicting direct mode motion according to:

[0011] ${\mathrm{MV}}_0=\frac{{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{\mathrm{MV}}_C,{\mathrm{MV}}_1=\frac{{\mathrm{TD}}_D-{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{\mathrm{MV}}_C,$

[0012] wherein MV_C denotes the motion vector, TD_B denotes a distance from a B picture, of the block for which coding is sought, to a P picture of the co-located block, and TD_D denotes a distance from a P picture of the block to which the co-located block makes reference with the motion vector to the P picture of the co-located block.

[0013] Further, weighting may be based on how far the reference picture is away from the camera that took the picture of the block for which coding is sought.

[0014] When a P picture (f(j,t)) taken by a j^th camera at a time t makes reference to a picture (f(j−1,t−3)) taken by a (j−1)^th camera at a time t−3, and a picture to which a B picture taken immediately before the P picture makes forward reference exists between the picture taken by the (j−1)^th camera at the time t−3 and a picture (f(j,t−3)) taken by the j^th camera at the time t−3, the forward motion vector may given by:

[0015] ${\mathrm{MV}}_{\mathrm{F1}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{c}_{f\left(j-1,t-3\right)},{\mathrm{MV}}_{\mathrm{F2}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{c}_{f\left(j,t-3\right)},$

[0016] where TD_B indicates a distance between the B picture and the picture taken by the j^th camera at the time t−3, TD_D indicates a distance between the P picture and the picture taken by the j^th camera at the time t−3, MV|_{f(j−1, t−3)} indicates a motion vector calculated by making reference to the picture taken by the j−1^th camera at the time t−3 and C is a weight value, with reference pictures of the forward motion vectors MV_F1 and MV_F2 being R_F1 and R_F2, respectively, a forward reference picture is given by R_F=(3R_F1+R_F2+2)/4.

[0017] When a P picture (f(j,t)) taken by a j^th camera at a time t makes reference to a picture (f(j−1,t−3)) taken by a (j−1)^th camera at a time t−3, and a picture to which a B picture taken immediately before the P picture makes forward reference exists between the picture taken by the (j−1)^th camera at the time t−3 and a picture (f(j,t−3)) taken by the j^th camera at the time t−3, the backward motion vector is given by:

[0018] ${\mathrm{MV}}_{\mathrm{B1}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B-{\mathrm{TD}}_D}{{\mathrm{TD}}_D}{c}_{f\left(j,t\right)},{\mathrm{MV}}_{\mathrm{B2}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B-{\mathrm{TD}}_D}{{\mathrm{TD}}_D}{c}_{f\left(j+1,t\right)},$

[0019] where TD_B indicates a distance between the B picture and the picture taken by the j^th camera at the time t−3, TD_D indicates a distance between the P picture and the picture taken by the j^th camera at the time t−3, MV|_f(j,t) indicates a motion vector calculated by making reference to the picture taken by the j^th camera at the time t, and C is a weight value, with reference pictures of the backward motion vectors MV_B1 and MV_B2 being R_B1 and R_B2, respectively, a backward reference picture is given by R_B=(3R_B1+R_B2+2)/4.

[0020] To achieve the above and/or other aspects and advantages, embodiments of the present invention may include an apparatus for predicting direct mode motion of a block a multi-angle moving picture, the apparatus may include a determiner locating a block to which a co-located block, of the block for which coding is sought, makes reference with a motion vector, a motion vector calculator calculating a forward motion vector and/or a backward motion vector for the block for which coding is sought by making reference to a picture of an adjacent camera that exists in a direction parallel to a direction corresponding to the motion vector, when a reference picture is not taken by a same camera as a picture of the block for which coding is sought, and a reference picture calculator calculating forward and backward reference blocks of the block for which coding is sought by applying weighting to the calculated motion vectors.

[0021] Here, the apparatus may be a coding and/or decoding apparatus coding and/or decoding the moving picture. In addition, the coding and/or decoding apparatus may correspond to 3-dimensional video coding/decoding.

[0022] To achieve the above and/or other aspects and advantages, embodiments of the present invention may include at least one medium including computer readable code/instructions implementing embodiments of the present invention.

[0023] Additional aspects and/or advantages of the invention will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the invention.

[0035] Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below to explain the present invention by referring to the figures.

[0036] FIG. 1 illustrates how a motion vector of a B picture 110 may be calculated. As illustrated, when a soccer ball moves from an upper left region to a lower right region of a screen in a soccer game. Each frame may be called a picture. A moving picture can include I pictures, B pictures, and P pictures, in an IBBPBBP configuration or an IBPBPBP configuration. Each of the B pictures can be used to predict motion for coding according to one of 5 modes, including a forward prediction mode, a backward prediction mode, a bidirectional prediction mode, a direct prediction mode, and an intra prediction mode. In the direct prediction mode, motion of a block of the B picture 110, for which coding is presently sought, can be predicted based on a motion vector of a co-located block of a P picture 120, which is a temporally subsequent picture, being used to obtain a forward motion vector MV₀ and a backward motion vector MV₁.

[0037] FIG. 2 illustrates a direct prediction mode of the B picture 210. In order to calculate the forward motion vector MV₀ and the backward motion vector MV₁ of a direct mode block 202, for which motion prediction is sought, in the B picture 210, the forward motion vector MV_c is located as to a reference list 0 (RL₀) picture 230, to which a co-located block 204 makes reference, by a motion vector located in the same location in a reference list 1 (RL₁) picture 220, which is a temporally subsequent picture, as the direct mode block 202 in the B picture 210. The forward motion vector MV₀ and the backward motion vector MV₁ of the direct mode block 202 of the B picture 210 can be represented by Equation 1, as follows.

[0038] $\begin{array}{cc}{\mathrm{MV}}_0=\frac{{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{\mathrm{MV}}_C,{\mathrm{MV}}_1=\frac{{\mathrm{TD}}_D-{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{\mathrm{MV}}_C& \mathrm{Equation}1\end{array}$

[0039] Here, MV_C denotes a motion vector of the co-located block 204 of the reference list 1 (RL₁) picture 220, TD_B denotes the distance from the B picture 210 to the reference list 0 (RL₀) picture 230, and TD_D denotes the distance from the reference list 0 (RL₀) picture 230 to the reference list 1 (RL₁) picture 220.

[0040] FIG. 3A illustrates locations of cameras that may be used to obtain a multi-angle moving picture. Referring to FIG. 3A, in this embodiment, the plurality of cameras are arranged along a j-axis. Such an arrangement of the cameras can be used to obtain a moving picture when a single scene is taken from several angles.

[0041] FIG. 3B illustrates pictures taken by cameras for obtaining a multi-angle moving picture, with respect to time. If a picture taken by a camera at location j and time t is f(j,t), then, for example, f(0,0) can correspond to a picture taken by a camera at a center location at a beginning of a recording. Pictures taken by each of the cameras have been arranged along the time axis, with an angle θ between a line extending from a current frame in a camera to a subsequent frame in an adjacent camera and the j-axis. As illustrated, the units of the j-axis may correspond to a separation between cameras, the time axis may correspond to units of time intervals between subsequent pictures, and the angle θ may be 45°, noting that embodiments of the present invention are not limited thereto.

[0042] FIG. 4 is a graph illustrating a direct prediction mode of a multi-angle moving picture, according to an embodiment of the present invention. The aforementioned method of predicting direct mode motion, as described with reference to FIG. 2, may be used only when a reference picture indicated by the motion vector f(5,6), taken subsequently by the same camera that had taken f(5,5), is a picture that exists on the same j axis as the f(5,5) picture when predicting motion of the f(5,5). However, in the case of a multi-angle moving picture, a reference picture indicated by the motion vector of f(5,6), is not necessarily a picture taken by the same camera as f(5,5), and may be f(3,3), for example. In this case, the reference picture of f(5,5) should be predicted toward f(3,3), and toward f(6,6). Likewise, when a reference picture of f(−4,9) taken by a different camera is f(−1,6), the direct prediction mode described with reference to FIG. 2 may not be used, and a motion vector calculated by the direct prediction mode should be predicted toward f(−1,6), which is a reference picture indicated by f(−4,9).

[0043] According to an embodiment of the present invention, a method of applying a direct mode in a multi-angle moving picture will now be described in greater detail. A change in camera distance to change in time Δj/Δt can be calculated based on the picture to which a co-located block of a subsequent picture of the B picture, for which the motion prediction is sought, makes reference. Calculation of the motion vector will now be described for four cases below according to the calculated Δj/Δt values.

[0044] First, in the case where ΔjΔt=0, since f(j,t) would have been taken by the same camera as a reference picture, the forward motion vector MV₀ and the backward motion vector MV₁ can be calculated using Equation 1 above.

[0045] As only an example, FIG. 5A illustrates direct prediction when (Δj/Δt is ⅓, according to an embodiment of the present invention. Referring to FIG. 5A, f(j,t) 505, which is a P picture, makes reference to f(j−1,t−3) 510 taken by a (j−1)^th camera. It can be thus seen that a picture to which a B picture 515 makes forward reference exists between f(j−1,t−3) 510 and f(j,t−3) 520, and is spatially much closer to and more similar to f(j−1,t−3) 510. That is, since a picture does not exist between f(j−1,t−3) 510 and f(j,t−3) 520, the respective motion vectors of f(j−1,t−3) 510 and f(j,t−3) 520 may be predicted.

[0046] Here, forward motion vectors MV_F1 and MV_F2 make reference to f(j−1,t−3) 510 and f(j,t−3) 520, respectively, and can be predicted by Equation 2, below.

[0047] $\begin{array}{cc}{\mathrm{MV}}_{\mathrm{F1}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{c}_{f\left(j-1,t-3\right)},{\mathrm{MV}}_{\mathrm{F2}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{c}_{f\left(j,t-3\right)}& \mathrm{Equation}2\end{array}$

[0048] Here, TD_B indicates the distance between the B picture 515 and f(j,t−3) 520, and TD_D indicates the distance between the f(j,t) 505 and f(j,t−3) 520. If reference pictures of MV_F1 and MV_F2 are considered R_F1 and R_F2, respectively, a forward reference picture R_F is given by (3R_F1+R_F2+2)/4. The weight of 3 may be given to R_F1 since it is spatially closer to the reference picture of MV_F1. C_{ƒ(j−1,t−3)}, for example, is a weighting designated for f(j−1,t−3).

[0049] Likewise, it can be seen that a picture to which the B picture 515 makes backward reference exists between f(j,t) 505 and f(j+1,t) 525, and may be spatially much closer to and more similar to f(j,t) 505.

[0050] Backward motion vectors MV_B1 and MV_B2 make reference to f(j,t) 505 and f(j+1, t) 525, respectively, and can be predicted by Equation 3, below.

[0051] $\begin{array}{cc}{\mathrm{MV}}_{\mathrm{B1}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B-{\mathrm{TD}}_D}{{\mathrm{TD}}_D}{c}_{f\left(j,t\right)},{\mathrm{MV}}_{\mathrm{B2}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B-{\mathrm{TD}}_D}{{\mathrm{TD}}_D}{c}_{f\left(j+1,t\right)}& \mathrm{Equation}3\end{array}$

[0052] If reference pictures of MV_B1 and MV_B2 are considered R_B1 and R_B2, respectively, a backward reference picture R_B may be given by (3R_B1+R_B2+2)/4, for example.

[0053] According to another embodiment of the present invention, FIG. 5B illustrates direct prediction when Δj/Δt is ⅔. Referring to FIG. 5B, f(j,t) 505 makes reference to f(j−2,t−3) 530, taken by a (j−2)^th camera. It can be thus seen that a picture to which the B picture 515 makes forward reference exists between f(j−2,t−3) 530 and f(j−1,t−3) 510, and may be spatially much closer to and more similar to f(j−2,t−3) 530, for example.

[0054] Forward motion vectors MV_F1 and MV_F2 make reference to f(j−2,t−3) 530 and f(j−1,t−3) 510, respectively, and can be predicted by Equation 4, for example.

[0055] $\begin{array}{cc}{\mathrm{MV}}_{\mathrm{F1}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{c}_{f\left(j-2,t-3\right)},{\mathrm{MV}}_{\mathrm{F2}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{c}_{f\left(j-1,t-3\right)}& \mathrm{Equation}4\end{array}$

[0056] If reference pictures of MV_F1 and MV_F2 are considered R_F1 and R_F2, respectively, a forward reference picture R_F may be given by (R_F1+3R_F2+2)/4. The weight of 3 can be given to R_F2 since it is spatially closer to the reference picture of MV_F2, for example

[0057] Likewise, a picture to which the B picture 515 makes backward reference exists between f(j,t) 505 and f(j+1,t) 525, and may be spatially much closer to and more similar to f(j+1,t) 525.

[0058] Accordingly, backward motion vectors MV_B1 and MV_B2 make reference to f(j,t) 505 and f(j+1,t) 525, respectively, and can be predicted by Equation 5, below.

[0059] $\begin{array}{cc}{\mathrm{MV}}_{\mathrm{B1}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B-{\mathrm{TD}}_D}{{\mathrm{TD}}_D}{c}_{f\left(j,t\right)},{\mathrm{MV}}_{\mathrm{B2}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B-{\mathrm{TD}}_D}{{\mathrm{TD}}_D}{c}_{f\left(j+1,t\right)}& \mathrm{Equation}5\end{array}$

[0060] If reference pictures of MV_B1 and MV_B2 are considered R_B1 and R_B2, respectively, a backward reference picture R_B may be given by (R_B1+3R_B2+2)/4, for example.

[0061] According to still another embodiment of the present invention, FIG. 5C illustrates direct prediction when Δj/Δt is 1. Referring to FIG. 5C, f(j,t) 505 makes reference to picture f(j−3,t−3) 540, taken by a (j−3)^th camera. As illustrated can be thus seen that a picture to which the B picture 515 makes forward reference is f(j−2,t−3) 530.

[0062] Here, a forward motion vector MV_F making reference to f(j−2,t−3) 530 may be given by Equation 6, below.

[0063] $\begin{array}{cc}{\mathrm{MV}}_F={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B}{{\mathrm{TD}}_D}{c}_{f\left(j-2,t-3\right)}=R_F& \mathrm{Equation}6\end{array}$

[0064] Likewise, since the B picture 515 makes backward reference to f(j+1,t) 525, a backward motion vector MV_B making reference to f(j+1,t) 525 may be given by Equation 7, below.

[0065] $\begin{array}{cc}{\mathrm{MV}}_{\mathrm{B1}}={\mathrm{MV}}_C\frac{{\mathrm{TD}}_B-{\mathrm{TD}}_D}{{\mathrm{TD}}_D}{c}_{f\left(j+1,t\right)}=R_B& \mathrm{Equation}7\end{array}$

[0066] In Equations 2 through 7, MV|_x, e.g., MV|_{f(j−1, t−3)}, references a motion vector that makes reference to f(j−1,t−3), again for example. In the foregoing examples, the weight can be varied case by case. For example, the weighting may be (0,1), (1, 0) instead of (3, 1), (1, 3). For example, the weight can be further defined by Equation 8, below.

[0067] $\begin{array}{cc}c=\frac{c_p\left({\mathrm{TD}}_D-{\mathrm{TD}}_B\right)+c_S{\mathrm{TD}}_B}{{\mathrm{TD}}_D}& \mathrm{Equation}8\end{array}$

[0068] Here, C_p can be the prediction block of a previous reference picture, and C_s can be the prediction block of a subsequent reference picture. In the H.264 standard, each picture can have a plurality of reference frames. When each picture has five reference frames, the reference frames for f(j,t) may be f(j−1,t−3), f(j,t−3), f(j+1,t−3), f(j,t−6), and f(j,t−9), for example. Here, the method of predicting two-dimensional direct mode motion, according to an embodiment of the present invention, may be used only when Δj/Δt is ⅓. When each picture has seven reference frames, the reference frames for f(j,t) may be f(j−2,t−3), f(j−1,t−3), f(j,t−3), f(j+1,t−3), f(j+2,t−3), f(j,t−6), and f(j,t−9), for example. Here, the method of predicting direct mode motion in a multi-angle moving picture, according to an embodiment of the present invention, may be used only when Δj/Δt is ⅓, for example.

[0069] FIG. 6 is a flowchart illustrating a method of predicting direct mode motion of a multi-angle moving picture, according to an embodiment of the present invention.

[0070] A block to which the co-located block of a block, for which coding is sought, makes reference with a motion vector can be located (operation 610). Where a reference picture to which the located block belongs is not taken by the same camera at the same location as the picture to which the block for which coding is sought belongs, a forward motion vector and a backward motion vector can be calculated by making reference to pictures of an adjacent camera that exists in the direction parallel to the direction corresponding to the motion vector (operation 620). Calculation of the forward motion vector and the backward motion vector varies depending on how far away the camera is from the camera that took the picture to which the block for which coding is sought belongs, which has been described with reference to FIGS. 5A through 5C, for example. When a reference picture is taken by a camera at the same location as the camera that took the picture to which the block for which coding is sought belongs, forward and backward motion vectors can also be calculated according to a conventional method of predicting direct mode motion, as described with reference to Equation 1, for example.

[0071] Forward and backward reference blocks of the block for which coding is sought are calculated by applying weights to the calculated motion vectors (operation 630).

[0072] FIG. 7 is a block diagram of an apparatus for predicting a direct mode motion of a multi-angle moving picture, according to an embodiment of the present invention.

[0073] Referring to FIG. 7, a determiner 710 can locate a block to which the co-located block of a block for which coding is sought makes reference with a motion vector. A motion vector calculator 720 can calculate forward and backward motion vectors. When a reference picture to which the located block belongs is not taken by the same camera as the picture to which the block for which coding is sought belongs, a forward motion vector and a backward motion vector can be calculated by making reference to pictures of an adjacent camera that exists in the direction parallel to the direction corresponding to the motion vector. Calculation of the forward motion vector and the backward motion vector depends on how far away the camera is from the camera that took the picture to which the block for which coding is sought belongs, which has been described with reference to FIGS. 5A through 5C, for example. When a reference picture is taken by a camera at the same location as the camera that took the picture to which the block for which coding is sought belongs, forward and backward motion vectors may be calculated according to a conventional method of predicting direct mode motion, as described with reference to Equation 1, for example.

[0074] A reference picture calculator 730 can calculate forward and backward reference blocks of the block for which coding is sought by applying weighting to the calculated motion vectors.

[0075] Embodiments of the present invention may also be embodied as computer readable code/instructions on a medium, e.g., computer readable recording medium. The medium may be any data storage device that can store/transfer data which can be thereafter read by a computer system. Examples of the medium/media can include read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and carrier waves, noting these are only examples. The medium can also be distributed network coupled computer systems so that the computer readable code is stored/transferred and executed in a distributed fashion. Also, functional programs, codes and code segments for implementing embodiments of the present invention can be easily construed by programmer skilled in the art to which the present invention pertains.

[0076] As described above, embodiments of the present invention can perform coding and decoding by locating motion vectors of previous and subsequent pictures without predicting motion, for example, thereby reducing the amount of calculation and performance time.

[0077] While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents. The above embodiments should be considered in descriptive sense only and not for purposes of limitation.