US 5,625,417 AGrant
Image Processing System Using a Feature Point-Based Motion Estimation
Issue Date:1997-04-29
•2 Claims
•4 Drawing Sheets
Abstract
An apparatus, for use in a motion-compensated video signal encoder, for determining a predicted current frame based on a current and a previous frame, includes: a feature point motion vector(MV) detector for detecting a first set of MVs for feature points and a second set of MVs for edge points in the previous frame; a quasi-point detector for detecting quasi-feature points and a third set of MVs therefor, and quasi-edge points and a fourth set of MVs therefor; a non quasi-point MV detector for determining a quasi-feature point range for each of the quasi-feature points by indexing each of non-quasi-points to its closest quasi-feature point, and detecting a distance between a quasi-feature point and each of the remaining pixels in a quasi-feature point range and setting the distance as a search radius for the remaining pixel, and determining one or more quasi-feature point ranges dispersed within the search radius from each of the remaining pixels, and detecting a fifth set of MVs for the remaining pixels, each of the fifth set of MVs being determined by averaging motion vectors of the quasi-feature points included in the one or more quasi-feature point ranges; and a motion compensator for compensating the previous frame with the third, fourth and fifth sets of MVs.
Metadata
Assignee
- Daewoo Electronics Co. Ltd.
Inventor
- Min-Sup Lee
Application Information
Application Number:US 5276128
Filing Date:1995-09-13
Priority Date:1995-08-08
Art Unit:265
Classifications
IPC:
H04N 736
Field of Search:
348413;416;699
Patent Drawings (4 sheets)
Description
Field of the Invention
The present invention relates to an apparatus for encoding a video signal; and, more particularly, to an apparatus for encoding a video signal using a pixel-by-pixel motion estimation based on feature points.
Description of the Prior Art
As is well known, transmission of digitized video signals can deliver video images of a much higher quality than the transmission of analog signals. When an image signal comprising a sequence of image "frames" is expressed in a digital form, a substantial amount of data is generated for transmission, especially in the case of a high definition television system. Since, however, the available frequency bandwidth of a conventional transmission channel is limited, in order to transmit the substantial amounts of digital data through the limited channel bandwidth, it is inevitable to compress or reduce the volume of the transmission data. Among various video compression techniques, the so-called hybrid coding technique, which combines temporal and spatial compression techniques together with a statistical coding technique, is known to be most effective.
Most hybrid coding techniques employ a motion compensated DPCM(differential pulse code modulation), two-dimensional DCT(discrete cosine transform), quantization of DCT coefficients, and VLC(variable length coding). The motion compensated DPCM is a process of determining the movement of an object between a current frame and a previous frame, and predicting the current frame according to the motion flow of the object to produce a differential signal representing the difference between the current frame and its prediction. This method is described, for example, in Staffan Ericsson, "Fixed and Adaptive Predictors for Hybrid Predictive/Transform Coding", IEEE Transactions on Communications, COM-33, No. 12 (December 1985); and in Ninomiya and Ohtsuka, "A Motion-Compensated Interframe Coding Scheme for Television Pictures", IEEE Transactions on Communications, COM-30, No. 1 (January 1982).
Specifically, in the motion compensated DPCM, current frame data is predicted from the corresponding previous frame data based on an estimation of a motion between the current and the previous frames. Such estimated motion may be described in terms of two dimensional motion vectors representing the displacement of pixels between the previous and the current frames.
There have been two basic approaches to estimate the motion or displacement of pixels of an object: one is a block-by-block estimation; and the other is a pixel-by-pixel approach.
In the block-by-block motion estimation, a block in the current frame is compared with blocks in its previous frame until a best match is determined. From this, an interframe displacement vector (which indicates how much the block of pixels has moved between the frames) for the whole block can be estimated. However, in the block-by-block motion estimation, poor estimates may result if all pixels in the block do not move in a same way, to thereby decrease the overall coding efficiency.
In case of the pixel-by-pixel estimation, on the other hand, a displacement is determined for each and every pixel. Accordingly, this technique allows a more accurate estimation of pixel values and has the ability to easily handle scale changes (e.g., zooming, movement perpendicular to the image plane). However, in the pixel-by-pixel approach, since a motion vector is determined at each and every pixel, it is virtually impossible to transmit all of the motion vector data to a receiver.
One of the techniques introduced to ameliorate the problem of dealing with the surplus or superfluous transmission data resulting from the pixel-by-pixel approach is a feature point-based motion estimation method.
In the feature point-based motion estimation technique, motion vectors for a set of selected pixels, i.e., feature points, are transmitted to a receiver, wherein each of the feature points is defined as a pixel capable of representing its neighboring pixels so that motion vectors for non-feature points can be recovered or approximated from those of the feature points at the receiver.
In an encoder which adopts the motion estimation based on feature points, as disclosed in a copending commonly owned application, U.S. Ser. No. 08/367,520, entitled "Method and Apparatus for Encoding a Video Signal Using Pixel-by-Pixel Motion Estimation", a number of feature points are first selected from the pixels contained in the previous frame. Then, a first set of motion vectors for the selected feature points are determined, wherein each of the motion vectors represents a spatial displacement between one feature point in the previous frame and a corresponding matching point, i.e., a most similar pixel, in the current frame. The most similar pixel in the current frame is called a quasi-feature point. Thereafter, a second set of motion vectors for all of the pixels contained in the current frame is determined by using said first set of motion vectors. That is, one part of the second set of motion vectors for quasi-feature points is first determined by converting each of the first set of motion vectors; and the remaining part of the second set of motion vectors for non-quasi-feature points is determined by averaging the motion vectors for the quasi-feature points which are placed within a predetermined size of circular boundary.
Since the remaining part of the second set of motion vectors for non-quasi-feature points is determined without employing any actual information relative to the real motion of an object, it may not be able to accurately estimate the motion vectors for the non-quasi-feature points.
Summary of the Invention
It is, therefore, a primary object of the invention to provide an improved image processing system using a feature point-based motion estimation, which is capable of accurately estimating the motion vectors for non-quasi-feature points through the use of edge information.
In accordance with the invention, there is provided an improved apparatus, for use in a motion-compensated video signal encoder, for determining a predicted current frame based on a current frame and a previous frame of a digital video signal, comprising:
an edge detector for detecting edge points representative of pixels forming edges of an image in the previous frame;
a feature point detector for selecting a number of pixels among the edge points in the previous frame as feature points;
a feature point motion vector detector for detecting a first set of motion vectors for the feature points, each of the first set of motion vectors representing a spatial displacement between one of the feature points and a most similar pixel thereto in the current frame;
a range detector for determining a feature point range for each of the feature points, wherein the feature point range includes a feature point and one or more non-selected edge points lying on a same edge, and the distance along the edge between each of the non-selected edge points and the feature point within the feature point range is not greater than the distance along the edge between each of the non-selected edge points in the feature point range and a feature point in any other feature point range;
an edge point motion vector detector for detecting the distance between a feature point and each non-selected edge point in the feature point range and setting the distance as a search radius for said each non-selected edge point; and for determining, on a same edge, one or more feature point ranges dispersed within the search radius from each of the edge points, and detecting a second set of motion vectors for the non-selected edge points, each of the second set of motion vectors being determined by averaging the motion vectors of the feature points included in the one or more feature point ranges;
a quasi-point detector for detecting quasi-feature points in the current frame corresponding to the feature points and a third set of motion vectors for the quasi-feature points based on each of the first set of motion vectors and detecting quasi-edge points in the current frame based on each of the second set of motion vectors and a fourth set of motion vectors for the quasi-edge points;
a quasi-feature point range determiner for determining a quasi-feature point range for each of the quasi-feature points by indexing each of the remaining pixels in the current frame, excluding both the quasi-feature points and the quasi-edge points, to its closest quasi-feature point;
a non-quasi-point motion vector detector for detecting the distance between a quasi-feature point and each remaining pixel in the quasi-feature point range and setting the distance as a search radius for each remaining pixel; and for determining one or more quasi-feature point ranges dispersed within the search radius from each of the remaining pixels, and detecting a fifth set of motion vectors for the remaining pixels, each of the fifth set of motion vectors being determined by averaging the motion vectors of the quasi-feature points included in the one or more quasi-feature point ranges;
an arrangement block for arranging the third, the fourth and the fifth sets of motion vectors to determine a group of motion vectors for all of the pixels in the current frame; and
a motion compensator for providing a pixel value from the previous frame based on the position of each pixel in the current frame and a motion vector thereof, to thereby determine the predicted current frame.
Brief Description of the Drawings
The above and other objects and features of the present invention will become apparent from the following description of preferred embodiments given in conjunction with the accompanying drawings, in which:
FIG. 1 is an image signal encoding apparatus having a current frame prediction block of the present invention;
FIG. 2 shows a detailed block diagram of the current frame prediction block of FIG. 1;
FIGS. 3A to 3C describe a method for determining the feature point ranges on each of the edges; and
FIG. 4A and 4B represent a method for detecting a set of motion vectors for non-quasi-points.
Detailed Description of the Preferred Embodiments
FIG. 1 describes a preferred embodiment of the inventive image signal encoding apparatus. As shown in FIG. 1, a current frame signal of an input video signal is stored in a first frame memory 100 which is connected to a subtractor 102 through a line L9 and to a current frame prediction block 150 through a line L10.
In the current frame prediction block 150, a current frame signal on the line L10 retrieved from the first frame memory 100 and a reconstructed previous frame signal on a line L12 from a second frame memory 124 are processed to predict the current frame by using feature points to generate a predicted current frame signal onto a line L30 and a set of motion vectors for the feature points onto a line L20, respectively. Details of the current frame prediction block 150 will be described with reference to FIG. 2.
The predicted current frame signal on the line L30 is subtracted from the current frame signal on the line L9 at the subtractor 102, and the resultant data, i.e., an error signal denoting differential pixel values between the current and the predicted current frame signals, is dispatched to an image signal encoder 105, wherein the error signal is encoded into a set of quantized transform coefficients by using, e.g., a DCT and any of the known quantization methods. Thereafter, the quantized transform coefficients are transmitted to an entropy coder 107 and an image signal decoder 113. At the entropy coder 107, the quantized transform coefficients from the image signal encoder 105 and the motion vectors transmitted through the line L20 from the current frame prediction block 150 are coded together by using, e.g., a variable length coding technique. Thereafter, the coded signal is provided to a transmitter(not shown) for the transmission thereof.
In the meantime, the image signal decoder 113 converts the quantized transform coefficients from the image signal encoder 105 back to a reconstructed error signal by employing an inverse quantization and an inverse discrete cosine transform. The reconstructed error signal from the image signal decoder 113 and the predicted current frame signal on the line L30 from the current frame prediction block 150 are combined at an adder 115 to thereby provide a reconstructed current frame signal to be stored as a previous frame in the second frame memory 124.
Referring now to FIG. 2, there are illustrated details of the current frame prediction block 150 shown in FIG. 1. As shown in FIG. 2, a previous frame signal on the line L12 is provided to an edge detector 209 and a feature point motion vector detector 212, respectively. At the edge detector 209, a number of edge points representative of pixels forming edges of an image in the previous frame are detected through a conventional edge detection technique. The edge points in the previous frame are provided on a line L13 to a feature point detector 210, a range detector 214 and a quasi-point detector 218.
At the feature point detector 210, a plurality of feature points are detected among the edge points from the edge detector 209. In a preferred embodiment of the present invention, the feature points are detected at the intersection points of grid points and the edge points. The detection method is disclosed in a copending commonly owned application, U.S. Ser. No. 08/367,520, entitled "Method and Apparatus for Encoding a Video Signal Using Pixel-by-Pixel Motion Estimation", which is incorporated herein by reference in its entirely. The feature points from the feature point detector 210 are provided on a line L14 to the feature point motion vector detector 212 and the range detector 214.
At the feature point motion vector detector 212, a first set of motion vectors for the feature points is detected. Each of the first set of motion vectors represents a spatial displacement between a feature point in the previous frame and a most similar pixel thereto in the current frame called as a quasi-feature point. To detect a motion vector between a feature point and a quasi-feature point, there is used a block matching algorithm. That is, when a feature point is received from the feature point detector 210, a feature point block having the feature point at the center thereof, e.g., 5.times.5 pixels of the previous frame, is retrieved via the line L12 from the second frame memory 124(shown in FIG. 1). Thereafter, the motion vector for the feature point block is determined after a similarity calculation between the feature point block and each of a plurality of equal-sized candidate blocks included in a generally larger search region, e.g., 10.times.10 pixels, of the current frame retrieved from the first frame memory 100(shown in FIG. 1). The determined motion vector for the feature point block is assigned as the motion vector of the feature point contained in that feature point block.
After detecting the motion vectors for all of the feature points, the first set of motion vectors is provided to an edge point motion vector detector 216, a quasi-point detector 218 and the entropy coder 107(shown in FIG. 1) via the line L20.
In the meantime, at the range detector 214, each of the edges having a number of feature points is divided into a corresponding number of feature point ranges by indexing, in each of the edges, every edge point to a feature point which is closest thereto along the edge.
Referring to FIGS. 3A to 3C at this time, there is shown a method for determining the feature point ranges on each of the edges. As shown in FIG. 3A, in case that feature points, e.g., FP1 to FP5, are distributed along three edges EDGE 1, EDGE 2 and EDGE 3, an edge point EP on EDGE 2 is indexed to a feature point FP1 or FP5 depending on the degree of proximity therebetween. For instance, if the edge point EP is distanced from the feature points FP1 and FP5 by 5 and 3 pixels along the edge as shown in FIG. 3B, the edge point EP is indexed to the feature point FP5. In a similar manner, each of the edge points is indexed to a feature point which is closest thereto. If an edge point is disposed at the middle of two neighboring feature points, the edge point is indexed to either one of the two neighboring feature points. A feature point and edge points indexed thereto constitute a feature point range. In other words, each of the feature point ranges on an edge includes a feature point and one or more edge points, each of which having that feature point as a closest feature point thereto along the edge. Referring to FIG. 3C, there illustrated shown feature point ranges on the edges EDGE 1, EDGE 2 and EDGE 3.
After determining the feature point ranges on each of the edges, range information representing a feature point range to which edge points belongs is fed to the edge point motion vector detector 216. The edge point motion vector detector 216 first determines a search radius, i.e., the distance between an edge point and a feature point in a same feature point range, and detects one or more influential feature points on the same edge and calculates a motion vector of the edge point based on the motion vector of the detected influential feature points. As shown in FIG. 3B, if the search radius of the edge point EP is r and edge points included in feature point ranges FPR 1 and FPR 5 on the same EDGE 2 are included within the range defined by the search radius r, the feature points FP1 and FP5 are determined as the influential feature points of the edge point EP. A motion vector MV.sub.EP for the edge point EP may be calculated as: ##EQU1## wherein MV.sub.j is a motion vector for a jth feature point, M is the total number of feature point ranges in the range, L.sub.j is the distance between the jth feature point and the edge point EP; and w.sub.j is a weight factor for the jth feature point. If a given feature point is an influential feature point, the weight factor for the feature point is 1; and otherwise, 0. According to Eq. (1), the motion vector MVEP is determined by averaging the two motion vectors for the two feature points FP1 and FP5. The edge point motion vector detector 216 provides a second set of motion vectors for the edge points exclusive of the feature points.
Based on the edge point information from the edge detector 209 and the first and second sets of motion vectors from the motion vector detectors 212 and 216, the quasi-point detector 218 determines quasi-feature points and quasi-edge points in the current frame by shifting each of the feature points and the edge points by the motion vector thereof; and provides a third set of motion vectors for the quasi-feature points and a fourth set of motion vectors for the quasi-edge points. Since each of the first and the second sets of motion vectors represents the displacement between a pixel point in the current frame shifted from its corresponding pixel point in the previous frame, the magnitude of each of the motion vectors in the third and the fourth sets is identical to that of the corresponding motion vector in the first and second sets, except that the directions of the two motion vectors are opposite each other. The third and the fourth sets of motion vectors from the quasi-point detector 218 are provided via a line L16 to a quasi-feature point range determiner 220, a non-quasi-point motion vector detector 222 and an arrangement block 224.
At the quasi-feature point range determiner 220, each of the non-quasi-edge and non-quasi-feature points is assigned to a quasi-feature point range. The non-quasi-edge and non-quasi-feature points represent all of the remaining pixel points in the current frame excluding both the quasi-edge points and the quasi-feature points, and, hereinafter, each of them shall be called as "a non-quasi-point". As shown in FIG. 4A, when quasi-edges EDGE 4 and EDGE 5 are formed in the current frame by both quasi-feature points QF1 to QF4 and quasi-edge points, non-quasi-points between those two quasi-edges EDGE 4 and EDGE 5 are divided into four quasi-feature point ranges QFR1 to QFR4. The determination of the quasi-feature point ranges QFR1 to QFR4 is carried out by indexing each of the non-quasi-points to its closest quasi-feature point with a condition that none of the quasi-edges intervene therebetween. In other words, each of the quasi-feature point ranges includes non-quasi-points, each of which having a quasi-feature point as their closest quasi-feature point thereto.
After determining the quasi-feature point ranges for all of the non-quasi-points in the current frame, information for the quasi-feature point ranges is fed to a non-quasi-point motion vector detector 222.
At the non-quasi-point motion vector detector 222, a fifth set of motion vectors for the non-quasi-points is detected based on the quasi-feature point ranges. First, a distance between each of the non-quasi-points and a quasi-feature point is determined in a same quasi-feature point range and the determined distance is set to a search radius for the non-quasi-point. Determination of a motion vector for the non-quasi-point is carried out based on the search radius. That is, one or more quasi-feature point ranges included within the circular boundary formed with respect to a non-quasi-point by the search radius of that non-quasi-point is determined first. Quasi-feature points within those determined quasi-feature point ranges are influential quasi-feature points for the non-quasi-point. For instance, as shown in FIG. 4B, quasi-feature point ranges QFR1 to QFR4 are included in a circles with a search radius r and having a non-quasi-point NQP at the center thereof; and accordingly, quasi-feature points QF1 to QF4 are determined as the influential quasi-feature points for the non-quasi-point NQP. Motion vector MV.sub.NQP for the non-quasi-point NQP may be calculated as: ##EQU2## wherein MV.sub.i is a motion vector for an ith quasi-feature point, N is the total number of quasi-feature point ranges in the search radius r, L.sub.i is the distance between the ith quasi-feature point and the non-quasi-point NQP; and w.sub.i is a weight factor for the ith quasi-feature point. If the ith quasi-feature point is an influential quasi-feature point, the weight factor is 1; and otherwise, 0. The fifth set of motion vectors for the non-quasi-points is provided to an arrangement block 224.
The arrangement block 224 arranges the third set of motion vectors for the quasi-feature points, the fourth set of motion vectors for the quasi-edge points and the fifth sets of motion vectors for the non-quasi-points in order to determine a group of motion vectors for all of the pixels in the current frame. The group of motion vectors for all of the pixels in the current frame is sent to a motion compensator 226.
The motion compensator 226 provides a pixel value from the previous frame stored in the second frame memory 124 (shown in FIG. 1) based on the position data of each pixel in the current frame and a motion vector thereof, to thereby determine the predicted current frame.
While the present invention has been shown and described with respect to the, particular embodiments, it will be apparent to those skilled in the art that many changes and modifications may be made without departing from the spirit and scope of the invention as defined in the appended claims.
Claims
What is claimed is:
1. An apparatus, for use in a motion-compensated video signal encoder, for determining a predicted current frame based on a current frame and a previous frame of a digital video signal, comprising: means for detecting edge points representative of pixels forming edges of an image in the previous frame; means for selecting a number of pixels among the edge points in the previous frame as feature points; means for detecting a first set of motion vectors for the feature points, each of the first set of motion vectors representing a spatial displacement between one of the feature points and a most similar pixel thereto in the current frame; means for detecting a second set of motion vectors for non-selected edge points in the previous frame by using the first set of motion vectors and the edge points; means for detecting quasi-feature points in the current frame corresponding to the feature points and a third set of motion vectors for the quasi-feature points based on each of the first set of motion vectors and detecting quasi-edge points in the current frame based on each of the second set of motion vectors and a fourth set of motion vectors for the quasi-edge points; means for determining a quasi-feature point range for each of the quasi-feature points by indexing each of the remaining pixels in the current frame, excluding both the quasi-feature points and the quasi-edge points, to its closest quasi-feature point; means for detecting the distance between a quasi-feature point and each of the remaining pixels in a quasi-feature point range and setting the distance as a search radius for said each of the remaining pixels; means for determining one or more quasi-feature point ranges dispersed within the search radius from each of the remaining pixels, and detecting a fifth set of motion vectors for the remaining pixels, each of the fifth set of motion vectors being determined by averaging the motion vectors of the quasi-feature points included in said one or more quasi-feature point ranges; means for arranging the third, the fourth and the fifth sets of motion vectors to determine a group of motion vectors for all of the pixels in the current frame; and means for providing a pixel value from the previous frame based on a position of each pixel in the current frame and a motion vector thereof, to thereby determine the predicted current frame.
2. The apparatus of claim 1, wherein said means for detecting a second set of motion vectors includes: means for determining a feature point range for each of the feature points, wherein the feature point range includes a feature point and one or more non-selected edge points on a same edge, and the distance along the edge between each of the non-selected edge points and the feature point within the feature point range is not greater than the distance along the edge between each of the non-selected edge points in the feature point range and a feature point in any other feature point range; means for detecting the distance between a feature point and each non-selected edge point in the feature point range and setting the distance as a search radius for said each non-selected edge point; and means for determining, on the same edge, one or more feature point ranges dispersed within the search radius from each of the edge points, and by averaging the motion vectors of the feature points included in said one or more feature point ranges, thereby to detect the second set of motion vectors for the non-selected edge points.
Patent Citations (1)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US5546129 | 1996-08-01 | Lee |