Layered video coding using perceptual coding criteria for error resilience in packet networks

[0002] I. Field of the Invention

[0003] The present invention relates to a video encoder for coding of a video signal prior to transmission through a medium. More particularly, the present invention pertains to a technique for encoding a signal using a perceptual preprocessor for considering human vision perceptual limitations.

[0004] II. Description of the Related Art

[0005] Video information in the form of a data bit stream is typically transmitted over a medium, e.g. the internet, in a packetized stream. Such video information is obtained from a video camera which generates a video signal containing a series of successive-in-time video frames depicting video images. Each frame consists of a number of picture elements or pixels which are divided into groups to form blocks. The video signal is processed, frame-by-frame, and encoded when necessary, such as when an error signal is detected between two adjacent-in-time frames, i.e. between a present frame and a prior frame. In this manner, only data affecting a change in object motion or pixel intensity need be transmitted, thus yielding an efficient use of available transmitter communication bandwidth.

[0006] For example, in a typical video signal encoder scheme each frame is divided into macro blocks of 16×16 pixels and an error signal representing the difference between pixel intensity of a current macro block and a corresponding prior macro block is obtained. The error signal is then converted from the spatial domain to the frequency domain by a transform function, such as a discrete cosine transform device. The digitized error signal is then quantized and entropy coded for transmission to a receiver. A feedback loop is provided which uses the quantized error signal to compensate for and estimate motion of objects in a current frame by comparing the position of the objects to a position from the prior frame.

[0007] Video data transmitted in packets is usually protected by a parity check scheme as known by those having ordinary skill in the art. Thus, the majority of internet video errors results from the loss of an entire packet during, for example, network congestion and buffer overflow. Such packet loss causes single or multiple macro block losses in the decoding process which results in severe degradation in perceived quality and error propagation. To combat video signal degradation and error propagation resulting from packet loss, layered video signals are used, wherein the video signal is divided into a base layer and an enhancement layer. The enhancement layer typically provides refinement information that is added to the base layer to provide higher quality, and which is transmitted with lower priority than the base layer. A known alternative approach is to use the second or enhancement layer to transmit a redundant version of the base layer when high losses are detected. Thus, the base layer will be transmitted on two distinct data streams with the hope that at least one of the streams will be correctly received.

[0008] The drawbacks of such prior art schemes is that they fail to adequately take into account the limitations of human visual perception when viewing a video signal. For example, not all lost video information, in the form of lost data packets, is visually perceptible by a viewer. Thus, the redundant transmission of such information is an inefficient use of bandwidth, as such information is not necessary and will not result in an appreciable difference to a viewer of the video information that is contained in the redundantly transmitted data packets.

[0009] The present invention is directed to a layered video encoder which considers human video perception limitations in determining whether to re-transmit lost video packet information. The inventive encoder is a block based coder that processes a single macro block portion of a frame of a video signal at any given time by comparing intensity values of picture elements contained in a subject macro block of a current frame to intensity values of corresponding picture elements contained in the subject macro block of a prior frame adjacent-in-time to the current frame. In the event that a transmission error is detected, such as when the subject macro block is not received successfully by a receiver, it may be necessary for the subject macro block to be re-transmitted. A perceptual preprocessor is used to determine, according to a set of criteria, whether to re-transmit the subject macro block and to what priority the re-transmitted macro block should be assigned. The criteria used to determine if and how retransmission of a subject macro block should occur are based, at least in part, on limitations in human visual perception. If the information contained in the subject macro block is of the type which is highly perceivable by a human viewer, the subject macro block will be re-transmitted at a high priority by encoding the subject macro block onto a base layer and an error resilience layer of data. If the information is at a lower priority wherein the information contained in the subject macro block would be only slightly, if at all, perceived by a human viewer, then the subject macro block will be re-transmitted at a lower priority by encoding the subject macro block onto the enhancement layer, or by simply not re-transmitting the data.

[0010] Other objects and features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed solely for purposes of illustration and not as a definition of the limits of the invention, for which reference should be made to the appended claims.

[0011] In the drawings:

[0012] The FIGURE depicts a block diagram of an inventive block based video encoder in accordance with the present invention.

[0013] A block diagram of the inventive video encoder is depicted in the figure and identified by the general reference numeral 10. Encoder 10 includes an input port 5 which receives a block of video information from a video signal. For example, a source such as a video camera (not shown) or the like generates a video signal in the form of a plurality of successive-in-time frames, with each frame being represented by a fixed number of picture elements (pixels) having intensity values for reproducing images of objects contained on a subject frame when the video signal is received by a video receiver. Each frame is divided into blocks of pixels by a block splitter (not shown), as is known in the art, such as by dividing each frame into macro blocks sized at 16×16 pixels. For each frame, each macro block is provided, one at a time, to input port 5 for encoding.

[0014] As is known in the art, in order to achieve efficient use of bandwidth, among other reasons, each frame of video information is not individually encoded and then transmitted for reproduction at a down stream video display, such as a CRT. Rather, only the differences in the form of pixel intensity and object motion, represented as “F” and “M”, respectively, in the figure between a current frame and a presently displayed (e.g. prior-in-time) frame on a display are encoded and transmitted so that the presently displayed frame need only be altered by such differences to thereby display the current frame information on the display. Such differences are usually referred to as error signals or difference signals and are represented in the figure as “D_k”, wherein k refers to the frame designation.

[0015] The intensity values of a current frame F_kare provided to input port 5 and received by a subtractor 18 which produces an error signal D_k, and to a motion estimator 48 for estimating motion of an object in the current frame by comparing it to a prior frame adjacent-in-time to the current frame. The current frame is also provided to a perceptual preprocessor 50 for determining, as explained more fully below, whether differences between a current frame which is subject to transmission errors and a prior frame are such as are noticeable to a viewer's visual perception. Depending on certain criteria, the current frame that experienced transmission errors may be encoded and re-transmitted to a receiver.

[0016] Encoder 10 is a layered encoder which, in the preferred embodiment, generates two layers of coded data, namely, a base layer and an enhancement or error resilience layer. The base layer contains necessary information for the video signal and is, therefore, transmitted at a high priority. The enhancement layer contains video enhancing information which is transmitted at a lower priority than the base layer. Encoder 10 includes a first encoder branch 12 for encoding an error signal of a current frame D_konto the base layer. The first encoder branch 12 includes a transformer 20, such as a discrete cosine transform (DCT) device for converting the error signal D_kfrom the spatial domain to the frequency domain whereupon the converted signal is quantized by a quantizer 22. In the preferred embodiment, quantizer 22 has an adjustable step size to vary the quantization of the transformed error signal between a coarse step and a fine step. The quantized signal is then provided to a node 23 and to an entropy coder 24 which operates, as is known by those having ordinary skill in the art, to code the quantized signal. The resulting coded signal is finally provided to a buffer (not shown) and transmitted to a receiver (not shown).

[0017] Encoder 10 also includes a second encoder branch 14 for coding an error signal D* for the enhancement or error resilience layer. Like first encoder branch 12, branch 14 includes a DCT transformer 30, a quantizer 32 and an entropy coder 34, all of which function in a similar manner to like devices in the first branch 12. The coded error resilience layer is also provided to a buffer (not shown) and is transmitted at a lower priority than that of the base layer.

[0018] A feedback loop 16 connected between node 23 and subtractor 18 is arranged for obtaining information on prior-in-time frames and is used for, inter alia, estimating the motion of an object as it moves across a display field from one frame to the next. Feedback loop 16 contains an inverse transform device 40, such as an IDCT, which re-converts the quantized error signal D_kfrom the frequency domain to the spatial domain, shown as {circumflex over (D)}_k. Feedback loop 16 includes a frame store or delay unit 44, a motion compensator 46 and a motion estimator 48, which are used for producing an estimate of the current block, shown as signal {tilde over (F)}_k, that is provided to subtractor 18 for producing the error signal D. The estimate of the current block consists of a prediction of the image of the current block derived from the motion compensator 46—which derives the prediction or estimate from the image of a previous block—as well as a displacement measurement of the image. The displacement measurement is typically in the form of a motion vector generated by the motion estimator 48 which measures a displacement between images by comparing the current block to a corresponding previous block that was stored in delay 44. In the event that there is no movement of an object from a prior frame to a current frame, there is no motion to be estimated, in which case the motion estimator will function as a simple conditional replenishment coder.

[0019] The inventive encoder 10 includes a perceptual preprocessor 50 which considers and takes into account limitations inherent in the visual perception of a human in determining whether a lost or corrupted frame should be re-transmitted. For example, if a change in intensity and motion of a current frame to a prior frame is only slight such that the change would not be easily detectable by a viewer, it will be unnecessary to re-code and retransmit such a frame in the event that the frame is lost or corrupted as a result of an error such as a buffer overflow, etc. Depending on the change in luminance and motion, the corrupted or missing frame (“subject frame”) may require high priority because the information contained therein is necessary for continuity of a video signal. As such, an error signal D from such frame would be directed to both the first encoder branch 12 and second encoder branch 14 for encoding onto the priority base layer and the error resilience layer to ensure that the missing frame is received by a receiver. In other words, by encoding the missing information twice (once on a base layer and once on an error resilience layer), there is a greater likelihood that the missing information will be received by a receiver.

[0020] For less valuable information contained on a subject frame, the error signal can be directed to one or the other of the first and second coder branches 12, 14, respectively, depending on the priority accorded the information. Finally, if the corrupt or missing frame does not contain information which passes a predetermined minimum threshold criteria of human visual perception, the information need not be re-transmitted, as such information would not be visually perceptible by a viewer to an extent sufficient to warrant retransmission. The operation of the perceptual preprocessor will be explained in further detail below.

[0021] Perceptual preprocessor 50 receives as an input signal the pixel intensity values F_kof a current frame and the pixel intensity values of a prior reconstructed frame {circumflex over (F)}_k−1from a prior frame adjacent-in-time to the current frame. The reconstructed frame {circumflex over (F)}_k−1is obtained from a delayed version of the addition of the inverse transform of error signal {circumflex over (D)}_kand the pixel intensity of the current frame after motion compensation {tilde over (F)}_kwhich is added by adder 42. The perceptual preprocessor 50 outputs an output signal 52 that serves as one of three inputs to the motion estimator 48, the other inputs being F_kand {circumflex over (F)}_k−1, and the motion estimator outputs a motion vector M_k. The motion vector is, in turn, input to motion compensator 46, which also receives the pixel intensity of the prior reconstructed frame {circumflex over (F)}_k−1as an input and generates the pixel intensity signal of the frame after motion compensation F_k. This signal is provided to the subtractor 18 and subtracted from the intensity values of the current frame F_kto generate the error signals D_kand D_k*.

[0022] As explained above, the inventive encoder takes into account the limitations inherent in human visual perception to determine, in the event that a signal received from a transmitter indicates that a transmitted video frame was not received or was received with corrupted data, whether and at what priority a reconstructed version of the current frame should be transmitted. This is accomplished by estimating the visibility of changes which have occurred in each macroblock from the previously reconstructed frame using a model of the human visual system response to these changes. For each pixel in the current frame this model is a function of the current pixel value and the values of a neighborhood of pixels in the previously reconstructed frame. The result of the thresholding function is summed for all pixels in each macroblock and this sum is compared to selected parameters resulting in the assignment of the macroblock to one of a number of macroblock classes according to the degree of visible changes which have occurred for that macroblock in the current frame. The thresholding function is represented by:

[0023] T({circumflex over (F)}_k−1(N(x)),F_k(x))

[0024] wherein

[0025] N(x) is a neighborhood or area of pixels in a frame containing a subject pixel x;

[0026] F_k(x) is the pixel intensity of pixel x of the current frame; and

[0027] {circumflex over (F)}_k−1is the pixel intensity of a reconstructed prior frame.

[0028] Each frame consists of a plurality of macro blocks B. For the i^thmacro block B_i, the δ function on every pixel contained within that block is collected and the subject block is assigned to one of the groups of blocks in the frame F_k. A value for a summation of the pixel values in a particular block B_iis determined by: Δ(B_i)=Σ_xεBδ(X); based on the relationship of this value to selected parameters, the subject block B_iis assigned to one of an existing group of blocks Φ₁,Φ₂,Φ₃, etc. by the following relationship: $B_i\epsilon \{\begin{array}{cc}{\Phi }_1,& \mathrm{if}\Delta \left(B_i\right)\le n_1\\ {\Phi }_2,& \mathrm{if}{n}_1<\Delta \left(B_i\right)\le n_2\\ {\Phi }_3,& \mathrm{if}\Delta \left(B_i\right)>n_2\end{array}$

[0029] The values of n₁and n₂are adjustable. n₁can be chosen to be 0 to provide a perceptually distortion-free macro block. However, it has been observed that video generated with a small, non-zero value for n₁is virtually indistinguishable to a viewer from that generated with n₁=0, while providing for more efficient coding. The value of boundary n₂provides a tradeoff between picture quality and coding rate.

[0030] Based on the parameters n₁and n₂a subject block B_iis assigned to block group Φ₁,Φ₂or Φ₃. As the inventive encoder 10 is a layered encoder, the video signal is portioned, as explained above, into a base layer and an error resilience layer. The encoder is designed so that block groups Φ₂and Φ₃are encoded by first encoder branch 12 into the base layer, whereas block group Φ₁is not contained in that layer. On the other hand, the second encoder branch 14 which produces the error resilience layer only encodes block group Φ₃and skips block groups Φ₁and Φ₂. Thus, for example, if a current macro block B_iis assigned to block group Φ₃, it is deemed to contain important, high-priority and visually perceptible information and is, therefore, encoded onto both the base layer and the error resilience layer by first and second encoder branches 12, 14, respectively. If the subject macro block B_iis associated with macro block group Φ₂, it is deemed of less importance but still visually perceptible, and is only encoded by first encoder branch 12 onto the base layer. In other words, no enhancement data for the information is required. Finally, if the current macro block B_iis found to be associated with macro block group Φ₁, then the information is deemed to be unimportant and is not coded by either first encoder branch 12 or second encoder branch 14.

[0031] The preprocessor 50 may be implemented in any manner well known to those having ordinary shill in the art. For example, a software driven processor may be used to analyze a current frame and compare it to the necessary threshold conditions. In the alternative, discrete components may be utilized for analyzing the current frame and applying the user perception limitation parameters.

[0032] While there have been shown and described and pointed out fundamental novel features of the present invention as applied to a preferred embodiment thereof, it will be understood that various omissions and substitutions and changes in the methods described and in the form and details of the devices illustrated, and in their operation, may be made by those skilled in the art without departing from the spirit of the present invention. For example, it is expressly intended that all combinations of those elements and/or method steps which perform substantially the same function in substantially the same way to achieve the same results are within the scope of the invention. Substitutions of elements from one described embodiment to another are also fully intended and contemplated. It is also to be understood that the drawings are not necessarily drawn to scale but that they are merely conceptual in nature. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.