Sharpness enhancement

[00002] 1. Field of the Invention

[00003] The invention relates to a method and device for sharpness enhancement.

[00004] In scenarios where TV and PC systems merge their functionality, especially in the home environment, high-quality displaying of video images in PC architectures becomes a challenging issue for industries oriented to the consumer market. Since the spatial resolution of synthetic images of the PC outperforms that of natural video scenes, the desire for improved quality in TV images will increase. The subjective attribute sharpness, which is determinant by the human visual system, is one of the most important factors in the perception of image quality. Despite many algorithms for 2-D sharpness enhancement have already been proposed, their effectiveness drops in this context. In fact, in the image processing literature, it is often implicitly assumed that operators that have been designed for the enhancement of 2-D data can be straightforwardly applied to video sequences too. As far as contrast sharpening is concerned, this is far from true: conventional 2-D techniques introduce many small artifacts and non-homogeneities which are perfectly acceptable in a still picture but become very visible and annoying in an image sequence. Moreover, the visibility of a defect strongly depends on the sequence contents, namely, on the amount of details and of motion.

[00005] Moreover, even more attention has to be paid, in the PC environment, to the case of decompressed images like MPEG, AVI, etc. In such cases, indeed, network and bus throughput constraints and limited storage capacity impose the use of compression techniques, often operating at relatively low bit rates, which cause visible blocking effects in the decompressed images. The artificial high frequencies arising along the borders of the blocks themselves are usually amplified by sharpness enhancement algorithms, so that the blocking artifact would be emphasized.

[00006] The task of image enhancement techniques is often to emphasize the details of a scene so as to make them more visible to a human viewer or to aid some machine performance (e.g., object identification); at the same time, they should reduce noise or, at least, avoid its amplification. Algorithms for contrast enhancement are often employed in an interactive fashion with the choice of the algorithm and the setting of its parameters being dependent on the specific application on hand.

[00007] 2. Description of the Related Art

[00008] A large number of approaches have been devised to improve the perceived quality of an image (See Ref. [1]). The histogram equalization, a commonly used method, is based on the mapping of the input gray levels to achieve a nearly uniform output gray level distribution (See Refs. [1], [2]). However, histogram equalization applied to the entire image has the disadvantage of the attenuation of low contrast in the sparsely populated histogram regions. This problem can be alleviated by employing local histogram equalization, which is of high computational complexity.

[00009] Another method, called statistical differencing, generates the enhanced image by dividing each pixel value by its standard deviation estimated inside a specified window centered at the pixel (Ref. [2]). Thus, the amplitude of a pixel in the image is increased when it differs significantly from its neighbors, while it is decreased otherwise. A generalization of the statistical difference methods includes the contributions of the pre-selected first-order and second-order moments.

[00010] An alternate approach to contrast enhancement is based on modifying the magnitude of the Fourier transform of an image while keeping the phase invariant. The transform magnitude is normalized to range between 0 and 1 and raised to a power which is a number between zero and one (Ref. [3]). An inverse transform of the modified spectrum yields the enhanced image. This conceptually simple approach, in some cases, results in unpleasant enhanced images with two types of artifacts: enhanced noise and replication of sharp edges. Moreover, this method is of high computational complexity.

[00011] A simple linear operator that can be used to enhance blurred images is Unsharp Masking (UM) (Ref. [1]). The unsharp masking approach exploits a property of the human visual system called the Mach band effect. This property describes the visual phenomenon that the difference in the perceived brightness of neighboring region depends on the sharpness of the transition (Ref. [1]); as a result, the image sharpness can be improved by introducing more pronounced changes between the image regions. The fundamental idea of UM is to subtract from the input signal a low-pass filtered version of the signal itself. The same effect can, however, be obtained by adding to the input signal a processed version of the signal in which high-frequency components are enhanced; we shall refer to the latter formulation, schematically shown in FIG. 1. FIG. 1 shows that a linearly high-pass filtered (HPF) version of the input signal x(n) is multiplied by a factor λ and thereafter added to the input signal x(n) to form the output signal y(n). The output of the high-pass filter HPF introduce an emphasis which makes the signal variations more sharp. Its effect is exemplified in FIG. 2. In FIG. 2, curve “a” shows the original signal, curve “b” shows a first order derivative of the input signal, curve “c” shows the second order derivative of the input signal, and curve “d” shows the original signal minus k times the second order derivative.

[00012] In one dimension, the UM operation, shown by the block diagram in FIG. 1, can be represented mathematically as:
y(n)=x(n)+λz(n);
where y(n) and x(n) denote the enhanced signal and the original signal, respectively, z(n) means the sharpening component, and λ is a positive constant. A commonly used sharpening component is the one obtained by a linear high-pass filter which can be, for example, the Laplacian operation given by
z(n)=2x(n)−x(n−1)−x(n+1)

[00016] Even though this method is simple to implement, it suffers from two drawbacks that can significantly reduce its benefits. First, the operator introduces an excessive overshoot on sharp details. Second, it also enhances the noise and/or digitization effect. The former problem comes from the fact that the UM method assigns an emphasis to the high frequency components of the input, amplifying a part of the spectrum in which the SNR (signal-to-noise ratio) is usually low. On the other hand, wide and abrupt luminance transitions in the input image can produce overshoot effects; these are put into further evidence by the human visual system through the Mach band effect.

[00017] Several variants of the linear UM technique have been proposed in literature, trying to reduce the noise amplification. A quite trivial approach consists in substituting a bandpass filter for the high-pass one in FIG. 1. This reduces noise effect, but also precludes effective detail enhancement in most images.

[00018] In more sophisticated approaches, Lee and Park (Ref. [4]) suggest using a modified Laplacian in the UM scheme. They propose the order statistic (OS) Laplacian; its output is proportional to the difference between the local average and the median of the pixel in a window. They demonstrate that the resulting filter introduces a much smaller noise amplification than the conventional UM filter, with comparable edge-enhancement characteristics. A different approach is taken in Ref. [5]; in fact they replace the Laplacian filter with a very simple operator based on a generalization of the so-called Teager's algorithm. An example of such an operator is the simple quadratic filter given by
z(n)=x²(n)−x(n−1)x(n+1).

[00020] It can be shown that this operator approximates the behavior of a local mean-weighted high-pass filter, having reduced high-frequency gain in dark images areas. According to Weber's law (Ref. [1]), the sensitivity of the human visual system is higher in dark areas; hence, the proposed filter yields a smaller output in darker areas and, therefore, reduce the perceivable noise. Even though the above quadratic operators do take into account Weber's law, their direct use in unsharp masking may still introduce some visible noise depending on the enhancement factor (1 in FIG. 1) chosen.

[00021] In order to improve the performance of the UM, in Ref. [6], the output of the high-pass filter is multiplied by a control signal obtained from a quadratic edge sensors:
z(n)=[x(n−1)−x(n+1)]²[2x(n)−x(n−1)−x(n+1)]  (1.1)
Its purpose is to amplify only local luminance changes due to true image details. The first factor on the right-hand side of Eq. 1.1 is the edge sensor. It is clear that the output of this factor will be large only if the difference between x(n−1) and x(n+1) is large enough, while the squaring operation prevents interpreting small luminance variations due to noise as true image details. The output of the edge sensor acts as a weight for the signal coming from the second factor in Eq. 1.1, which is a simple linear high-pass filter.

[00024] Another nonlinear filter, the Rational UM technique, has been devised (Ref. [7]): the output of the high-pass filter is multiplied by a rational function of the local input data: $z\left(n\right)=\frac{{\left[x\left(n-1\right)-x\left(n+1\right)\right]}^2}{{k\left[x\left(n-1\right)-x\left(n+1\right)\right]}^4+h}\left[2x\left(n\right)-x\left(n-1\right)-x\left(n+1\right)\right].$

[00025] In this way, details having low and medium sharpness are enhanced; on the other hand, noise amplification is very limited and steep edges, which do not need further emphasis, remain almost unaffected. Under a computational viewpoint, this operator maintains almost the same simplicity as the original linear UM method.

[00026] A similar approach is also proposed in Ref. [8]; the method is similar to the conventional unsharp masking structure, however, the enhancement is allowed only in the direction of maximal change and the enhancement parameter is computed as a rational function like to the one described in Ref. [7]. The operator enhances the true details, limits the overshoot near sharp edges and attenuates noise in at areas. Moreover, it is applied to color image enhancement by using an extension of the gradient to multi-valued signal.

[00027] Finally, in Ref. [9], the unsharp masking technique is extended with an advanced adaptive control that uses the local image content. More precisely, they have concentrated on the following properties for adaptive control of the sharpness enhancement:

• • [00028] local intensity level and related noise visibility;
  • [00029] noise level contained by the signal;
  • [00030] local sharpness of the input signal;
  • [00031] aliasing prevention, where alias results from nonlinear processing such as clipping.

[00032] These four properties of the video signal are analyzed locally by separate units and the amount of the sharpness enhancement depending on this analysis.

[00033] All the proposed contrast enhancement methods have been designed for the enhancement of 2-D data and cannot be applied to video sequences. In fact, they introduce many small artifacts and non-homogeneities which are perfectly acceptable in a still picture but become very visible and annoying in an image sequence. Moreover, conventional 2-D techniques cannot be applied to the enhancement of block-coded image sequences, because they emphasize such artifacts that become very visible. The local spectra and bandwidth of both noise and the signal vary spatially, and the characteristics of the filters need to be locally adapted.

[00034] It is, inter alia, an object of the invention to provide an improved sharpness enhancement. To this end, the invention provides a method and device for sharpness enhancement.

[00035] In a method of sharpness enhancement according to the invention, an input signal is filtered to obtain a filtered signal, the filtered signal is multiplied by a controllable fraction to obtain multiplied signals, and the multiplied signals are added to the input signal. According to the invention, the controllable fraction is generated by a non-linear function.

[00036] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[00049] In this description, we propose a sharpness enhancement technique for video applications, taking into account the fact that the images can be blocked. The enhancement is accomplished by adding a correction signal to the luminance edges in an Unsharp Masking-like way. However, a nonlinear function is used to generate such a correction signal, and both spatial and temporal information is used to deal with significant but thin details and with blocking artifacts; at the same time, a temporal component of the operator reduces the noise amplitude.

[00050] This description is organized as follows. In Sections 2 and 3, a detailed description of the new method for video and block-coded image sequences is presented. In Section 4, an algorithm is described for the automatic discrimination between video and block-coded image, and an operator is proposed that can be applied for both type of sequences. In Section 5, conclusions are given.

[00051] Basically, the enhancement is accomplished by adding overshoot to luminance edges in an Unsharp Masking-like way. However, the optimal amount of overshoot added for a high image quality depends on the local image statistics. Controls are introduced, to improve the performances and to adapt it to moving sequences with different characteristics. Both spatial and temporal information is also used to enhance details, avoiding blocking artifacts and noise.

[heading-00052] 2. 3-D Unsharp Masking Technique for Video Image Sequences

[00053] The aim of a contrast enhancement technique is often to emphasize the details of a scene so as to make them more visible to a human viewer. Basically, the enhancement is accomplished by increasing the steepness of the significant luminance transitions, possibly also adding an overshoot to the edges around objects in the image.

[00054] In this section, an Unsharp Masking-based approach is presented for edge enhancing for TV application. The proposed scheme enhances the true details, limits the overshoot near sharp edges, and attenuates the temporal artifacts. More precisely, to try and cope with the problems of linear unsharp masking (noise sensitivity and excessive overshoot on sharp details), this technique has been extended with an adaptive control that uses the local spatio-temporal image content.

[00055] The block diagram of the proposed algorithm is depicted in FIG. 3. Let s(n, m, t+1) be the input signal; the enhanced signal u(n, m, t) results from the contribution of three terms:
u(n,m,t)=s(n,m,t)+λu_s(n,m,t)−u_t(n,m,t)  (2.1)
where u_s(n, m, t) is the output signal of a spatial non-linear high-pass filter SHPF that introduces an emphasis which makes the signal variations more sharp, without incurring significant noise amplification and temporal artifacts. u_t(n, m, t) is the output of a temporal high-pass filter THPF operating on three frames s(n, m, t+1), s(n, m, t), s(n, m, t−1), that suppresses noise meanwhile preserving the sharpness of moving objects. Delay lines Z⁻¹ are present to generate s(n, m, t) and s(n, m, t−1) from s(n, m, t+1). Multiplier M1 multiplies the output signal u_s(n, m, t) of a spatial non-linear high-pass filter SHPF by the factor λ. Adder A1 adds the output signal of the multiplier M1 to the once-delayed input signal s(n, m, t). The output signal u(n, m, t) is displayed on a display device D.

[00058] The effect of the spatial and the temporal control functions on the image quality is explained in Sections 2.1 and 2.1.2, respectively.

[heading-00059] 2.1 Spatial Operator

[00060] The proposed algorithm is formed by a set of directional high-pass filters, followed by nonlinear correlation filters. In particular, we consider the separate effects of the control signal along the horizontal and the vertical direction: this choice, apart from being simple, is also justified by the fact that the eye is more sensitive to the lines and edges having these orientations (Ref. [1]).

[00061] The block diagram of the spatial high-pass filter SHPF is depicted in FIG. 4, and its expression is:
u_s(n,m,t)=z_x(n,m,t)c_x(n,m,t)+z_y(n,m,t)c_y(n,m,t)  (2:2)
where
z_x(n,m,t)=2s(n,m,t)−s(n,m−1, t)−s(n,m+1,t)  (2.3)
z_y(n,m,t)=2s(n,m,t)−s(n−1,m,t)−s(n+1,m,t)
respectively, are the outputs of respective Laplacian filters (horizontal and vertical linear high-pass filters HLHPF and VLHPF) applied horizontally and vertically to the input image. Their amplitude response is a monotone function of the frequency and is used to introduce the required emphasis on the signal variation.

[00067] In order to improve the performance of the high-pass filter we need to condition its operation so as to emphasize only the medium-contrast details. To achieve this purpose, z_x and z_y are multiplied by the control functions c_x and c_y that depend on the local gradient of the data. The control functions c_x and c_y are obtained by respective horizontal and vertical control functions blocks HCF and VCF that each receive the input signal s(n, m, t) of the spatial high-pass filter SHPF, z_x and z_y. Multiplier M2 multiplies the output signal z_x(n, m, t) of the horizontal linear high-pass filter HLHP by the output c_x(n, m, t) of the horizontal control function HCF. Multiplier M3 multiplies the output signal z_y(n, m, t) of the vertical linear high-pass filter VLHP by the output c_y(n, m, t) of the vertical control function VCF. Adder A2 sums the outputs of the multipliers M2, M3 to obtain the output signal u_s(n, m, t) of the spatial high-pass filter SHPF.

[00068] For the sake of simplicity, the operation of the method is explained referring only to the x-direction.

[heading-00069] 2.1.1 Horizontal Control Function

[00070] The control function is a nonlinear control that uses the local image content to avoid noise amplification, overshoot near sharp edges and temporal artifacts. To better understand its behavior, the output of the control function is shown in FIG. 5, which shows the part of the block diagram enclosed in the dashed box in FIG. 4.

[00071] The control function consists of three major functional blocks:

• • [00072] a non-linear rational controlling function RF for the sharpening term;
  • [00073] a specialized control TLE for the enhancement of thin lines;
  • [00074] an IIR low-pass filtering IIR-LPF of the controlling function.

[00075] In the following, the effect of these controls on the image quality will be briefly explained.

[heading-00076] Rational Function RF

[00077] The controlling function used is a rational function: $\begin{array}{cc}{\stackrel{\_}{c}}_x\left(n,m,t\right)=\{\begin{array}{cc}1& \mathrm{if}{d}_1\le d_x\le d_2\\ \frac{\left(d_1^2+d_2^2\right)d_x^2\left(n,m,t\right)}{d_x^4\left(n,m,t\right)+d_1^2{d}_2^2}& \mathrm{if}{d}_x<d_1,d_x>d_2\end{array}& \left(2.4\right)\end{array}$

[00078] The value this function assumes depends on an edge sensor:
d_x(n,m,t)=s(n,m−1,t)−s(n,m+1,t)  (2.5)

[00080] It is a band-pass filter that discriminates between signal and noise. To this purpose, two implicit hypotheses are made on the significant image details: first, that they are represented by local gradient values which are larger than those introduced by noise; second, that they contribute to the data spectrum mainly in its mid-frequency range. Therefore, this function tends to privilege high-gradient areas and it is less sensitive to slow signal variations. Moreover, this mechanism makes the proposed operator less sensitive to the Gaussian-distributed noise that is always present in the data.

[00081] To better understand its behavior, FIG. 6 shows the non-linear control term {overscore (c)}_x as a function of d_x for a specific choice of d₁ and d₂. The main characteristics of the rational function {overscore (c)}_x are:

• [00082] 1. {overscore (c)}_x=1 for d₁≦d_x≦d₂: this characteristic makes the operator able to emphasize details that are represented by the low and medium amplitude luminance transitions;
• [00083] 2. {overscore (c)}_x→0 for d_x→0: in homogeneous areas, where the noise is usually more visible, impulsive noise yields small value of d_x (smaller than d₁), and then a small value of the added high-pass signal;
• [00084] 3. {overscore (c)}_x→0 for d_x→∞: in this way, undesired overshoots near sharp edges are avoided. In fact, sharp edges will yield high values of d_x (larger than d₂) and {overscore (c)}_x will be small.
  The best balance among these effects can be achieved by setting the right positions for d₁ and d₂. The intensity of the enhancement can be obviously adjusted by changing the value of 1.
  Controlling the Enhancement of Thin Lines TLE

[00087] Even though experimental results support the validity of the rational function {overscore (c)}_x, it cannot be used to emphasize thin lines. In fact, to reduce the noise sensitivity, the edge sensor is a bandpass filter and is not able to detect lines having a thickness of one pixel. Therefore, thin lines are erroneously interpreted as noise and this drawback causes some artifacts.

[00088] For example, when noisy pixels are adjacent to the line itself, the output of the sensor varies along the line. Hence, even for low-amplitude noise, the contribution of the correction term c_xz_x may strongly vary because of the high slope of the control function near the origin combined with the very high value of the Laplacian. Temporal changes of the noise samples give rise to a non-uniform enhancement, appearing as an annoying flicker effect close to the line.

[00089] Thin lines are emphasized in a non-homogeneous way. Moreover, the enhancement in the first frame (left) is different from that one in the second frame (right), and introduce a temporal artifact which is very visible and annoying in an image sequence. Many small artifacts and non-homogeneities are introduced also near lines having slope close to, but different from, 0 or 90 degrees. In fact, in digital images, these lines unavoidably show jagged borders (staircase effect). In particular, one-pixel-wide lines are formed by segments whose width alternates between one and two. The rational function, due to its insensitivity to very thin lines, enhances these lines in a non-homogeneous way, and the staircase effect is therefore emphasized.

[00090] In order to overcome such problems, a simple but effective method is to include a proper control to detect thin lines. To this purpose, the outputs z_x and z_y of the Laplacian filters themselves are exploited. In correspondence of a very thin line, the following three cases are possible:
horizontal line: |z_x|≅0 and |z_y|>>0  (2.6)
vertical line: |z_x|>>0 and |z_y|≅0  (2.7)
diagonal line: |z_x|>>0 and |z_y|>>0  (2.8)

[00094] From these relationships, it is straightforward to discriminate between a pixel belonging to a line and a noisy pixel, simply by setting a threshold S_n which can be derived from statistical considerations, as it will be shown in the following. For example, when
|z_x|≧S_n and |z_y|≦S_n,
a vertical line is detected and it is only needed to limit the overshoot when such a line is already well visible. Therefore, once a thin line is detected by thin line detector TLD, the rational function {overscore (c)}_x is replaced by following overpeaking controlling function OCF (see FIG. 5) by means of a switch S that is controlled by the output signal of thin line detector TLD: ${\hat{c}}_x=\{\begin{array}{cc}1& \mathrm{if}z_x\langle d_2\\ \frac{d_0}{z_x}& \mathrm{if}z_x\ge d_2\end{array}$
which simply limits the correction term z_x when its value is larger than d₂.

[00098] The extension to cases (2.7) and (2.8) is similar. In fact, when the condition (2.7) is true, a horizontal line is detected and the vertical rational function ĉ_x is replaced by ĉ_y. Finally, if condition (2.8) is verified, both {overscore (c)}_x and {overscore (c)}_y are replaced by ĉ_x and ĉ_y, respectively.

[00099] S_n is chosen to discriminate between a pixel belonging to a line and a noisy pixel. The value of S_n cannot be too small because many noise pixels would be erroneously interpreted as thin lines; on the other hand, S_n cannot be too high, otherwise the thin line would not be individualized.

[00100] A good solution is to determine the relationship between S_n and the noise variance present in the image. For this purpose, the probability of erroneously labeling a noise pixel as important detail is estimated. Supposing a sequence of images degraded with a zero mean Gaussian noise having variance σ², the probability error is estimated only when noise is present.

[00101] Because {s(n+k,m)|k=−1,0,1} and {s(n,m+k)|k=−1,0,1} are uncorrelated Gaussian random variables with mean zero and variance σ²(N(0,σ²)), it is easy to show that z and z are Gaussian random variables, with mean zero and variance σ_zx²=σ_zy²=6σ².

[00102] Since z_x and z_y depend on the value of s(n, m) (Eq.2.3), they are v.a. correlated. Hence, the joint probability density function is: $\{\begin{array}{c}{f}_{\mathrm{Zr},\mathrm{Zy}}\left(z_x,z_y\right)=\frac{1}{2{\mathrm{\pi \sigma }}_{\mathrm{zy}}\sqrt{1-{\rho }^2}}{e}^{-Q}\\ Q=\frac{1}{2\left(1-{\rho }^2\right)}[{\left(\frac{z_x}{{\sigma }_{\mathrm{zx}}}\right)}^2+\left(\frac{z_y}{{{\sigma }_{\mathrm{zy}})}^2}-2\rho \frac{z_x}{{\sigma }_{\mathrm{zx}}}\frac{z_y}{{\sigma }_{\mathrm{zy}}}\right]\end{array}\mathrm{where}\rho =\frac{\mathrm{COV}\lfloor Z_x{Z}_y\rfloor }{{\sigma }_{\mathrm{zx}}{\sigma }_{\mathrm{zy}}}$
Since Z_x,Z_y≈(0,6σ) $\rho =\frac{E\lfloor Z_x{Z}_y\rfloor -E\left[Z_x\right]E\lfloor Z_y\rfloor }{6{\sigma }^2}=\frac{4E\left[{s\left(n,m\right)}^2\right]}{6{\sigma }^2}=\frac{4{\sigma }^2}{6{\sigma }^2}=\frac{2}{3},$
and the probability of error is: $\begin{array}{c}{P}_e=1-P(Z_x\langle S_{n}eZ_y\langle S_n)\\ =1-\frac{1}{4\sqrt{5}{\mathrm{\pi \sigma }}^2}\int \frac{S_n}{-S_n}{e}^{-\frac{3}{20{\sigma }^2}\left[z\frac{2}{x}+z\frac{2}{y}-\frac{4}{3}{z}_x{z}_y\right]}{d}_{\mathrm{zx}}{d}_{\mathrm{zy}}\end{array}$
By selecting S_n=3σ_zx=3√{square root over (6)}σ, we obtain $P_e\cong 1-0.995=0.005=\frac{1}{200}$

[00106] This means that in a 720×576 image, for example, more than 2000 noise points are amplified. However, most of these errors will not be perceived thanks to the masking effect of the details and of motion. Experimental results have shown that S_n=3σ_zx=3√{square root over (6)}σ is a satisfying threshold value.

[00107] It should be noted that this method can be applied only if the noise variance σ² is known, but it is not easy to estimate. However, the annoying flicker effects close to the line and staircase effect are visible above all in those parts of a sequence that are almost stationary. In this case, is easy to estimate the noise variance (Ref. [10]). The simplest method to estimate σ² is frame averaging, which is very effective when the image does not change from frame to frame but the degradation does. The simplest and most common form of frame averaging is estimating an image f(n, m) from a sequence of N degraded image frame g_i(n, m) for 1≦i≦N.

[00108] Suppose a sequence of degraded images g_i(n, m) can be expressed as
g_i(n,m)=f(n,m)+η_i(n,m), 1≦i≦N  (2.9)
where n_i(n, m) is zero-mean stationary white Gaussian noise with variance of σ² and n_i(n, m) is independent from n_i(n, m) for i≠j. If we assume that f(n, m) is non-random, the maximum likelihood (ML) estimate of f(n, m) that maximize
P_{gi(n,m)|f(n,m)}(g_i¹(n,m)|f¹(n,m))
is given by $\begin{array}{cc}\hat{f}\left(n,m\right)=\frac{1}{N}\sum _{i=1}^N{g}_i\left(n,m\right)& \left(2.10\right)\end{array}$
From (2.9) and (2.10), $\begin{array}{cc}\hat{f}\left(n,m\right)=f\left(n,m\right)+\frac{1}{N}\sum _{i=1}^N{\eta }_i\left(n,m\right)& \left(2.11\right)\end{array}$
From (2.11), the degradation in the frame-averaged image remains a zero-mean stationary white Gaussian noise with variance of σ²/N, which represents a reduction of noise variance by a factor of N in comparison with η_i(n,m)

[00115] This suggests estimating η_i from g_i-{circumflex over (f)}. The value of N must be high in order to obtain an image {circumflex over (f)}(n,m) with a small noise variance. It cannot be too much high, because the estimate of would be wrong on sequences that are almost stationary.

[00116] Supposing to apply the enhancement operator on sequences with a maximum value of σ²=30, we choose N=5. The estimate of σ² will be wrong in conditions of fast motion, but in this case the artifacts are less visible.

[heading-00117] IIR Low-Pass Filtering

[00118] From FIG. 5, it should be noted that the final value of the horizontal control function c_x is obtained by a spatial low-pass filtering, for the reasons that will be explained in the following.
Formally, its expression is: $\begin{array}{cc}{c}_x\left(n,m,t\right)=\frac{1}{2}\left[c_x\left(n-1,m,t\right)+c_x\left(n,m,t\right)\right]& \left(2.12\right)\end{array}$
where c_x′ coincides with ĉ_x or {overscore (c)}_x, depending on whether a thin line is detected or not.

[00121] A non-uniform enhancement effect is present in highly detailed areas with complex and fine contours, still due to small changes in the output of the sensor amplified by the slope of the rational function, together with the high value of the Laplacian in such areas. This effect is usually not perceived in 2-D enhancement, thanks to the masking effect of the details themselves; however, it generates a flickering which is very annoying in those parts of a sequence which are almost stationary. The sharpening is effective, but the edges are amplified in a non-homogeneous way.

[00122] To make the operator more robust against the appearance of false details, c_x is IIR filtered as is shown in Eqs. (2.12): the values assumed by on c_x neighboring pixels are computed by averaging the current value with the previous one along the direction which is orthogonal to the control action. In such a way, it is possible to reduce the discontinuity of these values mainly along the direction of the local borders, avoiding the generation of false details and the related visible artifacts.

[00123] It should be noted that an order of the low-pass filter more than two may generate artifacts because, in this case, pixels could be mediated that don't belong to the same object.

[heading-00124] 2.1.2 Temporal Operator

[00125] In video sequences, temporal correlation can be exploited to further reduce the noise. To suppress noise while preserving the sharpness of moving objects, we propose a temporal filter working on three frames. The basic assumption is that when a noisy pixel occurs the probability it occurs also in the previous or the next frame in the same spatial position is very small.

[00126] Let us define the value of s in three subsequent frames as s(n,m,t-1), s(n,m,t) and s(n,m,t+1). When the following conditions
s(n,m,t−1)≅s(n,m,t+1)  (2.13)
s(n,m,t)−s(n,m,t−1)≠0  (2.14)
are both verified, probably a noisy pixel is present in a locally static area. The proposed temporal filter is able to evaluate the conditions (2.13) and (2.14) and at the same time to perform low-pass filtering. Such a filtering is achieved by subtracting, from the input signal, the following high-pass signal: $\begin{array}{cc}{u}_t\left(n,m,t\right)=\frac{1}{4}{z}_t\left(n,m,t\right)c_t\left(n,m,t\right),& \left(2.15\right)\end{array}$
where z_t a temporal Laplacian filter:
z_t(n,m,t)=2s(n,m,t)−s(n,m,t−1)−s(n,m,t+1),  (2.16)
and c_t controlling function: $\begin{array}{cc}{c}_t\left(n,m,t\right)=\{\begin{array}{cc}\frac{d_0^2-d_t^2\left(n,m,t\right)}{d_0^2}& \mathrm{if}d_t(·)\le d_0\\ 0& \mathrm{if}d_t^2(·)\rangle d_0\end{array}& \left(2.17\right)\end{array}$
Its shape is shown in FIG. 7; it can be seen that c_t≧0 for |d_t|≦d₀; therefore, the temporal low-pass filter attenuates signal variations which are smaller than a certain threshold. This allows us to smooth selectively the signal variations which are attributed to noise corruption.

[00134] The temporal sensor d_t is chosen to discriminate between a noisy pixel and a detail. It must be a simple function which is able to evaluate the conditions (2.13) and (2.14). To this purpose, the bandpass filter d_t=s(n,m,t−1)−s(n,m,t+1) gives acceptable results with an extremely low computational complexity. As an example, let us consider the case of s(n, m, t) corrupted by the noise, no motion, and presence of an uniform area in the masks, so that the spatial part of the filter is not allowed to operate: the values of d_x, d_y and d_t will be small, so that c_x≅c_y≅0 and c_t=1.
Hence, from Eq. (2.1), the output of the filter becomes: $u\left(n,m,t\right)\cong \frac{s\left(n,m,t\right)+s\left(n,m,t-1\right)+s\left(n,m,t+1\right)}{4}$
which corresponds to a linear low-pass filter that reduces the noise. On the contrary, in correspondence of a local movement d_t>>0, so that c_t≅0 and no filtering is performed.

[00137] It should be noted, however, that this version of the filter may generate artifacts in particular conditions of fast motions. Let us consider, for example, the case of an object moving quickly on a uniform background: if the object is present in the filter mask at time t, but not at times t-1 and t+1, the value of s(n, m, t) is considered as impulse noise by the temporal part of the filter, and the object may be deleted. In an example, a disk with white and black slices rotates with such a frequency to satisfy conditions (2.13) and (2.14). Applying the temporal operator with the bandpass sensor, some artifacts are introduced, which are visible as light and dark thin lines. If a larger computational load can be accepted, this type of artifact is eliminated by resorting to a Range Filter (Ref. [11]). Experimental results show that the noise reduction is the same with both operators. Therefore, the expression of d_t becomes:
d_t(n,m,t)=d_max−d_min
d_max=max{s(n,m,t−k)|k=−1,0,1}
d_min=min{s(n,m,t−k)|k=−1,0,1}

[00141] Simulations show that the processed image presents sharper edges which make it more pleasant to the human eye. Moreover, the amount of the noise in the image and the temporal artifacts are clearly reduced.

[heading-00142] 2.2 Computational Complexity Analysis

[00143] Now, we analyze the computational complexity of our proposed algorithm. From FIG. 4, it can be seen that the operator is separable, therefore, we consider the separate effect of the spatial and temporal operator. The operator needs 19 multiplications and 13 sums. However, it has to be noted that the values of the non-linear functions c_x, c_y and c_t, depend on the value of the edge sensors (d_x, d_y and d_t) which assume only 255 different values. Hence, the rational functions can be calculated by using a look-up table; in this case, the number of multiplications per pixel becomes 7 instead of 19. Hence, under a computational viewpoint, it must be stressed that the proposed solution gives acceptable results with an extremely low computational complexity.

[heading-00144] 3. Unsharp Masking Technique for Block-Coded Image Sequences

[heading-00145] 3.1 Introduction

[00146] Most video data compression standards use a block-based motion estimate and block-based discrete cosine transform (DCT). In particular, they use an N×N pixel-block DCT for information packing into a few transform coefficients. This block-based DCT scheme takes advantage of the local spatial correlation property of images. However, this type of coding induces the well-known blocking artifacts, corner outliers, and ringing noise when the image is highly compressed.

[00147] Conventional 2-D techniques for contrast sharpening cannot be applied to the enhancement of block-coded image sequences, because they emphasize such artifacts that become very visible and annoying in an image sequence. The local spectra and bandwidth of both the noise and the signal vary spatially, and the characteristics of the filters need to be locally adapted.

[00148] For this purpose, in this section, we first present a comprehensive characterization of the coding artifacts which are introduced into reconstructed video sequences through the use of the video coding algorithm and then propose a simple modification to the algorithm described in the previous chapter in order to emphasize the details and reduces the visibility of the most typical coding artifacts.

[heading-00149] 3.2. Video Coding Distortions

[00150] The following discussion will be limited to descriptions of the artifacts' visual manifestations, causes and relationships. We also describe the spatial and temporal characteristics of video sequences that are susceptible to each artifact and in which the artifacts are visually prominent.

[heading-00151] 3.2.1 Blocking Effect

[00152] The blocking effect is an annoying side result of techniques in which blocks of pixel are treated as single entities and coded separately. In brief, due to the isolated nature in which individual blocks are coded, the level and characteristic of the coding error introduced into a block may differ from one block to another. So, a smooth change of luminance across a border can result in a step in the decoded image.

[00153] When we consider a smoothing operation to remove blocking artifacts in video coding, we find three interesting observations. First, the human visual system is more sensitive to blocking artifacts in at regions than in complex regions. Therefore, a strong smoothing filter is required on those regions. In complex regions, however, smoothing of few pixel around the block boundary is enough to achieve a desired de-blocking effect. Second, smoothing operations tend to introduce more undesirable blur in complex regions that in at regions. Hence, adaptive smoothing to preserve image details is desirable in complex regions. Third, because of motion compensation, blocking artifacts are propagated, and the propagated artifacts are more visible in at regions that in complex ones. Therefore, smoothing in these regions must cover the inside of a block as well as block boundaries.

[heading-00154] 3.2.2 Blurring

[00155] Blurring manifests as a loss of spatial detail and a reduction in sharpness of edges in moderate to high spatial activity regions of frames, such as in roughly textured areas or around scene object edges.

[00156] For intra-frame coded macroblocks, blurring is directly related to the suppressions of the higher-order DCT coefficients through coarse quantization, leaving only the lower-order coefficients to represent the contents of a block; therefore, blurring can be directly associated with low-pass filtering.

[00157] For predictive coded macroblocks, blurring is mainly a consequence of the use of a predicted macroblock with a lack of spatial detail. However, blurring can also be induced in bi-directionally predicted macroblocks, where the interpolation of the backward and forward predictions results in an averaging of the contents of the final bi-directional prediction. In both these cases, the blurred details are supplemented by the prediction error that supplies some higher-frequency information to the reconstruction, thereby reducing the blurring effect.

[00158] From the observations above, an adaptive enhancement filter is required in complex regions in order to emphasize the details and reduce the visibility of the coding artifacts.

[heading-00159] 3.2.3 Staircase Effect

[00160] The DCT basic images are not attuned to the representation of diagonal edges. Consequently, more of the higher activity basis images are required to satisfactory represent diagonal edges. Due to the typically low magnitude of the higher-order basic images, coarse quantization results in their truncation to zero. The contribution originally made by the higher-order basic images in forming the diagonal edge is diminished, resulting in a reconstruction exhibiting only the characteristics of the lower frequency basic images, which are generally either horizontally on vertically oriented. So, for a block containing a diagonal edge angled towards the horizontal, coarse quantization will result in a reconstruction with horizontal orientation, and vice versa for blocks angled towards the vertical. Note that the step-wise discontinuities occur at block boundaries.

[heading-00161] 3.2.4 Ringing

[00162] The ringing effect is related to the high-frequency distortions introduced by the DCT coefficient quantization. In fact, the representation of a block can be considered as a carefully balanced aggregate of each of the DCT basic images. Therefore, quantization of an individual coefficient results in the generation of an error in the contribution made by the corresponding basis image to a reconstructed block. Since the higher-frequency basic images play a significant role in the representation of an edge, the quantized reconstruction of the block will include high-frequency irregularities. This effect is most evident along high contrast edges in areas of generally smooth texture in the reconstruction.

[heading-00163] 3.2.5 False Edges

[00164] Because of motion compensation, blocking artifacts are propagated; hence, false edges are a consequence of the transfer on the block-edge discontinuities formed by the blocking effect into the current frame. As with the blocking effect, false edges are mainly visible in smooth areas of predictive coded frames. The prediction error in such areas would typically be minimal, or quantized to zero, and, therefore, the false edges would not be masked.

[heading-00165] 3.2.6 Motion-Compensation Mismatch

[00166] Motion-compensation (MC) mismatch can be generally defined as the situation in which a satisfactory prediction cannot be found for a particular macroblock, resulting in a prediction whose spatial characteristics is mismatched with those of the current macroblock. The consequence of such a situation is a high level of prediction error which result in a reconstruction with highly visible distortions that is, typically, a high-frequency noise-like effect.

[heading-00167] 3.3 Proposed Algorithm

[00168] Many de-blocking schemes have been proposed in still image coding, such as JPEG, under the assumption that blocking artifacts are always located at block boundaries. In video coding, however, the blocking artifacts of the previous frame can be propagated to the current frame, and can be located at any position in a block due to motion-compensated prediction (Sec. 3.2.5). Therefore, simple block boundary smoothing is not good enough to remove blocking artifacts appearing in video coding. Moreover, a low-pass effect is also needed inside the macroblock to reduce ringing and motion compensation mismatch effects. Iterative methods based on projection on a convex set (Refs. [12], [13]) may be a candidate algorithm. However, it is not adequate for real-time video coding due to its complexity.

[00169] In this section, we present an unsharp masking-based approach for noise smoothing and edge enhancing. The block diagram is similar to the one shown in FIG. 3 and FIG. 4, but it is necessary to modify the rational functions (Eq. 2.4) in order to enhance true details and reduce coding artifacts.

[00170] The new rational function {overscore (c)}_xb and {overscore (c)}_yb) has to be simple and must obey several restrictions, in order to enhances true details and reduce noise:

• [00171] 1. It has to attenuate signal variations which are smaller that a certain threshold h. This will allow us to smooth selectively the signal variations which are attributed to coding artifacts;
• [00172] 2. The mid-range details which are the signal transitions having magnitude larger than h must be enhanced;
• [00173] 3. To avoid the excessive overshoot over sharp edges: lim_Δ→∞{overscore (c)}_xb(Δ)=0

[00174] A simple function verifying all these conditions can be formulated as a rational function that is the ratio of two polynomial functions: ${\stackrel{\_}{c}}_{\mathrm{xb}}\left(n,m,t\right)=\{\begin{array}{cc}\frac{{{\hat{d}}_x\left(n,m,t\right)}^2-h^{\prime }}{k{{\hat{d}}_x\left(m.,.t\right)}^4+4\lambda h^{\prime }}& \mathrm{if}\frac{{{\hat{d}}_x\left(n,m,t\right)}^2-h^{\prime }}{k{{\hat{d}}_x\left(n,m,t\right)}^4+4\lambda h^{\prime }}\le 1\\ 1& \mathrm{otherwise}\end{array}{\stackrel{\_}{c}}_{\mathrm{yb}}\left(n,m,t\right)=\{\begin{array}{cc}\frac{{{\hat{d}}_y\left(n,m,t\right)}^2-h^{\prime }}{k{{\hat{d}}_y\left(m.,.t\right)}^4+4\lambda h^{\prime }}& \mathrm{if}\frac{{{\hat{d}}_y\left(n,m,t\right)}^2-h^{\prime }}{k{{\hat{d}}_y\left(n,m,t\right)}^4+4\lambda h^{\prime }}\le 1\\ 1& \mathrm{otherwise}\end{array}$
where the edge sensors {circumflex over (d)}_x and {circumflex over (d)}_y are as: $\begin{array}{cc}{\hat{d}}_x\left(n,m,t\right)=d_x\left(n,m,t\right)\frac{{\sigma }^2}{{\sigma }^2+{\sigma }_{\mathrm{th}}^2}{\hat{d}}_y\left(n,m,t\right)=d_y\left(n,m,t\right)\frac{{\sigma }^2}{{\sigma }^2+{\sigma }_{\mathrm{th}}^2}& \left(3.3\right)\end{array}$
Variance-based measurements proved to be a reliable and simple estimator of the degree of local activity (Ref. [14]) and have been used for this purpose also in the present work. Hence, σ² is the variance of the pixels belonging to a 3×3 window centered on the current pixel, and σ_th² is a value to be fixed based on the average variance of the image. The parameters k and h′ are proper positive factors that are chosen to achieve the best trade-off between enhancement of details having low and medium sharpness, reduction of noise and overshoot near already sharp edges. The relations between the parameters d₁, d₂, k, h $k=\frac{1}{d_1^2+d_2^2},h^{\prime }=\frac{d_1^2{d}_2^2}{\left(d_1^2+d_2^2\right)\left(4\lambda +1\right)},$
while with the parameter λ we can adjust the enhancement intensity.

[00178] FIG. 8 shows the shape of the horizontal rational control function. It can be seen that {overscore (c)}_xb≦0 for d≦h′; therefore, it attenuates signal variations that are smaller than a certain threshold. To better understand the behavior of the filter, we describe the main characteristics in at and complex regions.

[heading-00179] 1. In uniform areas, σ² is negligible with respect to σ_th², so that {circumflex over (d)}_x, and {circumflex over (d)}_y tend to zero and {overscore (c)}_xb≅{overscore (c)}_yb≦0. Hence, the output of the spatial filter s+λ└z_x{overscore (c)}_xb+z_yc_yb┘ is a low-pass signal and the coding artifacts are reduced.

[00180] In particular, supposing that a vertical border occurs between uniform areas (blocking effect; see FIG. 9); σ² will be small and {overscore (c)}_xb≅−{fraction (1/4λ)}. Since we consider a vertical block boundary VB, the previous value of the control function c_x(n−1,m) is similar to {overscore (c)}_xb(n,m). Then, the output of the IIR filter coincides with {overscore (c)}_xb(n,m): $c_x\left(n.m\right)=\frac{1}{2}\left[c_x\left(n-1,m\right)+{\stackrel{\_}{c}}_{x_b}\left(n,m\right)\right]\cong {\stackrel{\_}{c}}_{x_b}\left(n,m\right)\cong -\frac{1}{4\lambda }.$

[00181] Moreover, considering that on such a border z_y will be small too, the output of the filter only depends on the horizontal control function, and we obtain a horizontal linear low-pass filter: $\begin{array}{c}{u}_s\left(n,m\right)\cong s(n,m(-\frac{1}{4}{z}_x\left(n,m\right)\\ \cong \frac{2s\left(n,m\right)+s\left(n-1,m\right)+s\left(n+1,m\right)}{4}\end{array}$
that reduces the discontinuities between the boundaries of adjacent blocks.

[00183] Our operator also reduces ringing noise. Ringing noise along a vertical edge is like that considered above; in fact, the value of z_y is small and the output of the filter only depends on the horizontal control function. So, we obtain a horizontal low-pass filter.

[00184] As regards ringing noise around a diagonal edge, the value of z_x and z_y are not zero and the output of the filter depends on both the horizontal and vertical control functions:
u_s(n,m)=s(n, m)+λ[z_x(n, m)c_x(n, m)+z_y(n,m)c_y(n, m)]
Since σ² is negligible with respect to σ_th², the values of the rational functions are: ${\stackrel{\_}{c}}_{x_b}\cong {\stackrel{\_}{c}}_{y_b}\cong =\frac{1}{4\lambda }$

[00187] It has to be noted that the previous values of the conrol functions are greater than the current value:
c_x(n−1,m)>{overscore (c)}_xb(n,m) and c_y(n,m−1)>{overscore (c)}_yb(n,m)
because they are closer to the edge. This means that the output of the IIR low-pass filter is greater than −{fraction (1/4λ)}. Experimental results show that
c_x(n,m)≅½[c_x(n−1,m)+{overscore (c)}_xb(n,m)]≅−{fraction (1/6λ)}
c_y(n,m)≅½[c_y(n,m−1)+{overscore (c)}_yb(n,m)]≅−{fraction (1/6λ)}
In this way, we obtain a 2-D linear low-pass filter; in fact, the expression of u_a(n, m) is: $\begin{array}{c}{u}_s\left(n,m\right)\cong \frac{2s\left(n,m\right)+s\left(n+1,m\right)+s\left(n-1,m\right)+s\left(n,m+1\right)}{6}+\\ \frac{s\left(n-m-1\right)}{6}\end{array}$
2. In detail areas, σ²>>0, and then {circumflex over (d)}_x≅d_x,{circumflex over (d)}_y≅d_y. Therefore, the mid-range details represented by the signal transitions having magnitude larger than h′ will be enhanced. On the other hand, when d_x≅0(or d_y≅0) and σ²>>0, the area is uniform in the horizontal (vertical) direction, so that the low-pass filtering is applied.

[00194] If a smaller computational load is expected, we can calculate the variance of five pixels instead of nine pixels. So, if we define σ² like the variance of the following five pixels: s(n, m); s(n−1, m); s(n+1, m); s(n, m−1); s(n, m+1), we can use σ_c² in Eq.3.3.

[heading-00195] 3.4 Conclusions

[00196] From the observations above, the filter for at regions provides a smoothing effect inside a block as well as on block boundaries because in video coding the blocking and ringing artifacts can be located at any position in a block due to motion-compensated prediction.

[00197] On the other hand, the behavior of the filter is a nonlinear high-pass one in complex areas in order to enhances true details. The variance detector that regulates the filter behavior is embedded in the filter itself, which allows a compact expression of the operator.

[00198] It has to be noted that the operator for the enhancement in block-coded image sequences is similar to the one described in Section 2. This suggests defining an enhancement algorithm for both video and block-coded image sequences. For this purpose, a simple and effective solution is proposed in Section 4.

[heading-00199] 4. Automatic Discrimination Between Video and Block-Coding Image Sequences

[00200] In Section 2, we have described an enhancement technique for video image sequences that enhances the true details, limits the overshoot near sharp edges and attenuates the temporal artifacts. This operator cannot be applied to the enhancement of block-coded image sequences, because it emphasizes such coding artifacts that become very visible and annoying.

[00201] For this task, a simple modification has been introduced when processing block-coded image sequences in order to emphasize the details and reduce the visibility of blocking and ringing effects (see Section 3).

[00202] However, this operator cannot be applied in video sequences, because it introduces significant blurring and does not amplify the low-contrast details.

[00203] In order to define an enhancement technique for both video and coded image sequences, it is important to insert a mechanism in the operator that allows the behavior of the filter to change according to the presence of the coding artifacts. To this purpose, in this chapter we insert a weight factor in the expression of edge sensor that depends on a new distortion measure for blocking and ringing artifacts.

[heading-00204] 4.1 Enhancement Operator for Both Video and Coded Image

[00205] From a comparison of the enhancement techniques described in Sections 2 and 3, it can be seen that they defer for the rational function and the edge sensor. This suggests defining a rational function and/or an edge sensor for both video and block-coded image sequences. A simple and effective solution is the following: $\begin{array}{cc}{\stackrel{\_}{c}}_{x_n}(n,m=\{\begin{array}{cc}\frac{d_x^2\left(n,m\right)-h^{\prime }}{k{d}_x^4\left(n,m\right)+4\lambda h^{\prime }}f& \frac{d_x^2\left(n,m\right)-h^{\prime }}{k{d}_x^4\left(n,m\right)+4\lambda h^{\prime }}\le 1\\ 1& \mathrm{otherwise}\end{array}\mathrm{where}& \left(4.1\right)\\ {\hat{d}}_x\left(n,m\right)=d_x\left(n,m\right)\frac{{\sigma }^2}{{\sigma }^2+\beta {\sigma }_{\mathrm{th}}^2}& \left(4.2\right)\end{array}$
where the parameter β plays an important role in determining the bias of the filter (the criterion for the determination of this weight will be explained in the following). In particular, the value of β change according to the presence of the coding artifacts, i.e., β tends to zero in uncompressed video sequences, while increasing in block-coded sequences. In this way, the behavior of the filter becomes more and more low-pass as coding artifacts increases.

[00207] In video sequences, the constant −h′ in the numerator of c_x, introduces a low-pass effect aiming to reduce noise that is always present in a real image. Moreover, it has to be noted that in this case {circumflex over (d)}_x≃d_x, therefore, the filter does not introduce significant blurring near low-contrast details. As mentioned above, the value of β depends on the presence of coding artifacts; therefore, in the following, we introduce a new distortion measure for blocking and ringing artifacts.

[heading-00208] 4.1.1 Distortion Measure for Blocking and Ringing Artifacts

[00209] In order to define a metric for blockiness, various methods have been proposed in literature (Refs. [15], [16], [17], [18]). All of the techniques described suppose that the blocking artifacts are always located at block boundaries, but in video coding, the blocking artifacts can be located at any position in a block due to motion-compensated prediction. Therefore, these methods are not good enough to measure the presence of blocking artifacts. A simple solution would be to apply these algorithms with different value of N; however, this approach is not adequate for real-time video coding due to its complexity. Moreover, these techniques do not take into account the ringing noise that is very visible and annoying in a image sequences.

[heading-00210] We propose a method to evaluate the presence of both blocking and ringing artifacts that does not need information about the position and size of a block.

[heading-00211] Detection of the Blocking Effect

[00212] Among the most typical artifacts that can be introduced by coding algorithms, the blocking effect is an annoying side result of techniques in which blocks of pixels are treated as single entities and coded separately (see Sec. 3.3.1). In this case, a smooth change of luminance across the border of a block can result in a step in the decoded image if neighboring samples fall into different quantization intervals.

[00213] Then, if the coefficients of two adjacent blocks are coarsely quantized, we expect to see two blocks of low variance and a difference in the intensity slope across the block boundary. On the other hand, this abrupt change in intensity slope across the block boundaries of the original unquantized image is rather unlikely.

[00214] From the above consideration, it is clear that an appropriate method to detect a blocking effect is to find two adjacent blocks with a very small variance value and a different mean value.

[00215] For the sake of simplicity, we consider the horizontal direction. Let s(n, m) be the luminance level of the pixel in position (n, m); we define μ_1x,σ₁²,μ2_xσ₂² as the mean and variance of two 3×3 adjacent windows: $\begin{array}{cc}{\mu }_{1_x}=\frac{\sum _{i=-1}^{i=1}\sum _{j=-1}^{j=1}s\left(n+i,m+j\right)}{9}{\sigma }_{1_x}^2=\frac{\sum _{i=-1}^{i=1}\sum _{j=-1}^{j=1}{\lfloor s\left(n+i,m+j\right)-{\mu }_{1_x}\rfloor }^2}{9}{\mu }_{2_x}=\frac{\sum _{i=-1}^{i=1}\sum _{j=-1}^{j=1}s\left(n+i,m+3+j\right)}{9}& \left(4.4\right)\\ {\sigma }_{2_x}^2=\frac{\sum _{i=-1}^{i=1}\sum _{j=-1}^{j=1}{\left[s\left(n+i,m++3+j\right)-{\mu }_{2_x}\right]}^2}{9}& \left(4.5\right)\end{array}$
When the following conditions: $\begin{array}{cc}\{\begin{array}{c}{\sigma }_{2_x}^2+{\sigma }_{s_x}^2\simeq 0\\ {\mu }_{1_x}-{\mu }_{2_x}>0\end{array}& \left(4.6\right)\end{array}$
are both verified, probably a blocking effect is present in a locally static area.
The extension to the vertical direction is trivial.
Detection of Ringing Noise

[00220] As mentioned in Sec. 3.2.4, the ringing noise is the Gibb's phenomenon due to truncation of the high-frequency coefficients by quantization, and, as such, it is most evident along high contrast edges in areas generally of smooth texture. It appears as a shimmering or rippling outwards from the edge up to the encompassing block's boundary.

[00221] From this consideration, a simple method in order to detect ringing noise is to find two adjacent 3×3 windows with a similar mean value (because the ringing noise has mean zero), but a very different variance value. As done earlier, we consider only the horizontal direction; let be μ₁,σ₁²,μ₂,σ₂², the mean and variance of two adjacent windows (see Eq.4.4). When $\begin{array}{cc}\{\begin{array}{c}{\sigma }_{1_x}^2=0\\ {\sigma }_{2_x}^2>>0\\ {\mu }_{1_x}-{\mu }_{2_x}\simeq 0\end{array}\mathrm{or}\{\begin{array}{c}{\sigma }_{2_x}^2=0\\ {\sigma }_{1_x}^2>>0\\ {\mu }_{1_x}-{\mu }_{2_x}\simeq 0\end{array}& \left(4.7\right)\end{array}$
are verified, probably a ringing noise is detected. The ringing noise is most visible as the bit-rate decrease.
4.1.2 Determination of β Value

[00224] In order to define an enhancement technique for both video and block-coded image sequences, we have inserted a mechanism in the operator that allows the behavior of the filter to change according to the presence of the coding artifacts.

[00225] Equation 4.2 describes the approach that has been chosen in order to achieve this goal, where β takes a zero value in uncompressed images, while increasing in block-coded images. In this way, the behavior of the filter ranges from a quasi-linear low-pass one in uniform area, to a strongly nonlinear high-pass one in detailed areas. Moreover, it becomes more and more low-pass as coding artifacts increases.

[00226] From the consideration in the previous section, it is clear that β depends on the amount of the coding artifacts in the sequence. To this purpose, we will now show the relation between and the number of coding artifact detected, which-takes into account the size of the image and the number of the frame used for estimate β.

[00227] For the sake of simplicity, we define UM_video, UM_block and UM_adapt, the methods described in Sections 2, 3 and the preceding part of Section 4, respectively. In order to detect blocking and ringing artifacts, the conditions (4.6) and (4.7) have to be verified. Experimental results show that the following thresholds $\{\begin{array}{c}{\sigma }_{1_x}^2+{\sigma }_{2_x}^2=0.5\\ {\mu }_{1_x}-{\mu }_{2_x}>5\end{array}\mathrm{for}\mathrm{detection}\mathrm{of}\mathrm{blocking}\mathrm{effects},\mathrm{and}\{\begin{array}{c}{\sigma }_{1_x}^2-{\sigma }_{2_x}^2>25\\ {\sigma }_{1_x}^2+{\sigma }_{2_x}^2={\sigma }_{1_x}^2-{\sigma }_{2_x}^2\\ {\mu }_{1_x}-{\mu }_{2_x}<3\end{array}\mathrm{for}\mathrm{detection}\mathrm{of}\mathrm{ringing}\mathrm{noise}$
are satisfactory.

[00229] In our experiments, we used image sequences after MPEG coding of the 576×720 frames at various bit-rate. Since the blocking effect is more visible in I frames than in B and P frames due to the motion compensation, the results take into account different frames.

[00230] N_bl and N_ri indicate how many pixels are interpreted as blocking effect and ringing noise, respectively. N_bl=N_ri=0 for uncompressed video sequences, while it increases in block-coded sequences as the bit-rate decreases. It should be noted that in some sequences, the ringing noise is more visible than the blocking effect.

[00231] From the above considerations, it is clear that β depends on the values of N_bl, and N_ri. Obviously, the number of coding artifacts also depends on the size of the image and the number of the frame considered; hence, let N_c, and N_r be the width and the height of their image, and N_f be the number of frame used to calculate N_bl and N_ri, an appropriate measure for β is: $\begin{array}{cc}\beta =\alpha \frac{Nbl+Nri}{NcNrNf}& \left(5.1\right)\end{array}$
where α is a constant weight which can be derived from experimental results. In our experiment, N_c=720, N_r=576, N_f=4 and α=3000.

[00233] Our method detects both blocking and ringing artifacts and it does not need any information about the position and the size of the blocks. Moreover, β always take a zero value on uncompressed image; therefore, is very simple distinguish a coded image between a video one.

[00234] In order to further test the noise robustness of the proposed method, an original sequence has been corrupted using Gaussian-distributed noise of variance 50. If a prior art linear UM filtering (λ=0.5) is applied, the image is well sharpened, but the noise is amplified and it becomes so relevant to degrade the output scene. The best visual quality is obtained using the proposed UM_video filtering structure (λ=1.2, d₁=30 and d₂=40); in fact the spatial operator (see Sec. 2.1) enhances the medium-contrast details and the temporal filter (see Sec. 2.1.2) reduces the noise.

[00235] A comparison of the enhancement between UM_video, UM_block and UM_adapt methods, yields the following results. Using the UM_video operator, with λ=1, d₁=30 and d₂=40, the processed image is sharp and the noise is almost absent. The UM_block method (λ=1, d₁=30, d₂=0, σ_th²=400) shows an improvement of the quality inhomogeneous area; nevertheless, it introduces significant blurring. In fact, details of intermediate strength are not as well defined as in the previous case. Finally, applying the UM_adapt method to the original image, it can be seen that the processed image is similar to that of UM_video. In fact, in this sequence, N_bl=N_ri=0 and β=0. Therefore, the low-pass effect of the filter is reduced and the low-contrast details are amplified. This means that UM_adapt can be applied on both the video and compressed images.

[00236] The results on compressed image sequences are shown in the following. Applying a conventional linear UM operator (λ=0.5) to the coded and decoded image, the processed image is sharp, but the ringing effect is very visible in the uniform areas. The image restored by UM_block operator (λ=1, d₁=30, d₂=40, σ_th²=400) is free of ringing effect and the edges are enhanced. The image enhanced by UM_block method is free of blocking artifacts. There is practically no blurring of the texture. Further, edges are amplified and the image is more sharp. Obviously, the same results are obtained with UM_adapt operator, because in compressed images, β>>0. However, its effectiveness depends on the value of β. When β=0, λ=1, d₁=30, d₁=40 and σ_th²=200, as expected, the sharpening is effective, especially for medium-contrast details. However, the coding artifacts are amplified in smooth areas. Using β=1, the homogeneous areas are less noisy than in the case of β=0. Nevertheless, a good sharpening is achieved in the detail areas. Finally, with β=4, it can be seen that coding artifacts are removed, but it introduces significant blurring.

[00237] From these considerations, it is clear that β cannot be a constant value, but it has to depend from the presence and the amount of the coding artifacts in the images. Equation 5.1 describes the approach that has been chosen in order to achieve this goal.

[heading-00238] 4.1.3 Observations

[00239] 1. A major proposed algorithms to define a metric for blockiness suppose that the blocking artifacts are always located at block boundaries, but in video coding, the blocking artifacts can be located at any position due to the motion-compensated prediction. Our distortion measure does not need any information about the position and the size of the blocks and it also detects the ringing noise.

[00240] 2. Under a computational viewpoint, it must be stressed that the proposed solution maintains almost the same simplicity as the method described in Section 3 for the enhancement on compressed sequences. In fact, in order to detect blocking and ringing artifacts we use the variance value which has already been estimated in the expression of the rational function (see Eq. 3.3).

[00241] 3. If our method is applied to the I frames, it is possible to determine the size of a block; in fact, in these frames, the blocking and ringing artifacts are always located at block boundaries. A simple method is described in the following, where, for the sake of simplicity, we consider only the horizontal direction. First, we define a vector pos_x of length N_c, where N_c is the width of the image. When in a block-coded image BCI, a blocking or ringing artifact A is detected, the respective position of pos_x is set to 1. In this way, a binary vector pos_x is generated where a 1 value indicates the position of a coding artifact (see FIG. 10). Since the blocking and ringing artifacts A are located at block boundaries BB, pos_x can be expressed as: ${\mathrm{pos}}_x\left(n\right)\simeq \sum _{\mathrm{ii}=1}^{Nc/B}\delta \left(n-kB\right)$
where B is the size of the blocks. Then, the autocorrelation of $\begin{array}{cc}{R}_x\left(n\right)=\sum _{i=1}^{Nc}{\mathrm{pos}}_x\left(I\right)·{\mathrm{pos}}_x\left(i+n\right)& \left(4.8\right)\end{array}$
is maximum when n=kB. Therefore, in order to determine the size of the block we can simply calculate the position of the first maximum of R_x and R_y.

[00244] As an example, FIG. 11 shows the result on a given coded scene after coding at 1.5 Mb/s. It has to be noted that R_x(n) and R_y(n) take a zero value for n≠k8, therefore the image is segmented into 8×8 size blocks.

[00245] 4. Our method cannot be applied in a graphical image because an abrupt change in intensity slope is rather likely; hence, many sharp edges would be erroneously interpreted as blocking effect. Moreover, in these images, the variance in uniform areas takes a zero value; hence, the conditions (4.6) is verified for many pixels.

[00246] In order to distinguish a coded image between a graphical one, we observe that in the latter, the sharp edges can be located at any position; therefore, the autocorrelation of pos_x and pos_y takes a high value also for n=1 (see Eq. 4.8): $R_{x\left(1\right)}=\sum _{i=0}^{Nc}{\mathrm{pos}}_x\left(i\right)·{\mathrm{pos}}_x\left(i+1\right)$ $R_y\left(1\right)=\sum _{i=0}^{Nr}{\mathrm{pos}}_y\left(i\right)·{\mathrm{pos}}_y\left(i+1\right)$
From the above consideration, it is simple to discriminate the different type of images; in fact the following three cases are possible:

• [00248] (a) N_bl+N_ri>>R_x(1)+R_y(1)≃0: the image is classified as a block coded image;
• [00249] (b) N_bl+N_ri>>0 and R_x(1)+R_y(1)>>0: the image is classified as a graphical image;
  • [00250] (c) N_bl+N_ri≃0: the image is classified as a real uncompressed image.
    5. Conclusions

[00252] In this description, we have proposed a sharpness enhancement technique for video applications, taking into account the fact that the images can be blocked.

[00253] Conventional 2-D techniques for contrast sharpening cannot be applied to the enhancement of video image sequences. In fact, they introduce small artifacts and non-homogeneities which are perfectly acceptable in a still picture but become very visible and annoying in an image sequence; moreover, the visibility of a defect strongly depends on the sequence contents, namely on the amount of details and of motion.

[00254] A 3-D UM algorithm has been proposed in Section 2 in order to overcome such problems; despite its simple formulation, it is able to produce a good noise-insensitive sharpening action without introducing unpleasant overshoot and temporal artifacts. However, this operator cannot be applied to the enhancement of block-coded image sequences, because it emphasizes such coding artifacts that become very visible and annoying. For this task, a simple modification has been introduced when processing block-coded image sequences in order to emphasize the details and reduce the visibility of blocking and ringing effects (see Section 3).

[00255] Hence, in coded image sequences, the behavior of the filter ranges from a quasi-linear low-pass one in uniform areas, to a strongly nonlinear high-pass one in detailed areas. The variance detector that regulates the filter behavior is embedded in the filter itself, which allows a compact expression of the operator.

[00256] Finally, in order to define an enhancement technique for both the video and coded image sequences, a mechanism that allows the behavior of the filter to change according to the presence of the coding artifacts, has been inserted in the operator. To this purpose, a new distortion measure was introduced which does not need any information about the position and the size of the blocks. In fact, the major proposed algorithms to define a metric for blockiness suppose that the blocking artifacts are always located at block boundaries, but in video coding, the blocking artifacts can be located at any position due to the motion-compensated prediction. The algorithm has been evaluated by computer simulations and its results clearly show that it outperforms the existing algorithms. A specific component of the operator allows this technique to be employed for decoded sequences in a PC environment, also thanks to its low cost in terms of hardware and/or required processing power. The algorithm has proven to be robust against the heterogeneous contents of real sequences, giving very good performance even on very demanding images.

[00257] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word “comprising” does not exclude the presence of elements or steps other than those listed in a claim. The word “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[none]

• [00258] [1] A. K. Jain, Fundamentals of Digital Image Processing, Prentice Hall, Inc., Englewood Cliffs, N.J., 1989.
• [00259] [2] W. K. Pratt, Digital Image Processing, John Wiely and Sons, New York, 1991.
• [00260] [3] H. C. Andrews, A. G. Teschler, and R. P. Kruger, \Image processing by digital computer”, IEEE Spectrum, vol. 9, July 1972, pp. 20-32.
• [00261] [4] Y. H. Lee and S. Y. Park, “A study of Convex/Concave Edges and Edge-Enhancing Operator Based on the Laplacian”, IEEE Trans. on Circuits and System, vol. 37, no. 7, July 1990, pp. 940-946.
• [00262] [5] S. K. Mitra, H. Li, I. Li and T. Yu, “A new Class of Nonlinear Filters for Image Enhancement”, Proc. Inter. Conf. on Acoust., Speech and Signal Processing, ICASSP-91, pp. 2525-2528, 1991.
• [00263] [6] G. Ramponi, “A Cubic Unsharp Masking Technique for Contrast Enhancement”, IEEE Trans. Signal Processing, Vol. 67, No. 2, June 1998, pp. 211-222.
• [00264] [7] G. Ramponi and A. Polesel, “A Rational Unsharp Masking Technique” Journal of Electronic Imaging, vol. 7, no. 2, April 1998, pp. 333-338.
• [00265] [8] F. Alaya Cheikh and M. Gabbouj, “Directional Unsharp Masking-Based Approach for Color Image Enhancement”, Proc. of NM-BIA98, Glasgow, 1-3 Jul. 1998, pp. 173-178.
• [00266] [9] G. T. Jaspers and Peter H. N. de With, “A generic 2D sharpness enhancement algorithm for luminance signal”, Proc. IPA-97, IEE Conf., May 6, 1997.
• [00267] [10] S. I. Olsen, “Noise Variance Estimation in Images”, CVGIP, Vol. 4, No. 5, July 1993, pp. 521-528.
• [00268] [11] R. M. Hodgson, D. G. Bailey. “Properties, implementations and application of rank filters”, Image and Vision Computing, vol.3, no.1, February 1985.
• [00269] [12] R. Rosenholtz and A. Zakhor, “Iterative procedures for reduction of blocking effects in transform image coding”, IEEE Trans. Circuits Syst. Video Technol., vol. 2, pp. 91-95, March 1992.
• [00270] [13] Y. Yang, N. P. Galatsanos, and A. K. Katsaggelos, “Regularized reconstruction to reduce blocking artifacts of block discrete cosine transform compressed images”, IEEE Trans. Circuits Syst. Video Technol., vol. 3, pp. 421-432, December 1993.
• [00271] [14] S. Marsi, R. Castagno, and G. Ramponi, “A Simple Algorithm for the Reduction of Blocking Artifacts in Images and its Implementation”, IEEE Trans. on Consumer Electronics, vol. 44, no. 3, August 1998.
• [00272] [15] S. Minami and A. Zakhor, “An optimization approach for removing blocking effects in transform coding”, IEEE Trans. Circuits Syst. Video Technol., vol. 5, pp. 74-82, April 1995.
• [00273] [16] B. Jeon and J. Jeon, “Blocking Artifacts Reduction in Image Coding Based on Minimum Block Discontinuity Criterion”, SPIE Visual Communications and Image Processing, vol. 2501, pp.198-209, 1995.
• [00274] [17] C. J. Kuo and R. J. Hsieh, “Adaptive postprocessor for block encoded images”, IEEE trans. Circuits Syst. Video Technol., vol. 5, pp. 298-304, August 1995.
• [00275] [18] H. R. Wu, M. Yuen, “A generalized block-edge impairment metric for video coding”, IEEE Signal Processing, Vol. 4, No. 11, November 1997, pp. 317-320.