Edge dependent motion blur reduction

[0001] This invention relates to a method to reduce motion blur of images shown in non-stroboscopic display devices, in particular Liquid Crystal Display Panels (LCDs), Thin Film Transistor Displays (TFTs), Colour Sequential Displays, Plasma Display Panels (PDPs), Digital Micro Mirror Devices or Organic Light-Emitting Diode (OLED) displays, in which motion vectors depending on moving components in each image of an input video signal are calculated, in which anti-motion blur filtering of the input video signal is performed based on the calculated motion vectors to produce an output video signal, and in which images are generated on said display device depending on said output video signal. The invention further relates to a circuit arrangement providing anti motion blur functionality.

[0002] Non-stroboscopic displays, such as Liquid Crystal Displays (LCD), Plasma Panel Displays (PDP), Thin Film Transistor Displays (TFT), Colour Sequential Displays, Digital Micro Mirror Devices or Organic Light-Emitting Diode (OLED) displays consist of a display panel having a row and column array of picture elements (pixels) for modulating light, means for illuminating the display panel from the front or back side, and drive means for driving the pixels in accordance with an applied input video signal.

[0003] In state-of-the-art Cathode Ray Tubes (CRTs), each pixel of a displayed image is generated as a pulse, which is very short compared to the picture period T. Different to these state-of-the-art CRTs, in new flat, high quality, low cost displays devices, each pixel is displayed during most of the picture period. Of course, this non-stroboscopic behaviour also holds for types of CRTs whose pixels, e.g. slow phospor atoms, are active for a time not negligible to the picture period. In the sequel of this description, we thus will only differentiate between stroboscopic and non-stroboscopic displays, and in case of a non-stroboscopic display, we will use the term “pixel” for both the elements of a light modulation an-ay and the activated (slow) atoms of a CRT-type display.

[0004] In case any part of the image displayed on a non-stroboscopic display contains motion, the viewer will track this motion. As each pixel is displayed substantially the whole picture period, the intensity of pixels showing the motion is integrated along the motion trajectory as follows: $\begin{array}{cc}{F}_{\mathrm{out}}\left(\stackrel{->}{x},n\right)=\frac{1}{t_i}{\int }_0^{t_i}F\left(\stackrel{->}{x}+\frac{t}{T}\stackrel{->}{D},n\right)dt& \left(1\right)\end{array}$
with t_i as display time of each image, F as input video signal, F_out as output video signal, and T as picture period. The motion vector {right arrow over (D)}={right arrow over (v)}T is the product of the object velocity {right arrow over (v)} and the picture period T. In case t_i is constant, the integration is the same as a convolution of F({right arrow over (x)}, n) and a sample-and-hold function h(α): $\begin{array}{cc}\begin{array}{c}{F}_{\mathrm{out}}\left(\stackrel{->}{x},n\right)=\frac{T}{t_i}{\int }_0^{\frac{t_i}{T}}F\left(\stackrel{->}{x}+\alpha \stackrel{->}{D},n\right)d\alpha \\ ={\int }_{-\infty }^{\infty }F\left(\stackrel{->}{x}+\alpha \stackrel{->}{D},n\right)·h\left(\alpha \right)d\alpha \end{array}\mathrm{where}& \left(2\right)\\ h\left(\alpha \right)=\{\begin{array}{cc}T/t_i,& 0\le \alpha \le t_i/T\\ 0,& \mathrm{otherwise}\end{array}& \left(3\right)\end{array}$
is a 1D block function, oriented along the motion vector {right arrow over (D)}. It is therefore actually a 2D function h({right arrow over (x)}), which has zero value outside the line segment {right arrow over (x)}=k{right arrow over (D)},0≦k≦t_i/T, while the 2D integral area is normalised to 1. The 2D spatial Fourier transform is: $\begin{array}{cc}\begin{array}{c}{F}_{\mathrm{out}}\left(\stackrel{->}{f},n\right)={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{F}_{\mathrm{out}}\left(\stackrel{->}{x},n\right)e^{\left(-\mathrm{j2\pi }\stackrel{->}{x}\stackrel{->}{f}\right)}d\stackrel{->}{x}\\ =F\left(\stackrel{->}{f},n\right)·H\left(\stackrel{->}{f}\right)\end{array}& \left(4\right)\end{array}$
with F({right arrow over (f)},n) the 2D spatial Fourier transform of the original signal F({right arrow over (x)},n), and H({right arrow over (f)}) the 2D spatial Fourier transform of h({right arrow over (x)}): $\begin{array}{cc}H\left(\stackrel{->}{f}\right)=\frac{\sin \left(\pi \stackrel{->}{D}\frac{t_i}{T}\stackrel{->}{f}\right)}{\pi \stackrel{->}{D}\frac{t_i}{T}\stackrel{->}{f}}& \left(5\right)\end{array}$

[0005] Apparently the effect of the motion tracking/temporal sample-and-hold characteristic is a low-pass filtering in the direction of the motion with a sinc-shaped frequency response, with a cut-off-frequency being inversely proportional to the quantity t_i/T {right arrow over (D)}.

[0006] From EP 0 657 860 A2 it is known that motion blur in non-stroboscopic displays can be pre-compensated by anti-motion blur filtering of the input video signal that drives the pixels of the display panel. In EP 0 657 860 A2, this anti-motion blur filtering is implemented with a high spatial frequency boosting filter based on estimated motion vectors. As the viewer of moving objects on a matrix display integrates the intensity of the pixels along the motion trajectory, which, according to equation 5, corresponds to a low pass filtering in the spatial frequency domain, motion blur may be reduced by enhancing high spatial frequencies of moving objects. The higher the speed of moving components, the larger the part of the spectrum that needs enhancement.

[0007] A drawback of the proposed solution is that in areas where the motion vector is not reliable, i.e. in areas where there exists little contrast, filtering may be carried out without improving the picture. Furthermore, filtering may even cause noise modulation. In this case flat parts of the picture are filtered where the filtering cannot improve significant detail. It can however, result in visible differences in noise patterns, i.e. noise modulation.

[0008] It is thus the object of the invention to provide an improved spatial filtering with less noise enhancement and less noise modulation.

[0009] To solve the object of the invention, it is proposed that edge characteristics in each image of the input video signal are determined and that anti-motion blur filtering is further based on said determined edge characteristics. In this way, spatial filtering is concentrated on significant parts of an image only, and both noise enhancement and noise modulation are reduced.

[0010] Edge characteristics such as length and/or orientation of edge vectors and/or the presence of an edge are further proposed for a preferred embodiment of the invention.

[0011] Said edge characteristics are preferably derived from zero-crossings of the absolute values of the first derivatives of the image signal in two orthogonal directions subtracted by fixed thresholds to reduce sensitivity to noise.

[0012] Low and narrow edges are preferably not considered in the calculation of length and orientation of said edge vectors.

[0013] The presence of edges is preferably detected by sampling the edge vectors into binary signals.

[0014] According to a further preferred embodiment of the invention, the presence of edges may be smoothed out over space by dilating and/or corroding and/or low-pass filtering said binary signals.

[0015] A further preferred embodiment of the invention proposes that anti-motion blur filtering is performed by filtering the input video signal with a spatial filter, by subsequently combining the filtered input video signal with the input video signal itself to produce an intermediate video signal, by multiplying the intermediate video signal with gain factors to produce an amplified intermediate signal, and by combining said amplified intermediate signal with said input video signal to produce an output video signal. In said multiplication, the applied gain factors can be pixel-specific or as well be constant for a group of pixels.

[0016] The spatial filter applied to the input video signal may be favourably implemented as a high spatial frequency boosting filter. This filter can be implemented as 1D filter to save costs and simplify signal processing, as a standard 2D filter or even as a 3D filter, if also image data from previous or next images is used in the filtering process.

[0017] In a further advantageous embodiment of the invention, the filter coefficients of the high spatial frequency boosting filter, which determine the type of filter characteristic such as low-pass, mid-pass or high-pass, and the direction of the frequency characteristic of this spatial filter depend on the motion vectors, whereas the gain factors depend on the length of the motion vectors and the presence of edges. Noise enhancement and noise modulation is then reduced by suppressing image processing in low detailed parts of the image where no improvement will be achieved. By using edge characteristics as a point of interest selector, discrimination in the amount of processing for various parts of the image can be made.

[0018] In a further preferred embodiment, the filter coefficients of the high frequency boosting filter and the direction of the frequency characteristic of the filter depend on the motion vectors, whereas the gain factors depend on the inner product of edge vectors and motion vectors normalised to the length of the edge vectors.

[0019] By gain control, image processing is then concentrated on parts of the image where the angle between the motion vector and the edge vector is small, reflecting the fact that a vertical luminance transient with corresponding vertical edge vector will not be affected by motion blur if the corresponding motion vectors are oriented in horizontal direction.

[0020] Yet another advantageous embodiment of the invention proposes that the direction of the frequency characteristic of the filter depends on the orientation of the edge vectors, that the filter coefficients of the high spatial frequency boosting filter depend on the motion vectors and/or the edge vectors, and that the gain factors depend on the motion vectors and/or the edge vectors. This directional control of image processing considers the fact that motion vectors can be unreliable due to the aperture problem, stating that large estimation errors occur for motion vector components parallel to edges. Motion estimation parallel to an edge is difficult because in contrast to perpendicular to an edge, parallel to an edge little or no detail is available for the motion vector estimation algorithm to work with. By performing anti-motion blur filtering in the direction of the edge vectors rather than in the direction of the motion vectors, motion blur is thus reduced with less noise enhancement and noise modulation.

[0021] A low-cost embodiment of the invention is achieved by proposing that a fixed set of filter coefficients for the high spatial frequency boosting filter is used, that the direction of the frequency characteristic of the spatial filter depends on the orientation of the edge vectors and that the gain factors depend on the motion vectors and/or the edge vectors. The high spatial frequency boosting filter then is vastly simplified, for only the direction of the frequency characteristic of the spatial filter depends on local image characteristics. The gain, which defines the parts of the image where image processing takes place, is controlled by both motion and edge vectors.

[0022] The dependence of gain factors on motion vectors and/or edge vectors may be preferably related to the length of the motion vectors and the presence of edges.

[0023] The dependence of gain factors on motion vectors and/or edge vectors may be further preferably related to the inner product of edge vectors and motion vectors normalised to the length of the edge vectors. Then anti-motion blur filtering is mainly performed in parts of the image where the angle between motion and edge vectors is small, i.e. only the part of the motion vector across an edge is used for sharpness enhancement.

[0024] The invention further comprises a non-stroboscopic display device, in particular a Liquid Crystal Display (LCD), Thin Film Transistor Display (TFT), Colour Sequential Display, Plasma Display Panel (PDP), Digital Micro Mirror Device or Organic Light-Emitting Diode (OLED) display with means to calculate motion vectors in each image of an input video signal, with means to filter the input video signal depending on said calculated motion vectors to produce an output video signal, and with means to display the images of the output video signal on a display panel, where means to determine the edge characteristics of each image of the input video signal are provided and where means to filter the input video signal depending on both the calculated motion vectors and the determined edge characteristics are provided.

[0025] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. In the figures show:

[0026] FIG. 1 a schematic representation of a first embodiment of a display system with edge-dependent motion blur reduction, and

[0027] FIG. 2 a schematic representation of a second embodiment of a display system with edge-dependent motion blur reduction.

[0028] FIG. 1 depicts a first embodiment of a display system with the proposed edge-dependent reduction of motion blur. An input video signal 1 is filtered by an anti-motion blur filter 2 yielding an output video signal 3. To this aim, the input video signal 1 is fed into a motion estimation instance 4 and an edge estimation instance 5. The motion estimation instance 4 produces an estimate of the motion vectors of each image of the input video signal 1. The edge estimation instance calculates the edge vectors and/or determines the presence of edges in each image of the input video signal 1. The outcome of the motion 4 and edge 5 estimation instances is fed into the anti motion-blur filter 2.

[0029] Edges in each image of an input video signal can be determined by identifying luminance transients. To this end, the first derivatives of the luminance signal in both horizontal, $\begin{array}{cc}{Y}_h^{\prime }\left(\stackrel{->}{x}\right)=\frac{d}{d{x}_1}Y\left(\stackrel{->}{x}\right),& \left(6\right)\end{array}$
and vertical direction, $\begin{array}{cc}{Y}_v^{\prime }\left(\stackrel{->}{x}\right)=\frac{d}{d{x}_2}Y\left(\stackrel{->}{x}\right),& \left(7\right)\end{array}$
are calculated, where
Y({right arrow over (x)})=Y(x₁,x₂)  (8)
is the two-dimensional luminance (image) signal.

[0030] The edge width in these directions can then be calculated by measuring the distance between zero crossings in the derivative signal (i.e. the width of the transient in both directions). To reduce sensitivity to noise, thresholds T_h and T_v can be subtracted from the absolute derivative signal.

[0031] The horizontal edge width h({right arrow over (x)}) is the distance between zero crossings of the signal
|Y′_h({right arrow over (x)})|−T_h  (9)
and the vertical edge width v({right arrow over (x)}) is the distance between zero crossings of the signal
|Y′_v({right arrow over (x)})|−T_v.  (10)

[0032] The edge width perpendicular to the edge is given by: $\begin{array}{cc}\stackrel{->}{e}\left(\stackrel{->}{x}\right)=\frac{h\left(\stackrel{->}{x}\right)·v\left(\stackrel{->}{x}\right)}{\sqrt{h^2\left(\stackrel{->}{x}\right)+v^2\left(\stackrel{->}{x}\right)}}.& \left(11\right)\end{array}$

[0033] These widths are assigned to all points inside the edge (inside the zero crossings).

[0034] To increase further robustness against noise in the edge detection algorithm, the condition for h({right arrow over (x)}) and similar for v({right arrow over (x)}) can be used that: $\begin{array}{cc}h\left(\stackrel{->}{x}\right)=\{\begin{array}{c}h\left(\stackrel{->}{x}\right),\max \left(Y_h^{\prime }\left(\stackrel{->}{x}\right)\right)>\frac{T_r}{h\left(\stackrel{->}{x}\right)}\forall \stackrel{->}{x}\in \left[\stackrel{->}{x}-\frac{h\left(\stackrel{->}{x}\right)}{2},\stackrel{->}{x}+\frac{h\left(\stackrel{->}{x}\right)}{2}\right]\\ 0,\mathrm{otherwise},\end{array}& \left(12\right)\end{array}$
where T_r denotes a fixed threshold.

[0035] This equation states that the width of an edge should be reverse proportional to its height, resulting in the discarding of low and narrow edges (probably noise). Having knowledge of the horizontal and vertical edge width, combined with the gradient of the edge in both directions (white to black or black to white), it is possible to calculate the orientation angle arg({right arrow over (e)}) of the edge. Together with the edge width |{right arrow over (e)}({right arrow over (x)})| then the edge vector {right arrow over (e)}({right arrow over (x)}) can be determined.

[0036] If the anti-motion blur filter is implemented based on a High Spatial Frequency Boosting Filter (HFBF) 6, as depicted in FIG. 1, the input video signal is processed as follows:

[0037] The input video signal 1 is first filtered with the HFBF 6 to produce a filtered input video signal. This filtered video signal is then combined with the input video signal 1 yielding an intermediate signal. The intermediate signal is subsequently multiplied with a gain factor k to produce an amplified intermediate signal, which then is combined with the input video signal 1 to produce the output video signal 3. The output video signal 3 then drives the row and column pixels of the display panel.

[0038] If the intermediate signal is produced by subtracting the input video signal from the filtered input video signal, it is evident that the difference of both signals only has non-zero components for spatial frequencies where filtering took place. The gain factor k allows to select if and to what extent this filtering is desired, e.g. to reduce the dynamic range of the amplified intermediate signal.

[0039] The HFBF 6 can now be characterized by a set of filter coefficients f and a direction φ of the frequency characteristic of the HFBF. According to FIG. 1, both the filter coefficients f=f({right arrow over (v)},{right arrow over (e)}) and the direction φ=φ(arg({right arrow over (v)}), arg({right arrow over (e)})) are determined by a filter coefficients and filter direction control instance 7 based on the outcome of the motion 4 and edge 5 estimation instance. Similarly, the gain factor k=k({right arrow over (v)}, {right arrow over (e)}) is determined by a gain factor control instance 8 based on the outcome of the motion 4 and edge 5 estimation instance.

[0040] FIG. 1 describes the most general case of motion- and edge-dependent motion blur reduction. The filter coefficients f=f ({right arrow over (v)}, {right arrow over (e)}), the direction of the frequency characteristic of the fiter φ=φ((arg({right arrow over (v)}), arg({right arrow over (e)})) and the gain factor k=k({right arrow over (v)},{right arrow over (e)}) may depend on the outcome of both the motion 4 and the edge 5 estimation instance. E.g., if anti-motion blur filtering is desired only for certain points of interest in the image, the filter coefficients k=k({right arrow over (v)},{right arrow over (e)}) depend on the motion and/or edge vectors, the direction φ=φ(arg({right arrow over (v)})) of the frequency characteristic of the filter depends on the orientation of the motion vectors and the gain factor k=k(|{right arrow over (v)}|, p) depends on the presence of edges p({right arrow over (x)})=p({right arrow over (e)}({right arrow over (x)})) and the length of the motion vectors |{right arrow over (v)}|, so that anti-motion blur filtering is only performed if there are edges in the image. The presence of edges p({right arrow over (x)})=p({right arrow over (e)}({right arrow over (x)})) represents a function of the edge vectors and may be obtained by sampling the edge vectors into one bit signals (there is an edge or not) and possibly dilating and corroding these signals.

[0041] Quite similar, in gain-controlled filtering the filter coefficients f=f({right arrow over (v)},{right arrow over (e)}) depend on the motion and/or edge vectors, the direction φ=φ(arg({right arrow over (v)})) of the frequency characteristic of the filter depends on the motion vectors, but the gain factor k=k({right arrow over (v)},{right arrow over (e)}) depends on the inner product of edge vectors and motion vectors normalised to the length of the edge vectors, i.e. $k=k\left(\frac{\stackrel{->}{e}·\stackrel{->}{v}}{\stackrel{->}{e}·\stackrel{->}{v}}·\stackrel{->}{v}\right)=k\left(\frac{\stackrel{->}{e}·\stackrel{->}{v}}{\stackrel{->}{e}}\right),$
so that filtering is only performed when the angle between edge and motion vectors is small.

[0042] In contrast, in direction-controlled filtering the direction φ=φ(arg({right arrow over (e)})) of the frequency characteristic of the filter depends on the orientation of the edge vectors in order to avoid estimation errors of the motion vectors that are encountered if motion vector estimation parallel to an edge has to be performed. The filter coefficients f=f({right arrow over (v)},{right arrow over (e)}) and the gain factor k=k({right arrow over (v)},{right arrow over (e)}) once again may depend on the motion vectors and/or the edge vectors. For the gain factor k=k({right arrow over (v)},{right arrow over (e)}), a dependence k=k(|{right arrow over (v)}|,p) on the presence of edges and the length of the motion vectors or a dependence $k=k\left(\frac{\stackrel{->}{e}·\stackrel{->}{v}}{\stackrel{->}{e}}\right)$
on the normalised inner product of motion and edge vectors may be imagined.

[0043] FIG. 2 depicts a second embodiment of a display system with edge-dependent motion blur reduction, in particular a low cost version. In contrast to FIG. 1, a fixed set of filter coefficients f≠f({right arrow over (v)},{right arrow over (e)}) is used for the HBEF 6, so that the filter coefficients and filter direction control instance 7 simplifies to a filter direction control instance. Furthermore, the outcome of the motion estimation instance 4 only influences the gain factor control instance 8 and no longer the filter direction control instance 7. This set-up ensures that anti-motion blur filtering is only performed in the direction of edge vectors, where motion blur actually occurs. Filtering in parallel to an edge, with does not improve image sharpness, but enhances noise and modulates noise, is thus omitted. To further reduce the enhancement of noise, the gain factor k=k({right arrow over (v)},{right arrow over (e)}) is controlled based on both motion and edge vectors, e.g. by choosing the gain $k=k\left(\frac{\overrightarrow{e}·\overrightarrow{v}}{\overrightarrow{e}}\right)$
proportional to the inner product of both vectors.

[0044] The filter coefficients f≠f ({right arrow over (v)},{right arrow over (e)}), which are kept fixed in this set-up irrespective of the amount of motion in the present image, may be based on average values of the amount of motion or may be updated with a dwell time of several image periods. As a 1D example of such a low-cost filter, an FIR filter with taps [−1 2 −1] may be applied to adjacent pixels to approach the required inverse filter to the filter of equation 5. This low-cost version of the HFBF does advantageously avoid the poles of the theoretically needed inverse filter to the filter of equation 5 and comprises only a few filter taps.