Real time human vision system behavioral modeling

[0002] The present invention relates to video picture quality assessment, and more particularly to realtime human vision system behavioral modeling for producing objective measures to predict a subjective rating of errors in a video image signal that is recorded and transmitted via methods that are lossy, such as video compression.

[0003] Existing methods for using human vision system models for predicting observer subjective reactions to errors introduced into a video image signal subjected to lossy processes, such as video compression, include computationally expensive human vision system (HVS) models, such as those described by J. Lubin, “A Visual Discrimination Model for Imaging System Design and Evaluation”, Vision Models for Target Detection and Recognition, World Scientific Publishing, River Edge, N.J. 1995, pp. 245-283, or by S. Daly “The Visible Differences Predictor: An Algorithm for the Assessment of Image Fidelity”, Digital Images and Human Vision, MIT Press, Cambridge, Mass. 1993, pp. 162-206. Measures used to predict subjective impairment ratings that do not use human vision models include ANSI/IRT measurements (see “Digital Transport of One-Way Signals-Parameters for Objective Performance Assistance”, ANSI T1.801.03-yyy) that are generally faster but, given a sufficiently varied set of video image content, do not correlate as well with subjective ratings as do the methods that include HVS models.

[0004] Most HVS models are based on methods of predicting the threshold of noticeable differences, commonly referred to as Just Noticeable Differences (JND), such as contrast detection and discrimination thresholds. Since the model components are based on mimicking behavior at threshold, behavior above threshold, i.e., at suprathreshold, is not guaranteed. These HVS models generally include one or more stages to account for one or more of the experimentally determined behaviors near incremental contrast detection and discrimination threshold as effected by the following parameters:

[0005] mean luminance

[0006] angular extent or size of target image on retina

[0007] orientation (rotational, both of target image pattern and masker)

[0008] spatial frequency (both of target image pattern and masker)

[0009] temporal frequency (both of target image pattern and masker)

[0010] surround (or lateral masking effects)

[0011] eccentricity (or angular distance from the center of vision/fovea)

[0012] What follows is a brief summary of how one or more of the effects of these seven parameters have been accounted for in HVS models.

[0013] First it is worth noting the approach to the image processing flow structure in prior art. A large portion of the processing time required in HVS models is due to two common implementation stages:

[0014] filter bank (image decomposition such as Gaussian pyramids)

[0015] contrast gain control (contrast masking non-linearity)

[0016] Filter banks are popular for image decomposition into neural images or channels with maximum response at various orientations, spatial frequency bands, polarities, etc. For a practical implementation a minimal decomposition of two orientations (horizontal, vertical), four spatial frequency bands and two polarities requires 2*4*2=16 images per processing stage for the reference image signal, and likewise for the impaired video image signal.

[0017] For the typical HVS model response sensitivity as a function of spatial frequency has been accounted for in what has been called the contrast sensitivity function. The contrast sensitivity portion of the model has been accomplished by:

[0018] Calculating the contrast at each pixel of each filter bank channel, corresponding to a unique combination of spatial frequency subband and rotational orientations, as the ratio of high frequency energy to low (DC) frequency energy, or the equivalent.

[0019] Scaling the contrast values depending on the sub-band and rotational orientations.

[0020] Calculating contrast requires two different filters, high pass and low pass, and a division for each pixel of each channel. Even with this complex and expensive algorithm stage, variation in spatial frequency sensitivity function of local average luminance and angular extent of segment or self-similar regions of the image is not taken into account. The “linear range” is not exhibited in these models. At frequencies where sensitivity is generally the greatest, between one and four cycles per degree, the contrast sensitivity increases roughly proportional to the square root of the average luminance and likewise for angular extent. Thus, while the prior art includes quite complex and computationally expensive methods, by ignoring the effects of average luminance and angular extent threshold predictions may be in error by greater than an order of magnitude. Though models for part of the HVS have been proposed to account for the effects of average luminance and angular extent, they apparently have not been adopted into subsequent full HVS models, ostensibly due to the further added complexity.

[0021] The contrast gain control portion of the model is generally based on the work of J. Foley, such as his “Human Luminance Pattern-Vision Mechanisms: Masking Experiments Require a New Model”, Journal of the Optical Society of America, Vol. 11, No. 6 June 1994, pp. 1710-1719, that requires a minimum of

[0022] Calculation of the sum of energy (square) of respective pixels of the scaled contrast images over all channels. Lower resolution channels are up-sampled in order to be summed with higher resolution channels. This channel to channel conversion increases the effective throughput at this stage and further complicates implementation.

[0023] One addition, two non-integer exponentiations and one division operation per pixel per channel. M. Cannon, “A Multiple Spatial Filter Model for Suprathreshold Contrast Perception”, Vision Models for Target Detection and Recognition, World Scientific Publishing, River Edge, N.J. 1995, pp. 88-117, proposed a model that extends to the suprathreshold region with a substantial increase in complexity. However it too apparently has not been adopted into subsequent full HVS models, ostensibly due to the further added complexity.

[0024] Temporal effects on spatial frequency sensitivity in these models mostly either have been absent, have tended to only include inhibitory effects, or have been relatively complex.

[0025] Finally the effects of orientation and surround are only represented to the extent that the orthogonal filters and cross-pyramid level maskings are capable, generally not well matched with HVS experimental data.

[0026] A current picture quality analyzer, the PQA-200 Analyzer manufactured by Tektronix, Inc. of Beaverton, Oreg., USA, is described in U.S. Pat. No. 5,818,520. This is a non-realtime system based on the JNDMetrix® algorithm of Sarnoff Corporation of Princeton, N.J., USA where a reference image signal is compared with a corresponding impaired video image signal to obtain differences which are processed according to an HVS model. In order to perform the assessment, the system under test is essentially taken out of service until the test is complete.

[0027] What is desired is a realtime HSV behavioral modeling system for video picture quality assessment that is simple enough to be performed in a realtime video environment.

[0028] Accordingly the present invention provides realtime human vision system behavioral modeling for performing picture quality analysis of video systems in a realtime video environment. A reference image signal and a test image signal derived from the reference image signal are processed in separate channels. The image signals are converted to luminance image signals and filtered by a two-dimensional low-pass filter to produce processed image signals. The processed image signals are segmented into regions having similar statistics, and the segment or region means are subtracted from the pixels of the processed image signals to produce segmented processed image signals that have been implicitly high pass filtered. Noise is injected into the segmented processed image signals, and variances are calculated for the reference segmented processed image signals and for the differences between the reference and test segmented processed image signals. The variance for the difference segmented processed image signal is normalized by the variance for the reference segmented processed image signal, and the Nth root of the result is determined as a measure of visible impairment of the test image signal. The measure of visible impairment may be converted into appropriate units, such as JND, MOS, etc.

[0029] The objects, advantages and other novel features of the present invention are apparent from the following detailed description when read in conjunction with the appended claims and attached drawing.

[0030] The FIGURE is a block diagram view of a system for video picture quality assessment using realtime human vision system behavioral modeling according to the present invention.

[0031] In examining the behavior of the human vision system revealed by data from the literature in combination with evidence of the influence of various noise sources in the human vision system, the following observations may be made:

[0032] 1. Amplitude sensitivity at the upper spatial frequency range, the linear region of luminance influence on contrast sensitivity function, is insensitive to changing average luminance, yet falls off as the square of the frequency. This corresponds to a second order low-pass filter.

[0033] 2. The Weber and Devries-Rose regions (see Taylor et al, “Contrast Detection and Discrimination for Visual Models” and N. Graham, “Visual Pattern Analyzers”, Oxford U. Press 1989) both correspond to the lower frequency range where amplitude sensitivity is most influenced by changing average luminance. Examination of amplitude versus frequency curves at various average luminance reveals the behavior similar to a variable high-pass filter dependent on the average luminance. A model based on a similar observation was proposed by Cornsweet and Yellott, “Intensity-Dependent Spatial Summation”, Journal of the Optical Society of America, Vol. 22, No. 10 1985, pp. 1769-1786. Rohaly and Buchsbaum, “Global Spatiochromatic Mechanism Accounting for Luminance Variations in Contrast Sensitivity Functions”, Journal of the Optical Society of America, Vol. 6, Feb. 1989, pp. 312-317, proposed a model based on contrast, as opposed to amplitude, using a fixed high pass filter and variable low pass filter. In all cases a band pass is formed by combining, or cascading, a low pass filter with a high pass filter, with one filter fixed and the other a function of average luminance. Since contrast calculation requires additional steps, amplitude representation, which implies a fixed low pass filter and a variable high pass filter, is more efficient.

[0034] 3. The pedestal effect in contrast discrimination between near threshold intensities is consistent with the evidence of spatially stationary (fixed) noise—errors dependent on foveal location, but independent of time—and other noise in the human vision system.

[0035] Below the noise floor contrast increment for detection decreases with increasing pedestal threshold, while above the noise floor it increases with increasing pedestal threshold. Others have proposed non-fixed noise, but apparently these have yet to be incorporated in a full HVS model used for predicting visible impairment of a video image signal.

[0036] 4. The square root law relationship between angular extent and both contrast and discrimination thresholds implies each image should be segmented into regions of similar statistics, thereby dividing the full image area into smaller sets of somewhat “coherent” areas. The larger the area, the lower the effective contrast sensitivity high pass cut-off frequency is when the mean is subtracted.

[0037] 5. Contrast discrimination increment elevation due to lowered average luminance, dissimilar masking signals and reduced patch area implies a process similar to optimal signal detection using matched filters. In such a system differences may be measured by subtracting correlation coefficients between processed reference and test images from unity.

[0038] 6. Though subtracting the correlation coefficients from unity predicts threshold and discrimination well, it does not predict perceived suprathreshold differences well. Well above threshold perceived contrast increases with approximately the square root of the actual contrast. In order to match behavior at threshold and suprathreshold simultaneously with one simple mathematical expression, the Nth root of the normalized variance of processed image differences is used, which in this example is the fourth root:

[0039] Response=K*(var(Itest_proc−Iref_proc)/var(Iref_proc))^0.25where Iref_proc and Itest_proc are the processed, filtered and noisy, reference and test images respectively and var( ) is variance.

[0040] (a) At threshold the desired behavior is approximately:

[0041] Response=K*(sum(Itest_proc²+Iref_proc²−2*Itest_proc*Iref_proc)^0.25)/(var(Iref_proc)^0.25

[0042] Response˜=K*(2−2*correlation(Iref_proc,Itest_proc))^0.25=1

[0043] (b) Above threshold the desired behavior is approximately:

[0044] Response=K*(sum(|Itest_proc−Iref_proc|²)/var(Iref_proc))^0.25and for a flat field image reference, Iref_proc=noise

[0045] Response=K*(sum(|Itest_proc|²)/var(noise))^0.25

[0046] Here the suprathreshold response to contrast, relative to zero contrast, increases as the square root of the contrast. Thus both the threshold and suprathreshold conditions are met. Conveniently the Nth root need not be taken until the entire image has been pooled over segment measurements, which is equivalent to taking the q-norm with q=4 for this example.

[0047] Corresponding model components and six calibration parameters are:

[0048] (a) Two dimensional infinite impulse response second-order low-pass filter calibration parameters include (i) input scale, which could be placed in the display model for convenience, and (ii) feedback coefficient, which sets a double pole. This may be implemented in an equivalent finite impulse response filter, as is well known to those in the art.

[0049] (b) Segmentation (region growing) is guided by (i) threshold for segment block mean totals and (ii) threshold for approximate segment variance of block variance. Recalculate block means as mean of luminance in segment, using segment's block means, and subtract local mean (means luminance of the corresponding segment) from each pixel.

[0050] (c) Noise generator and “obscurer” (greatest value operator) calibration involves selection of noise amplitude. For each segment the normalized difference variance (variance of processed reference minus impaired, var(Iref_proc−Itest_proc), divided by variance of the processed reference image, var(Iref_proc)) is calculated. No calibration constants are required for this step.

[0051] (d) For the entire image the aggregate measure is calculated as the Nth root of the average of the segment normalized difference variances, where for this example N=4. This aggregate measure is converted to appropriate units with a conversion or scaling factor (K above for threshold or JND units).

[0052] Referring now to the Figure a flow chart for a picture quality assessment apparatus is shown that improves on the prior art by combining the bulk of the accuracy of the HVS models with the efficiency comparable to non-HVS models. A reference video signal and an impaired (test) video signal are input to respective display models 11, 12 for conversion into luminance units. The spatial frequency response, corresponding to contrast sensitivity threshold data, is implemented as respective two-dimensional low-pass filters 13, 14 and one implicit high pass filter (local mean (low pass) from step 15, 16 is subtracted in step 17, 18 respectively from each individual pixel). This filter combination satisfies the requirements suggested by the data (Contrast Sensitivity versus Frequency) in the literature for each orientation, average luminance and segment area. There is only one image output from this stage to be processed by subsequent stages for each image, as opposed to a multiplicity of images output from filter banks in the prior art. A nonlinear temporal filter could be inserted next to account for a great deal of the temporal behavior of the HVS model, but is not shown in this implementation.

[0053] Block means are calculated 15, 16, such as for three pixels by three pixels blocks. In both channels the image is segmented 20 based on block average mean and other simple block statistics. However for simplification and reduction of computation resources the step 20 may be omitted. Block statistics used in the current segmentation algorithm include local (block) means luminance and previous variance. However simple max and min values may be used for region growing. Each block means is averaged over the segment to which it belongs to create new block means. These means are subtracted 17, 18 from each pixel in the respective blocks completing the implicit high pass filter.

[0054] Noise from a noise generator 24 is injected at each pixel via a coring operation 21, 22 by choosing the greater between the absolute value of the filtered input image and the absolute value of a spatially fixed pattern of noise, where:

[0055] Core(A,B)={(|A|−|B|) for |A|>|B|; 0 for |A|<|B|}

[0056] where A is the signal and B is the noise. Segment variance is calculated 27, 28 for the reference image segments and the difference 26 between the reference and test image segments. The two channel segment variance data sets are combined 30 by for each segment normalizing (dividing) the test (difference) channel variance by the reference variance.

[0057] Finally the Nth root of the average of each segment's normalized variance is calculated 32 to form an aggregate measure. Again for this example N=4, where N may be any integer value. The aggregate measure may be scaled or otherwise converted 34 to appropriate units, such as JND, MOS, etc.

[0058] Thus the present invention provides realtime human vision system behavioral modeling for determining impairment of a test video image signal from a reference video image signal by processing the two signals in parallel channels, obtaining an averaged segment variance for the reference image and for a differenced (reference—test) image, normalizing the average segment variances, taking the Nth root to form an aggregate measure, and then converting the result to appropriate units.