Omission and Subsequent Estimation of Zero Sequency Coefficients of Transformed Digitized Images to Facilitate Data Compression

This invention relates to the transmission of images in digital form. The invention may be used to reduce the amount of data which is required to be transmitted to convey an image at a given resolution.

It is known to produce a video signal representing an image by scanning it to produce a signal similar to that used for television, and it is also known to sample the video signal and to convert the samples into digital form so that each picture element (PEL) corresponding to a sample of the video signal is represented by a plurality of binary digits representing the brightness of the particular element or the intensity of a particular colour component of that element. Such a method of transmission might be made more efficient by the use of non-linear quantisation of the samples so that a better reproduction of the image is obtained for a given number of bits in the digital coding of each sample than would be obtained if a uniform coding were employed.

As an image contains a large amount of data, it is desirable to reduce as far as possible the amount of data needed to transmit an image. Straightforward digital transmission of an image as described above suffers from the disadvantages that a reduction of the amount of data representing the image will detract markedly from the quality of the reproduced image. In order to overcome this difficulty it has been proposed to convert the digital video signal into a transformed signal in which the coded samples are subjected to a two-dimensional unitary transformation, and then reduce the transformed data. At the receiver the transformed signal is subjected to the inverse transform to regenerate the original image. When transmitted in this way, the data reduction does not result in degradation of the image which is as objectionable subjectively as the same data reduction on the image data itself would be.

According to the present invention there is provided a method of transmitting or storing an image including producing an array of samples representing the picture elements of an array of such elements representing an original image, notionally dividing the array of picture elements into a plurality of blocks, for each block subjecting the array of samples to a two-dimensional transformation, transmitting or storing representations of at least some of the coefficients, subjecting the received coefficients in each transformed block to the inverse of the original two-dimensional transformation to produce a block of restored samples and reproducing an image from the adjusted blocks, characterised in that for each of at least some of the blocks representations of the zero sequency coefficient are not transmitted or, as the case may be, stored, and values for the mean levels of the restored blocks are calculated so as to substantially minimise visible brightness and/or in a colour system, colour changes between the particular block and at least one of the neighbouring blocks.

Normally, a unitary transformation, such as the Hadamard or Discrete Cosine transform would be used. The adjustment of the mean levels of the restored blocks may be carried out so as to minimise the mean square value of the brightness differences between the nearest adjacent elements in different blocks at the boundaries between the blocks. This manner of adjustment is satisfactory if no edges in the image coincide with block boundaries, but it can give erroneous results if there is a substantial brightness change in the image lying on a transition between blocks. This difficulty can be overcome by measuring the mean square brightness difference at each edge between blocks of the original image before transformation and sending an indication if the difference exceeds a threshold value so that the adjustment of the mean levels for adjacent blocks can be arranged to ignore brightness differences at such edges.

To take a practical example, suppose that the original image has 256.times.256 elements and each block is of 8.times.8 elements. It follows that for the whole image there are 1024 blocks. The calculations required to minimise the mean square difference for the whole image at once are too lengthy for execution in a reasonable period of time, and therefore it is proposed that the mean level of one block, e.g. that in the top left-hand corner, is set to an arbitrary value, but one that is likely to be reasonable, and those of the other blocks are taken one at a time in sequence are calculated so as to minimise the mean square differences at the one or two edges which each block shares with blocks whose mean level has already been calculated, they may all be subjected to an additive correction by the same amount to bring the average grey level of the entire reproduced image within a desired range.

Instead of calculating the mean level of each block separately, as described above, the mean levels of a row (or column) of blocks may be calculated in that way and then the blocks dealt with a row (or column) at a time. Yet another way would be to amalgamate the blocks into groups of four arranged in a square, adjusting the relative mean levels to minimise the mean square differences along the four edges shared by the blocks of the group, then to amalgamate those groups into larger groups (of four groups) using the same technique, and so on until all of the blocks are amalgamated together. As before an overall additive correction may be made to adjust the grey level of the entire image.

The invention permits a reduction in the amount of data needed to be transmitted (or stored) by omitting from the data transmitted the d.c. coefficient of all or many of the blocks. Since this coefficient is the one normally transmitted with the greatest accuracy its omission will result in a greater reduction in the data to be transmitted than would be the omission of any other coefficient. If the d.c. coefficient determining the mean level of one block is transmitted, this block could be used as a reference to which the mean levels of all of the remaining blocks are related directly or indirectly; if the image is such that substantial errors could arise in the mean levels of the other blocks it may be desirable to transmit the d.c. coefficients determining the mean levels of more than one block, e.g. a row or column, or possibly of a few blocks distributed throughout the image area, limiting the extent to which the effect of transmission errors can "propagate" through the image during the reconstruction process.

When the relative adjustment of the mean levels has been effected on the basis of an arbitrarily chosen value in one block selected as reference, the mean levels of all of the blocks may be subjected to a common additive correction to produce a mean grey level within a given range for the entire image.

If the d.c. coefficient of a single block is transmitted or stored to provide, when decoded, a reference for the whole image as described above, it may still be desirable to adjust the mean levels to give a mean grey level within a given range for the image, although, in theory at least, such adjustment ought not to be necessary.

Some embodiments of the invention will now be described with reference to the accompanying drawings, in which:

FIG. 1 illustrates the division of an image into blocks;

FIG. 2 shows an example of the numbers of bits allocated to each coefficient of a transformed 8.times.8 block;

FIG. 3 is a block diagram of one example of an image transmission system using a method according to the invention; and

FIG. 4 is a diagram to be used to help to describe a method according to the invention.

In FIG. 1, an image is shown as being divided into 64 blocks, each composed of 64 picture elements (pels). It is assumed that the number of pels in both horizontal and vertical directions is the same. As a practical example, it is likely that an image of, say, 256.times.256 PELs might be divided into 1024 8.times.8 blocks.

The image is converted into a video signal, and sampled. Each 8.times.8 array of samples corresponding to a block is subjected to a two-dimensional unitary transformation.

It is impractical to transform the entire image in a reasonable time because the calculation involved in producing the transforms of images having over 65,000 PELs are at present too lengthy even for the fastest computing techniques. Division of the image into smaller blocks, for example, consisting of 8.times.8 PELs means that the computations involved are simplified by a factor of 1,000 and can be executed in a reasonable time by present computing techniques and hardware. Suitable transforms are the Discrete Cosine Transform or the Hadamard transform, for example. The transformation produces an 8.times.8 array of coefficients (in general, an N.times.M sample array produces an N.times.M array of coefficients, although it is common to discard--or not calculate--the higher sequency coefficients). The sequency in a given direction is the number of sign changes in the corresponding row or column of the basis matrix of the transform used.

FIG. 2 shows the number of bits which, in a conventional system, would typically be allocated to coefficients of a transformed block of the image for transmission of that image in the transformed state, ordered as to sequency; thus the top left term indicates that 8 bits are allocated to the zero sequency term, whilst the right hand column and bottom row correspond to the highest sequency coefficient in the two directions.

Considering the block X shown in FIG. 1, this block is surrounded by eight blocks A, B, C, D, E, F, G and H, and the invention makes use of the fact that the lines of PELs of the block X adjacent its four edges will be similar in most cases to the lines of PELs of the blocks B, D, E and G adjacent the same edges. This will not be true if a discontinuity in the image separating areas of differing brightness lies along one of the boundary edges of the block X and a method of taking this into consideration is described below.

From a consideration of FIG. 2, it will be clear that the zero-sequency, or "d.c.", coefficient at the top left-hand corner, which represents the average brightness level of the whole block, is normally conveyed by more bits than the other coefficients. However, if the d.c. coefficient is discarded the amount of data needed to be transmitted to convey the transformed block is reduced by one-eighth, or possibly more depending on the block size, bit allocation and transformation used. The varying detail of the part of the image within a block is conveyed by the other coefficients so that the discarding of the d.c. coefficients does not affect the detail conveyed. However, the discarding of the d.c. coefficient does give rise to the difficulty that the mean brightness levels of the blocks will not automatically have the correct values when the image is reproduced by the inverse transformation and the sharp brightness changes at the edges of the blocks will then be objectionable. It is proposed therefore to adjust the mean levels of the blocks of the reproduced image by comparing the lines of PELs belonging to adjacent blocks on each side of an edge and adjusting the mean levels so as to reduce the differences. If, for example, the d.c. coefficient of the top left-hand block of the image is transmitted, then the mean levels of the adjacent blocks of the reproduced image can be derived from that transmitted d.c. coefficient by choosing mean levels which minimise the mean square value of the brightness differences between the adjacent lines of PELs at the edges between the blocks. In this way it is possible to build up the mean levels of all the blocks by reference to the one block of which the d.c. coefficient was transmitted.

Three alternative methods of building up the mean levels of the blocks over the entire image are possible, and in the following description it is assumed that the mean level of the top left-hand block, (1,1), is either transmitted or is set to zero or some other arbitrary value. In the latter case, then an overall adjustment of the mean levels of all of the blocks by the same amount may be made at the end to bring the overall brightness level to a satisfactory value.

In a first method of building up the mean levels, the levels of the blocks along the top edge of the image are worked out in sequence and then the blocks down the left-hand edge. After that the remainder of the blocks are dealt with individually by reference to the differences along both of the edges which each block shares with two blocks whose mean levels are already calculated. A second way in which the mean levels may be built up is to determine those of the blocks in the first row as described above and then to build up each subsequent row of blocks so as to minimise the edge differences both between the blocks of the row and between them and the blocks of the preceding row. The third way is to combine the blocks into sets of four and adjust the relative levels of the blocks within each set, so as to produce a set of blocks of twice the linear dimension of the original blocks and then to combine the larger blocks into four again and so on until the entire image has been processed.

FIG. 3 shows in block diagrammatic form one example of a transmission system using a method of transmitting an image according to the invention. In FIG. 3, a video signal is applied via a terminal 1 to a sampling circuit 2 from which the sampled analogue values are applied to an analogue/digital converter 3 which preferably includes some form of non-linear quantisation to optimise the encoding to digital form. The digital values are then applied to unit 4 which applies to the values in blocks corresponding to the blocks of the image a two-dimensional unitary transformation such as, for example, a discrete cosine transformation or a Hadamard transformation. At this stage the d.c. coefficients of most of the blocks, or possibly all of them, are not generated. The remaining coefficients are subjected to quantisation and encoding in a unit 5 as required for transmission or storage. The encoding arrangements could be, for example, constructed as described in our co-pending European patent application no. 82303825 (Ser. No. 0072117) corresponding to U.S. Pat. No. 4,504,860 of Nicol et al (which issued on Mar. 12, 1985) the only difference being that the zero sequency coefficients are not calculated; or at least, not transmitted.

The stored or transmitted data is subsequently received by a decoder 6 which decodes the received data to reproduce the quantised values of the coefficients and the coefficients so obtained are grouped together into their blocks and subjected to the inverse transform to that applied by the unit 4; this inverse transform is effected by the unit 7. The decoding process is, thus far, conventional, except that zero is assumed for the missing zero sequency coefficients. Thereafter the missing mean values are calculated as described above, so that discontinuities in the mean values do not appear at the block boundaries. Finally, the regenerated image information is passed to a utilisation means 9 which may, for example, be a display device. Although the stages in the manipulation of the data are shown separately in FIG. 3, they may in fact be effected by means of suitable computing means instead of by units dedicated to the separate operations.

FIG. 4 shows the nine blocks A to H and X of FIG. 1 with the rows of PELs of the block X adjacent to the edges shared with the blocks B and D indicated respectively by references 20 and 21. Adjacent to the row 20, there is in the block B, whose mean value is already known, a row 22 of PELs adjacent to the common edge with the block X. In the block D whose mean value is already known also a row 23 of PELs adjacent to the common edge with the block X is marked. If the values of the PELs in the row 22 are given by y.sub.i for i=1, 2, 3, . . . , n, where n is the number of PELs in the row, and x.sub.i for i=1, 2, 3, . . . , n represents the values of the elements of the row 20 in the block X whose mean level x' is to be calculated, then what is required for the mean value of the block X is a value of x' such that ##EQU1## should be a minimum. It can be shown that to achieve this what is required is that ##EQU2## If the differences between the PELs and the rows 21 and 23 were also to be taken into consideration in calculating the d.c. value of the block X, then the calculation above would be modified by the addition of the values of the PELs in the rows 21 and 23 to the expressions. Thus if u.sub.i and v.sub.i are the edge elements of rows 21 and 23 respectively: ##EQU3##

This corresponds to the first of the three methods outlined above ("row estimation").

In the second method (row estimation), for each row after the first the set of d.c. coefficients is found which minimizes the sum of the square magnitudes of the edge difference vectors between that row and the previous row, and also those between the individual blocks in that row. The method of estimation of the d.c. coefficients in the ith row from those in the i-1th row is now given. It is assumed that the receiver has the following information:

(i) [u(k,l)]i,j: the (i,j)th block of pels having zero d.c. level.

(ii) [v(k,l)].sub.i-1,j : the (i-1,j)th block of pels whose d.c. level have been adjusted according to the estimated d.c. coefficients.

where b.sub.i-1,j is the estimated d.c. coefficient for block (i-1,j) and V=[1/n . . . 1/n].sup.t.

Now we are to estimate the N-dimensional d.c. coefficient vector

from [u(k,l)].sub.i,j and [v(k,l)].sub.i-1,j. We define vertical and horizontal edge difference vectors for block (i,j) as shown in FIG. 2.

The vertical edge difference vector between the jth block and j+1th block in the ith row of blocks is ##EQU4## The horizontal edge difference vector between the (i-1,j)th block and (i,j)th block is ##EQU5## If the pels in the ith row are adjusted by the N d.c coefficients a.sub.1, a.sub.2, a.sub.3, . . . ,a.sub.N, then the edge difference vectors D.sub.1,j and D.sub.2,j are changed to W.sub.1,j and W.sub.2,j respectively:

Therefore, the sum of the squares of the magnitudes of these edge difference vectors becomes ##EQU6## Rewriting in the form of ##EQU7## it can be shown that e is minimum when

where ##EQU8##

The evaluation of the restoration schemes was carried out experimentally using computer simulation. The head and shoulder picture of a girl was first divided into blocks of size n by n. Each block was then transformed using the Walsh/Hadamard transform, and the d.c. coefficient set equal to zero. All blocks were then inverse transformed to return to the picture domain. The three d.c. coefficient restoration schemes were then applied to obtain the restored pictures as well as the sets of estimated d.c. coefficients. These procedures were repeated for block sizes 4.times.4, 8.times.8 and 16.times.16. No coefficient quantization was undertaken.

With a 4.times.4 block size, there were edging effects in all the three restored pictures. Furthermore, the accumulation of error due to each estimation produced impairment effects along the direction of estimation. In the picture restored by element estimation, if a block was very bright or very dark, this brightness or darkness tended to diffuse diagonally from top to bottom right. In the picture restored by row estimation, the diffusion runs vertically from top to bottom and is less severe than that occurring with element estimation. In contrast, the picture restored by the third method (plane estimation) did not exhibit this effect. However, accumulation of estimation errors makes the edging effects more prominent as the block size increases.

With the 8.times.8 block size, pictures restored by element and row estimation still had edging effects but no apparent diffusion effect. Noticeable edging effects still remained in the picture restored using plane estimation. Using a block size 16.times.16, row estimation restored the picture without perceptible error whilst element estimation produced a reasonably good picture. Again, there were noticeable edging effects in the picture restored by plane estimation.

As mentioned above, the calculations assume that no severe luminance discontinuities appear in the image coincident with the edges between the blocks. If such discontinuities exist, they could be detected when the blocks are first formed before the transformation is carried out and an indication could be transmitted with the transform coefficients for a block to indicate that the differences between the PELs along one edge and those of the adjacent block should be ignored because the image has a boundary at that place. A single such transition will not obstruct the "building up" processes discussed above, but two such transitions on a given block may do so in instances where the building up process used (eg the first of the three described) relies upon the boundaries with two adjacent block to evaluate the mean level of the block in question. This situation may be met by providing that the d.c. coefficient be transmitted in respect of the two blocks whose common boundary coincides with the luminance discontinuity; either in every case, or in those cases where a problem is recognised on the basis of an appropriate criterion--eg the occurrence of two discontinuities. The actual criterion used would depend on the degree of sophistication of the building-up process; for example the number of occasions on which transmission of the d.c. term was required would be less if one employed a building-up process which could approach the block in question in two directions.

The correction of differences between PELs along the edges of blocks can also be effected by arranging that all of the blocks overlap the adjacent blocks by one row or column of elements so that the blocks of restored samples after inverse transform has been applied will have samples in common with the adjacent blocks and the relative adjustment of the mean levels of the blocks can be achieved simply because the common samples should have the same values in both blocks. There is no value in using this technique solely to permit d.c. term regeneration, since the redundancy would nullify the advantage obtained in reducing the amount of data needed to be transmitted to convey an image using the present invention; however, if the overlap technique is in any event used to counteract the "blocking" effect of omission of coarse quantisation of the higher sequency coefficients, it provides a ready aid to reconstruction of the omitted d.c. term.

The Hadamard transformation represents something of a special case, since its low sequency coefficients define the mean levels of sub-blocks within the block under consideration. For example, if c.sub.ij are the transform coefficients of a 16.times.16 block of PELs (i, j=1 to 16), ordered as to sequency, and m.sub.p,q are the mean values of four 8.times.8 sub blocks, then c.sub.1,1 (the "d.c." term), c.sub.1,2, c.sub.2,1 and c.sub.1,2 are linear combinations of m.sub.1,1, m.sub.1,2, m.sub.2,1 and m.sub.2,2. In fact ##EQU9## Thus the omission of the first four coefficients of the coefficient array of the larger block has precisely the same effect as the omission of the lowest sequency coefficients of the smaller blocks, i.e. loss of the mean level information for the smaller blocks, and these can be restored on a (smaller) block by (smaller) block basis exactly as described above.

In the transmission of image data in the form of transform coefficients, it has been proposed to use different block sizes depending upon the activity within the image and this technique could be used in conjunction with the present invention.

Although the above description has assumed a single array of samples per image, obviously the same techniques could be applied to the components (RGB or luminance/colour difference) of a colour image.

The technique of the present invention does not necessarily require all the coefficients for all the blocks to be received before the d.c. coefficient restoration process can be initiated. It can also be used with the "slow" inverse transformation, described in our earlier European patent application mentioned above, in which the definition is gradually increased as more coefficients are inverse transformed and added to the previous result. When using the "slow" method the estimates of the d.c. coefficients are also continually updated as the intensity levels at the block boundaries change.