Method for Color Control in the Production Process

The present invention relates to a method for color control in the production process in which a sample and a reference of the sample are recorded pixel-by-pixel with the aid of the same arrangement consisting of one or more color sensitive image-forming sensors, in particular color matrix cameras, color line cameras or color sensitive point scanners.

Continuous control and monitoring of color fidelity of products is an important stage in quality assurance. In addition to manual qualitative monitoring of color fidelity by means of a visual comparison of the product colors with control cards or reference samples, there exists a well-developed color measurement technology for the purposes of quantitative determination of the color and/or color deviation of a sample in comparison to a reference.

Essentially, two classes of methods are used:

1. methods based on evaluation of the color spectrum of the product,

2. methods based on measurement of the primary colors of the colored product.

In the case of both methods, the structure of the surface, such as curvature in the space, applied textures, grooves, scratches or similar features, has a disruptive effect on the measurement results. The methods known to the art therefore also require that the color measurement is carried out on an essentially diffusely reflecting and even sample. Uneven spots, shining spots and spots in shadow must be absolutely avoided. It must also be ensured that only the color to be measured and no other color is present in the measurement window of the color measuring device, e.g. colored markings, imprints, labels or similar features. Since numerous products, e.g. extruded plastics for the production of floor coverings, ready lacquered bodywork parts, foodstuffs such as baked products, vegetables, printed products, etc., do not meet these prerequisites, the user must be satisfied with coloring small flat test models along with the products and measuring them later in the laboratory. It is therefore, for example, customary in the production of linoleum flooring to take samples from the extruded and colored floor covering material, to roll this onto flat cardboard surfaces, so that an even and diffuse surface results and to measure this sample in the color laboratory. This procedure is time consuming, manual and not directly part of the process; when the results are available, not inconsiderable batches have possibly already been produced with inadequate color fidelity.

In a method of the above-mentioned type known to the art from the document U.S. Pat. No. 4,812,904, in which a color video camera is used as an image-forming sensor, the color value signals delivered by the color video camera in the recording of the sample and in the recording of the reference are digitised pixel-by-pixel and stored. For the purpose of color control, the stored color vector components of the sample and the reference are statistically compared. This is done, for example, by forming the mean values of the color vector components for the sample and for the reference and determining whether the mean values of the color vector components of the sample deviate by more than a pre-set tolerance threshold from the mean values of the color vector components of the reference. The reduction of the comparison to mean values and to exceeding the pre-set tolerance thresholds results, however, only in a yes/no statement as to whether or not there is color deviation; on the other hand, no measure is obtained for the magnitude of the color deviation.

It is an object of the present invention to create a method of the kind mentioned at the beginning, which is capable of detecting the color deviation of a product quickly, continuously and to a large extent automatically and providing one or more color deviation measures for the magnitude of color deviation.

This object is solved according to a first embodiment of the invention by the following method steps:

the color values of all pixels or a subset of the pixels in the whole or in individual parts of the image field of the reference of the or each sensor recording the reference are put in the form of one or more multidimensional histograms of the color vector components and stored in this form;

in the same way, the color values of all or a subset of the pixels in the corresponding image field or in the corresponding parts of the image field of the or each sensor recording the sample are put in the form of one or more multidimensional histograms of the color vector components;

from the comparison of the histograms of the sample and the histograms of the reference, one or more color deviation measures are determined for each image field to be controlled or for each part to be controlled of an image field.

A second embodiment of the invention consists in that the color values of all pixels or a subset of the pixels in the whole or individual parts of the image field of the or each sensor are stored during the recording of the sample and during the recording of the reference and that from the comparison of the color vectors of the reference with those of the sample, a scalar and/or vectorial color deviation measure is determined.

The method will be explained, by way of example, but not restrictively, by means of color control of lacquered automobile bodywork. For this purpose, reference is made to the following illustrations:

FIG. 1 shows the storage of the color vector components of a picture detail of a bodywork component, which contains both shining spots and shadow zones, using the IHS color space;

FIG. 2 shows the frequency histogram of the color vector comp onents, reduced to the components I and H for a more comprehensible presentation and the elimination of shining and shadow zones by generating a threshold value;

FIG. 3 illustrates color control from the deviation of the histogram of the color vector components of the reference from the histogram of the sample, for clarity reduced to the "hue" component;

FIG. 4 illustrates the obtaining of smoothed histograms by jointly incrementing neighboring values;

FIG. 5 illustrates the reduction of the histograms by generating a threshold value to a binary flagging of those color vector components which exceed this threshold.

The method is explained in the example of a color control paint shop for automobile bodies. Reference is made to FIG. 1. The body -1- is illuminated by one or more than one illumination devices -3- as evenly as possible and the image recorded by one or more than one color cameras -2-. The electrical signals are transmitted to a computer -4-, in which the method steps are carried out according to the present invention. It is assumed, for example, that the signals from the color camera are present in the IHS color space (I=Intensity; H=Hue; S=Saturation) or are transformed into this color space. This is not a compulsory prerequirement of the inventive idea but simplifies explanation of the underlying principle.

The area recorded by the camera is neither diffuse nor smooth nor even, as is a prerequisite in the color measurement methods known to the art. Thus, shining spots are formed at folding points, or shadow zones arise on pockets, door gaps and door handles. According to the present invention it is not provided that the surface observed is integrally recorded and from this either a set of 3 primary colors (RGB,.Lab, Luv or similar) or a spectrum is determined, but instead a three-dimensional histogram of the color vector components of all pixels in the picture detail recorded by the color camera is formed. Detail -5- demonstrates, for clarification purposes, how the color vectors are distributed in the selected IHS color space.

As a consequence of the unevenly diffuse surface, as a consequence of system noise in the measurement arrangement and as a consequence of the local color fluctuations caused by the process, the measured color vectors spread and form a "cloud" of color vectors in the color space. For better recognition, the projection of this cloud onto the HS plane is hatched. Shining spots and shadow spots can be recognised as outliers from this color cloud: they correspond to color vectors with low saturation and either high intensity (in the case of shining spots) or low intensity (in the case of shadow spots). It is therefore possible by means of a simple threshold value operation to recognise such color vectors automatically and exclude them from the color control. Likewise according to the present invention, outliers can be recognised automatically along the hue axis, e.g. as a consequence of inclusion of an unwanted zone of another color, a marking, a label or similar and excluded from the color control.

This automatic recognition of non-valid areas is possible only by means of the use according to the present invention of image-forming color sensors and is in principle not possible with the currently used color measurement devices which integrate over the measurement field.

Colour control according to the present invention is performed by means of the multidimensional histogram of the color vector components. Whilst in FIG. 1 in detail -5- a number of different color vectors are presented as a cloud of points in the IHS area, it cannot be shown in such a graph how frequently a particular color vector has occurred in the picture detail under consideration. With the customary resolutions of color matrix cameras with 400 000 color picture elements (color pixels) it is naturally a very frequent occurrence that the same color vector is encountered several times. In the case of an ideally pure color and an ideal, noise-free measurement system, the same color vector would have to be encountered 400 000 times and therefore shrink the cloud to a point.

In order to represent the frequency of the IHS components, a 3+1=4-dimensional presentation is required, which can no longer be sketched as a black and white diagram. For the purposes of clarification of the inventive idea, we will therefore present the histograms of the color components not as a 3-dimensional histogram but simplified as a 2-dimensional histogram of the components H and S or as a 1-dimensional histogram of the component H and explain with the aid of this simplified representation.

FIG. 2 shows the HS histogram of the example from FIG. 1. f(H,S) here stands for the compound frequency, i.e. the frequency with which a particular value H and a particular value S occur. The detail -1- in FIG. 2 shows this 2-dimensional frequency distribution. It can be recognised from this that certain color components occur significantly more frequently than others. The value range of the H and S values occuring can also be seen.

In a similar way, the three-dimensional histogram f(I,H,S) must be pictured as a presentation of the frequencies with which particular I, H and S combinations occur. Even if such a histogram can no longer be sketched graphically, it can nevertheless be realized in a calculated form as a three-dimensional field.

The recording according to the present invention of the histogram f(I,H,S) of the color pixels recorded in the picture detail under consideration enables, in addition to the consideration of the value range of the I, H and S components occurring, a further practically important automatic sorting of uncertain color measurement values: all color values which occur too rarely can be considered as disturbances and automatically excluded from further control. These are, for example, all color pixels, which are located on the periphery of an area with a uniform color. Colour values on such a "color edge" are always uncertain, since they are found in a transitional area between two colors. By introducing a further threshold -5-, all such color vectors can be masked whose the frequency is too low for them to be considered as a significant color.

The use of an image-forming color sensor with very many color pixels and the production of a multidimensional, generally three dimensional histogram of the color vector components therefore enables automatic masking of:

a) Shining spots and shadow zones of such color vectors, the components of which lie outside a given, permissible value range,

b) uncertain color vectors, which occur too rarely, to be considered significant.

Since, by the use of image-forming color cameras, color control, even after masking non-valid and uncertain color values, is still carried out over, in comparison with the traditional color measurement point device, an extremely high number of separate color pixels and as a consequence of the statistics underlying the histogram, despite the relatively poor signal/noise ratio of the individual color pixels, it is still possible to achieve excellently stable histograms.

The color control is carried out according to the present invention by comparing the multidimensional histogram f.sub.ref (I,H,S) of the reference modified accordingly via the formation of threshold values with the histogram f.sub.samp (I,H,S) of the sample as illustrated in FIG. 3.

For a better understanding, we have sketched here only the 1-dimensional histogram f(H). The detail -1- in FIG. 3 shows the graph of the learned histogram of the reference, the detail -2- shows the corresponding histogram of the sample which differs from the reference by a slight shift in the hue to GREEN. The differential .DELTA.f of the two histograms is used for example as a comparative measure. This differential function can be condensed to a scalar value, e.g. to the normalized or non-normalized area underneath the graph .DELTA.f. It is also reasonable to make other statements condensed to scalar terms, e.g. the position of the center of gravity or higher moments of plane area of the differential histogram .DELTA.f.

Besides the pure differential, weighted differentials, differentials of the squares and similar measures of distance may also be used, as are known to an expert in pattern recognition and classification. Comparative measures adapted to the individual task in question can therefore be developed in the same system structure.

FIG. 3 illustrates the inventive idea with the example of the histogram f(H) reduced to a single dimension. Naturally, the differential histogram of the three dimensional histograms f.sub.ref (I,H,S) and f.sub.samp (I,H,S) is generally formed and a corresponding scalar distance measure obtained from this. These scalar distance measures can indicate the total deviation between the color of the reference and the color of the sample in a normalized percentage scale and present data which is easily understandable even for untrained operating personnel. This measure can obviously also be used electronically as an actuating signal for a quality sort.

It is obvious that instead of distance measure, an output can also be given of the complementary quantity, the coincidence measure, as an indication of color fidelity.

The scalar interval measures do not provide any conclusion as to the type of color shift, as it is given in the traditional color measurement devices as .DELTA.f(L,a,b) values.

It is therefore a further inventive idea to condense the three deviation histograms .DELTA.f'(I), .DELTA.f'(H) and .DELTA.f'(S) into one deviation vector .DELTA.(I,H,S), from which the user can determine the form of color deviation. This deviation vector can be converted, with the conversion methods known to the art from color theory, into customary differential measures, e.g. the .DELTA.(L,a,b) values referred to, and displayed. In this way, the values determined with the color control according to the present invention can be presented and interpreted in the usual manner.

The determination of a three dimensional histogram is preferably effected by incrementing a three dimensional accumulation field. Each color pixel from the reference picture detail provides an address pointer after transformation into the IHS color space, which address pointer results from combining the three digitised components of its color vector. This address pointer points to an accumulation field set to zero at the start of the learning phase; each addressed memory cell is incremented. If the individual color vector components are each digitised into 8 bits, this table has a magnitude of 2.sup.24 *N bits, with N>=ld Zmax. Zmax here is the maximum number of possible equal color vectors, i.e. in an image sensor with 400 000 color pixels, this number is 400 000. This kind of accumulation in a three-dimensional field generates the absolute frequency function, which function is converted into a relative frequency function, the actual histogram, by division by the number of pixels used. Despite the generally very numerous color pixels, this three dimensional histogram will contain numerous blanks, because it is very improbable that all IHS combinations actually occur within the color range given.

It is therefore a further inventive idea to close these blanks by means of a three-dimensional smoothing. This occurs according to the present invention by means of a three-dimensional convolution with a likewise three-dimensional filter core which, for example, corresponds to a three-dimensional Gaussian function. Another useful way of closing the blanks involves three-dimensional morphological operations such as dilatation or closing. Both methods are known to the expert in digital image processing and do not need to be explained further. Reference is made for this purpose to the relevant literature, e.g. Haberacker: Digitale Bildverabeitung ›Digital image processing!, Hanser Verlag.

A further inventive idea is to close the only locally occupied histogram by means of a method which corresponds to the human principle of "generalisation". The individual color pixels encountered in the training phase provide color vectors which are taken as representatives of a stochastic process. The remaining blanks, as shown in FIG. 4, detail -1-, are caused purely by chance. The "generalisation" means that the fact that precisely an observed field is occupied in the accumulation field and that the neighboring fields are empty, has no significance in the sense of color deviation, but is based only on the random sample being too small. It is therefore sensible to close the neighboring spots, "neighboring" to be understood in the sense of the geometric, e.g. Euclidean distance of the color vectors. This generalisation is achieved according to the present invention by the fact that not only the memory space addressed by the IHS address pointer is incremented but also its geometric neighbors, however with a decaying proportion with increasing distance. This decay function k(d) -2- ensures that closed histograms -3- result, which no longer possess any random blanks.

It is a further inventive idea that this generalisation function may be different for the three components I,H,S. If it is desired to carry out a color control relatively independently of the intensity I, because e.g. a drift of the intensity is of less importance to the quality than a drift of the hue and of the saturation, a decay function k(d) will be selected, which decays more slowly along the I components than along the H and S components. As a result of this, the histogram is enlarged in the I direction and therefore a drift of the intensity will enter less into the differential histogram .DELTA.f than a drift of H and S. The same inventive idea can also be applied to the smoothing and morphological operators, by these operators receiving a correspondingly rotationally asymmetric core.

The method for color control according to the present invention can be simplified further by not comparing the entire histograms f'(I,H,S) of the reference and of the sample, but instead binarising these histograms by means of a threshold value operation. FIG. 5 shows how the histogram is compressed to a class affiliation function. All values of the histogram field which exceed the threshold are occupied with a code which contains the information: "This color vector belongs to the learned reference". A binary table results from this both for the reference and the sample. Scalar measures can be determined as a comparative measure of both histograms, such as the number of different storage locations, or vectorial measures such as the number of different storage locations determined separately for each component. These operations are pure counting operations. They can be carried out according to the present invention in video real time, by producing both the table memory and also the corresponding counting mechanism with digital modules such as static memories and counters.

A coincidence measure between the reference and sample using such binarised histograms can be deduced particularly simply, by the color vectors of the relevant pixels of the sample addressing the histogram field of the reference and being read each time, as to whether this memory location is occupied with a flag "belongs to the color reference". The sum of all memory locations read which display this flag is a measure of the color coincidence between the reference and the sample.

It is a further inventive idea not to restrict the learning of the reference to the deduction of the histogram of the pixels of interest. If, for example, undesired parts with a color not relevant for the color control, are to be found in the image field, e.g. labels, markings, stickers, screw holes, etc., these are eliminated by the fact that the pixels of these areas do not increment but decrement the histogram accumulator. It is obviously necessary for this purpose that these areas are either detected automatically (e.g. by means the value range of the color components, or by means of their form and position in the image field with methods of pattern recognition) or that they are marked interactively. By decrementing, the objective is achieved that in the critical peripheral zones between the color space to be controlled and the adjacent irrelevant area a steeper drop in the histogram results and therefore fewer blanks arise in this uncertain area.

In the example of color control of automobile bodywork, the color control according to the present invention is performed with a plurality of color cameras simultaneously, and the taking of the pictures is then advantageously triggered by means of an external trigger device when the parts of interest are in the image field. According to the present invention, these parts can also be found automatically with methods of color processing and pattern recognition known to the art. This is a very important option for measurement in production, since as a rule, the products cannot be guided very precisely past the cameras. This form of automatic determination and follow-up of control zones is not possible with traditional non-image-forming color measurement devices.

The image taking is performed according to the present invention in the case of samples in motion, by means of stroboscopic illumination and/or shortened integration time of the image sensor.