Automatic Focusing Control of a Video Camera for Industrial and Military Purposes

This application is related to the commonly assigned, co-pending U.S. application Ser. No. 07/380,470, filed July 17, 1989, entitled "AUTOMATIC FOCUSING CONTROL OF A VIDEO CAMERA FOR INDUSTRIAL AND MILITARY PURPOSES". The disclosure of this application is incorporated herein by reference.

The present invention relates to a new and improved construction of an apparatus for, and a method of, automatic gain/offset control of a video camera.

The invention thus generally relates to the fields of optics and electronics and, specifically, to the field of image processing techniques. In its more particular aspects the present invention specifically relates to a new and improved construction of an apparatus for, and a method of, automatic gain/offset control of a video camera for automatically optimizing the brightness and the contrast of a video image which is used for controlling the adjustment of a manipulator unit. In particular, the video camera is utilized for controlling the operation of an industrial robot. The video camera also may constitute a FLIR camera, i.e. a forward looking infrared camera and may be connected to an image control unit constituting a so-called video tracker in an automatic target tracking device. An infrared camera which is sensitive to heat radiation, is conventionally designated as a forward looking infrared camera.

Means for optimizing the brightness distribution in images are well-known. Thus, in modern photo cameras, the film exposure is optimized, for example, either by automatically adjusting the aperture at a predetermined exposure or exposure duration or by means of a programmable optically controlled exposure time-aperture combination. In such systems, the required brightness data are obtained either from the mean brightness value of the entire image, the brightness of a small pre-selected section or frame of the image by means of a so-called spot measurement, or from a comparison of a predetermined number of partial image sections or frames. This can be derived from the data sheets of respective photo cameras.

When using cameras employing electronic image recording such as, for example, video cameras, there can not only be optimized the brightness by means of the offset control but also the image contrast may be subjected to an optimizing operation. For this purpose there is varied, in addition to the offset, a second parameter, namely the gain which corresponds to a displacement of the image brightness distribution along the brightness axis. In such case, reference is made to an automatic gain/offset control.

Conventionally, the gain/offset control in video cameras is adjusted to provide an image which is as attractive as possible and as rich in information as possible for the human eye.

Hitherto and, for example, with respect to robotics, the gain/offset control was used in connection with video cameras for the recognition or detection of objects and for controlling related manipulator units. In such uses, the gain/offset control adjustment was optimized, if optimized at all, in accordance with the criterion of good visual recognizability. Thus the image was adapted in an optimum manner with respect to the eye of the machine operator but not with respect to the mode of operation of the control device connected with such video camera for controlling the manipulator unit.

In the same manner the gain/offset control adjustment was carried out in connection with video cameras utilized for target tracking, in accordance with optimum visual recognizability. This means that the video camera like, for example, the aforementioned forward looking infrared camera was controlled in a manner such that no saturation was observed merely within a predetermined image section or frame. Although thereby there was favorably insured an additional visual target monitoring process, however, the image did not constitute in any case an optimum image for automatically tracking the target by means of the target tracking platform.

It will be apparent that the human eye and an electronic image evaluating unit place different requirements on the image quality. The reason therefore is that the physiological process of vision by the human eye is proportional to the logarithm of the light intensity whereas, during electronic image recording, the recorded signal is linearly dependent upon the light intensity in a first approximation.

Furthermore, and by virtue of the integral image perception by the human eye, a structure or object is relatively readily distinguished or discriminated with respect to an associated background even at a poor signal-to-noise ratio or poor contrast Contrary thereto, automatically operating structure or object recognition means can extract the information related to the object or structure from a comparison of adjacent image brightness values only at great expenditure and on the basis of a fairly exceptional or extensively devised algorithm. Consequently, it is highly significant that the brightness of the object to be recognized compares as favorably as possible with the brightness of the background associated with the object to be recognized.

Regarding the automatic gain/offset control of a video camera, it should be noted that it is not intended to generate an image which is as favorable as possible for the human eye. Instead, the image should be optimized with respect to further electronic processing operations particularly in connection with the control of the performance of the manipulator unit. In order to obtain such optimization, it is thus required that the object to be monitored by the video camera can be distinguished or differentiated from the associated background. Such precondition can be satisfied at least for some periods of time by objects which move relative to the associated background.

Therefore with the foregoing in mind it is a primary object of the present invention to provide a new and improved construction of an apparatus, and a method of, automatic gain/offset control of a video camera and which apparatus and method are not afflicted with the drawbacks and limitations of the prior art heretofore discussed.

Another and more specific object of the present invention is directed to the provision of a new and improved construction of an apparatus for, and a method of, automatic gain/offset control of a video camera and which apparatus and method permit providing an image of optimum quality in terms of brightness and contrast.

A further important object of the present invention is directed to a new and improved construction of an apparatus for, and a method of, automatic gain/offset control of a video camera and which apparatus and method permit adjusting the gain/offset control means of a video camera such that an optimum image is provided for controlling the operation of an industrial robot.

Still another significant object of the present invention aims at providing a new and improved construction of an apparatus for, and a method of, automatic gain/offset control of a video camera and which apparatus and method permit adjusting the gain/offset control means of a video camera in a manner such as to provide an optimum image for carrying out a target tracking operation.

Now in order to implement these and still further objects of the invention, which will become more readily apparent as the description proceeds, the apparatus of the present development is manifested, among other things, by the features that, there is provided an image control unit operated according to an image evaluating algorithm. The image control unit is connected with the video camera and generates control signals for adjusting the gain/offset control means of the video camera for automatically recognizing or detecting the image of an object relative to the image of the background associated with the object in an optimum manner in terms of brightness and contrast.

Specifically, the video camera is connected with a manipulator unit for controlling the position of the video camera or, if desired, an object to be monitored by the video camera. In such case, the image control unit is connected to a main computer or computer means which in turn is connected with the manipulator unit through a suitable servo system.

As alluded to above, the invention is not only concerned with the aforementioned apparatus aspects, but also relates to a new and improved method carried out by operating the inventive apparatus. Generally speaking, the inventive method relates to an automatic gain/offset control of a video camera.

To achieve the aforementioned measures, the inventive method, in its more specific aspects, comprises the following steps:

imaging an object and a background associated with the object by means of a video camera;

determining, by means of a gray-scale histogram analysis, the brightness of the image of the object and the brightness of the background associated with the object as well as the contrast between the image of the object and the image of the background associated with the object;

on the basis of the values of an evaluating algorithm, generating control signals by means of an image control unit for automatically adjusting the image of the object and the image of the background associated with such object in terms of brightness and contrast and feeding the control signals to the gain/offset control means of the video camera.

More specifically, the video camera may be employed for controlling the adjustments of a computer-controlled manipulator unit.

The inventive apparatus as well as the inventive method thus rely upon an automatic control which, in turn, relies upon real-time image evaluation, for adjusting the gain/offset control means of a video camera. Such video camera is used, for example, in connection with an industrial robot or may constitute a FLIR camera, i.e. a forward looking infrared camera of a target tracking platform in a manner such that the image control unit provides an optimum image in terms of brightness and contrast under the prevailing imaging conditions and an optimum control of the operation of the manipulator unit constituting or comprising the aforementioned industrial robot or platform, for example, target tracking platform.

The invention will be better understood and objects other than those set forth above will become apparent when consideration is given to the following detailed description thereof. Such description makes reference to the annexed drawings wherein throughout the various figures of the drawings, there have been generally used the same reference characters to denote the same or analogous components and wherein:

FIG. 1 is a schematic block circuit diagram showing a first exemplary embodiment of the inventive apparatus for carrying out a first exemplary embodiment of the inventive method and comprising a video camera, an image control unit and a controllable manipulator unit constituting or comprising an industrial robot;

FIG. 2 is a schematic block circuit diagram showing the main components and connections of the image control unit in the apparatus shown in FIG. 1;

FIG. 3 is a histogram showing the brightness of the image elements or pixels of the image produced by the video camera, which is used in the apparatus shown in FIG. 1, and illustrates the brightness distributions respectively associated with the image of the object and the image of the background associated with such object;

FIG. 4 is a schematic block circuit diagram of a second exemplary embodiment of the inventive apparatus for carrying out a second exemplary embodiment of the inventive method, namely the basic construction of a target tracking device containing a target tracking platform, the video camera and the image control unit as its basic components; and

FIG. 5 is a schematic flow diagram illustrating the data flow in the apparatus shown in FIG. 1 during a gain/offset control adjusting operation.

Describing now the drawings, it is to be understood that only enough of the construction of the apparatus has been shown as needed for those skilled in the art to readily understand the underlying principles and concepts of the present development, while simplifying the showing of the drawings. Turning attention now specifically to FIG. 1 of the drawings, the apparatus illustrated therein by way of example and not limitation in a schematic block circuit diagram will be seen to comprise an automatically controllable manipulator unit 2 comprising, for example, an industrial robot 21 or a fully automated controllable machine tool. A video camera 1 monitors the grasping and manipulation of the object 7 which may constitute, for example, a workpiece 71 which is grasped and manipulated by the industrial robot 21, or a tool 72 of the aforementioned controllable machine tool operating upon a workpiece. Also, the manipulator unit 2 may comprise a target tracking platform 22 for tracking an object 7 which constitutes a target 73, as illustrated in FIG. 4 of the drawings.

The inventive apparatus schematically illustrated in FIG. 1, essentially comprises the aforementioned video camera 1 which images the object 7 and transmits digital electrical signals representative of the image data produced by the video camera 1 to an image control unit 3. In this image control unit 3 the data representative of the image are evaluated and control signals are generated and returned to the video camera 1 for achieving optimum adjustment of gain/offset control means lA of such video camera 1. The processed image data which are generated by the image control unit 3, are received by a main computer or computer means 4 and transformed or converted thereby into adjustment control signals. Such adjustment control signals are translated into a sequence of movements of the industrial robot 21 or the aforementioned fully automated controllable machine tool by means of a servo system 51. Additionally, the main computer or computer means 4 controls the entire course of movement or adjustment of the device and thus, for example, also a platform 22 through a further servo system 52. The platform 22 is mechanically rigidly connected to the video camera 1 and thus permits the video camera 1 to track the object 7. If desired, the image control unit 3 and the main computer or computer means 4 can be assembled to form a structural unit.

The basic construction of the image control unit 3 and its main connections are illustrated in FIG. 2. As shown, the image control unit 3 comprises an image evaluating unit 33, a gain/offset control 34 and an I/0 processor 35.

The electrical analog signals representative of the image data and produced by the video camera 1 are first digitized by means of an A/D converter, i.e. transformed or converted into digit sequences or digital data indicative of the brightness of the individual image elements or pixels in the imaging area of the video camera 1. Subsequently these digital image data are supplied to the image evaluating unit 33 of the image control unit 3. The digitized image data or video signal is interpreted or evaluated by means of the image evaluating unit 33. Such evaluation is based on the so-called gray-scale histogram analysis for distinguishing or discriminating between the image of the object 7 and the image of the background 8 associated with the object 7. The principle of such gray-scale histogram analysis is well known in the art and described, for example, in Chapter 7 of the book entitled "Digital Image Processing" by R. C. Gonzalez and P. Wintz, Addison-Wesley Publishing Company Inc. 1977, and, with reference to a particular imaging process, in U.S. Pat. No. 4,613,269.

As a result of such gray-scale histogram analysis by means of the image evaluating unit 33 of the image control unit 3, there is obtained a weighted brightness 13 associated with the image of the object 7 as well as a weighted brightness 14 of the image of the background 8 associated with the object 7. As already mentioned hereinbefore, the weighted brightness values 13 and 14 result from a measurement of the brightness of each individual image element or pixel in the imaging area or a preselected image section or frame of the video camera 1.

The weighted brightness values 13 and 14 are translated into control signals by means of the gain/offset control 34 of the image control unit 3. The transformation of the brightness values into the desired control signals is carried out in accordance with conventional principles such as known for example, from U.S. Pat. No. 4,204,229, granted May 20, 1980. The thus obtained control signals are fed to the video camera 1, specifically to the gain/offset control means lA thereof through the I/O processor 35 of the image control unit 3. The control signals permit operating the video camera 1 under optimum operating or imaging conditions.

For this purpose, and as illustrated in FIG. 3 of the drawings, an offset control signal is generated by the gain/offset control 34 of the image control unit 3. Such offset control signal, as illustrated in FIG. 3, causes the offset of the gain/offset control means lA to be adjusted such that the weighted brightness 13 of the brightness distribution 11 which is related to the image of the object 7, assumes a predetermined fixed value 10. Furthermore, the gain/offset control 34 of the image control unit 3 generates a gain control signal which causes the gain of the gain/offset control means lA of the video camera 1 to be adjusted such that there is obtained an optimum value for the contrast 15 between the weighted brightness 13 related to the image of the object 7 and the weighted brightness 14 related to the image of the background 8 which is associated with the object 7.

It is decisive for this operation that the offset control signal and the gain control signal are selected such that there is obtained minimum overlap of the brightness distributions 11 and 12 which are respectively associated with the image of the object 7 and the image of the background 8 associated with the object 7, at minimum contrast 15 between the respective weighted brightness values 13 and 14. The gain/offset control means lA provided at the video camera 1 are of conventional construction and their adjustment by means of respective control signals are known in the art, see, for example, the aforementioned U.S. Pat. No. 4,204,229, granted May 20, 1980, and the publication entitled "A Real-Time Video Tracking System" by A. L. Gilbert et al in IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-2, No. 1, pages 47 to 56 January 1980.

The chronological course of the gain/offset control adjusting operation will now be described with reference to FIG. 5. During the gain/offset control adjusting operation, a command generator 61 periodically generates commands or input data for activating the gain/offset control unit 34 in the image control unit 3. The periodic generation of the commands or input data is adapted to the response time of the video camera 1 like, for example, the forward looking infrared camera. In the event that the apparatus further contains the automatic focusing control as described in the initially mentioned and cross-referenced, commonly assigned and copending United States application, the periodic generation of the command or input data is also adapted to the response time of the focusing control. The presence of such command or input data as well as the presence of the image data received by the image evaluating unit 33 of the image control unit 3, is checked at a test point 62. In the absence of such command or input data and image data, there is departed from the gain/offset control cycle.

In the presence of such command or input data and image data, there is checked at a test point 63 whether the weighted brightness 13 of the object image already has the predetermined fixed value 10. If this is the case, the gain/offset control adjusting operation is continued in order to properly adjust the gain of the gain/offset control means 1A of the video camera 1. If the weighted brightness 13 of the object image is different from the predetermined fixed value 10, the offset adjustment is changed at a step 64.

At a next-following test point 65, there is checked whether the change in the offset adjustment effected during the step 64, has resulted in an overflow or underflow on the offset scale. If no such overflow or underflow on the offset scale is observed, there is again departed from the gain/offset control cycle. If an overflow or underflow on the offset scale is observed, the respective maximum value or minimum value is loaded into an offset command register during a step 66.

After the desired offset adjustment has been achieved, the gain/offset control adjusting operation is continued at a test point 67. At this test point 67, there is checked whether the difference between the weighted brightness 13 of the object image and weighted brightness 14 of the background image, i.e. the contrast 15 is in the desired range. If such is the case, there is departed from the gain/offset control cycle.

If the difference between the weighted brightness 13 of the object image and the weighted brightness 14 of the background image is insufficient, there is checked at a test point 68 whether the gain adjustment has reached a maximum value. If the maximum gain adjustment has been reached, there is departed from the gain/offset cycle. If the maximum permissible gain adjustment has not been reached, the gain adjustment is changed during a further step 69.

It should be noted that, when carrying out the aforedescribed gain/offset control adjusting operation, the term "departed from the gain/offset cycle" means that no further tests and readjustments are carried out during the running gain/offset cycle. In the next-following step, therefore, the gain/offset cycle is re-started by re-entering at the test point 62.

The aforedescribed method for optimizing the image produced by the video camera 1 in terms of brightness and contrast and which method is carried out by using the apparatus as illustrated in FIG. 1, can only be utilized, of course, when the object 7, be it the workpiece 71, the tool 72 or the target 73, can be recognized or detected relative to the background 8 on the basis of the aforementioned gray-scale histogram analysis. It should be noted, however, that movable objects 7 frequently move relative to a continuously changing background 8 whereby the recognizability or detectability of the object 7 is insured at least during some periods of time. This is true, for example, in the case of a workpiece 71 during the grasping phase but probably would no longer be insured after such workpiece 71 is automatically installed or mounted at a machine constituting the background 8 with respect to the workpiece 71. Likewise, a target 73 can be clearly recognizable or detectable relative to a clear sky prior to its immersion into the horizon silhouette.

It is the purpose of the inventive apparatus and the inventive method to adjust the imaging parameters of the video camera 1 by means of the gain/offset control means 1A at the time at which the object 7 is well recognized or detected, in a manner such that the recognizability or detectability of the object 7 is also insured with respect to a complex structured or camouflaging background 8. There can thus be achieved the result that the object 7, once it has been recognized or detected, remains clearly visible or recognizable even under adverse imaging conditions, i.e. in the event that the contour of the object 7 is hardly distinguished or differentiated from the contour or contours of the associated background 8.

In spite of the foregoing considerations it is still conceivable that an unambiguous automatic recognition or detection of the object 7 cannot be insured at any moment of time. In the event that the contrast in brightness between the object 7 and the associated background 8, in fact, is insufficient for permitting optimum gain adjustment, the gain adjustment by means of the gain/offset control means 1A is controlled by means of the gain control signal generated by the image control unit 3 in a manner such that the weighted brightness 13 related to the object 7 and the weighted brightness 14 related to the background 8 associated with the object 7 have the greatest possible contrast 15 without one of the two brightness distributions 11 and 12 which are respectively associated with the object 7 and the associated background 8, becoming saturated.

The gain/offset control 34 of the image control unit 3 delivers digital data. Such digital data, as already mentioned hereinbefore, are converted or transformed by means of the I/O processor 35 into a format acceptable for the gain/offset control means lA of the video camera 1 and supplied to such gain/offset control means 1A.

In principle, it is also conceivable that the image evaluation is not effected in digital manner in the image control unit 3 via the digital image evaluating unit 33 conjointly with the digitally operating gain/offset control 34 and the I/O processor 35 series-connected therewith but is directly carried out in analog format. This possibility could be advantageous in the event that the automatic gain/offset control unit 34 is integrated with the video camera 1.

In addition to the weighted brightness 13 related to the object 7 and the weighted brightness 14 related to the background 8 associated with the object 7, the image evaluating unit 33 of the image control unit 3 communicates to the I/O processor 35 of the image control unit 3 data indicative of the position of the object image within the imaging area of the video camera 1. Such position data are supplied to the main computer or computer means 4 which causes the servo system 51 to re-adjust the position of the object 7 by means of the industrial robot 21. In addition thereto it may prove necessary or desirable for the video camera 1 to follow or track the object 7. For this purpose, related control signals could be generated and supplied directly by the main computer or computer means 4 or indirectly through a further or additional servo system 52 controlling the positional adjustment of the platform 22. The video camera 1 is mechanically rigidly connected to the platform 22 and the adjustment of the platform 22 under the action of the control signals permits the video camera to track the object 7.

A monitor 6 is connected to the image control unit 3 and receives from the image control unit 3 via an A/D converter the signals required for reproducing the image, which is produced by the video camera 1, on the screen of the monitor 6. The operation of the inventive apparatus thus can be visually monitored by observing the reproduced image at the screen of the monitor 6. It should be noted, however, that the monitor 6 cannot offer an image which constitutes the best possible image for the operator because, as already explained hereinbefore, the image optimization in terms of brightness and contrast is not optimumly adapted to the eye of the person viewing the monitor 6 but in fact, is optimumly adapted for the control of the manipulator unit 2.

A second exemplary embodiment of the inventive apparatus for carrying out a second exemplary embodiment of the inventive method is illustrated in FIG. 4 showing a target tracking apparatus. Such target tracking apparatus comprises an adjustable platform 22 which can be adjusted with respect to azimuth and elevation, the video camera 1 constituting, for example, a forward looking infrared camera which is mechanically rigidly connected with the adjustable platform 22, the servo system 52 for controlling the positional adjustment of the platform 22, and the aforedescribed image control unit 3 constituting a so-called video tracker. The image control unit 3 or video tracker monitors the operation of the video camera 1 as well as, with the aid of the main computer or computer means 4, the positional adjustment of the adjustable platform 22 on the basis of the image of the object 7, i.e. the target 73 and the related signals which are produced by means of the video camera 1. The monitor 6 is connected with the image control unit 3 and permits the operator to visually track the target 73.

While there are shown and described present preferred embodiments of the invention, it is to be distinctly understood that the invention is not limited thereto, but may be otherwise variously embodied and practiced within the scope of the following claims. ACCORDINGLY,