Real-Time Automated Scene Display for Infrared Cameras

The present invention relates generally to infrared (IR) cameras, and more specifically the invention pertains to a histogram projection system which automatically optimizes, tracks changes in luminance and adjusts in real time the display of wide dynamic range imagery from IR cameras.

The task of providing a histogram projection system which automatically tracks changes in luminance to adjust the display of wide dynamic range IR imagery is alleviated, to some extent, by the systems disclosed in the following U.S. patents, the disclosures of which are incorporated herein by reference:

U.S. Pat. No. 4,656,665 issued to Pennebaker;

U.S. Pat. No. 4,670,653 issued to McConkel;

U.S. Pat. No. 4,719,350 issued to Alm;

U.S. Pat. No. 4,741,046 issued to Matsunawa; and

U.S. Pat. No. 4,856,528 issued to Yang.

The patent to Pennebaker describes a system for thresholding images to reduce the information content using histogram analysis. The patent to McConkle teaches an infrared detection and imaging system. The patent to Alm describes a radiation imaging enhancement system employing a feedback loop to obtain an average value detection signal and a reference signal. The patent to Matsunawa teaches a histograming method discriminating picture wherein the picture is binary coded. The patent to Yang teaches a histograming method in a CT image in medical applications.

It is common for IR cameras to rely on manual gain and offset adjustment to obtain a real-time mapping of IR signals to the display values. The manual adjustment gives an excellent view in general of the useful IR information in a scene, but it requires frequent readjustment as the camera is panned or if the IR luminance changes (sun goes behind a cloud for example). Moreover as a linear process, the manual offset/gain expends, so to speak, useful dynamic range on empty levels within the histogram or information range of the IR levels. Two alternative candidates, which we have considered, for implementing automatic adjustment are direct scaling and histogram equalization.

Direct scaling (also called "black level and gain") requires that the histogram of each frame be analyzed to set a "black" and "white" level. The optimum way to do this is scene and application dependent and requires either a user interactive parameter in the implementation or a plethora of choices. In either case, more complexity is involved than with the new algorithm described below. Further, as with the manual adjustment to which it is closely analogous, one is still faced with the disadvantage cited above for a linear process.

The familiar technique, described in text books, of "histogram equalization" assigns dynamic range to each occupied level in proportion to how many pixels are at that level. Extensive simulations in software on actual imagery: indoor and outdoor; day and night, show that due to the nature of IR histograms, histogram equalization amplifies shot noise and loses gray scale resolution for smaller (usually hotter) objects in the scene. The purpose of the present invention is to avoid this pitfall.

The present invention includes an infrared signal processing system which uses an infrared camera, a microprocessor, and an 8 bit monitor to display the IR scenes.

The dynamic range of a signal such as an image is a measure of how many increments or steps of meaningful information are contained in the signal with noise considered. Commercially available IR cameras output IR data signals with 4095 (12 bits) levels of intensity, but commercially available monitors only display signals with 256 levels of intensity (8 bits). The function of the microprocessor is therefore twofold. First, it maps the 12 bit levels of IR data signals into commensurate 8 bit levels for the display monitor. Mapping a mathematical correlation of the 12 bit signals into an 8 bit scale is accomplished as described below.

The second and more interesting process performed by the microprocessor adds a programmable flexibility to the 12 bit to 8 bit mapping by means of an undersampling feature. The first step in using histogram projection is to determine which of the possible raw signal levels from 0 to 4095 are in fact present (occupied) in the image. If one uses less than all the pixels (undersample) in determining these occupancy statistics, say every 2nd, 4th, 8th etc pixel, one can gradually shift the allocated dynamic range on the 8 bit displacy to the more densely occupied pizel levels, usually the background. In an IR scene, the background is usually at ambient temperature such as ground or foliage and creates the densely occupied levels. Smaller objects in the scene if cooler or warmer that the background create the sparsely occupied pixel levels. If these smaller objects are of interest, maximum display dynamic range can be assigned to them by low undersampling, while if one is more interested in background details, one can assign more dynamic range to these elements of the scene by a higher degree of undersampling. Hence the algorithm can easily adapt to the imagery.

It is an object of the present invention to provide an IR data processing system with adjustable allocation of dynamic range.

It is another object of the present invention to provide an IR signal processing system which automatically maps 12 bit IR data signals into 8 bit levels for display.

These objects together with other objects, features and advantages of the invention will become more readily apparent from the following detailed description when taken in conjunction with the accompanying drawings wherein like elements are given like reference numerals throughout.

FIG. 1 is an illustration of the infrared signal processing system elements used in the present invention;

FIG. 2 is an illustration of the mapping process that converts 12 bit data signals into scaled 8 bit display signals;

FIG. 3 is an example of mapping of 12 bit data signals into 8 bit display signals;

FIG. 4 is a block diagram of the flowchart of the histogram projection system used in the present invention;

FIG. 5 is an electrical schematic of the hardware which can perform the process displayed in FIG. 4;

FIG. 6 is a block diagram of an IR signal processing system which presents mapped and projected IR data from the camera to the display;

FIG. 7 is a chart which displays the characteristics in histogram form of 12 bit data signals produced by IR cameras; and

FIG. 8 is a chart of the display histogram after processing by the present invention.

The present invention includes a histogram projection system for mapping the histogram of "raw" scene data of 12 bit IR cameras to produce a display scene data stream which automatically optimizes and adjusts changes in luminance to display a wide dynamic range of IR imagery. The principle of the present invention in its purest form is to assign the display dynamic range of an IR scene equally to each occupied level regardless of how many pixels are at that level. This differs from the prior art histogram equalization, approach, which allocates display dynamic range according to the number of pixels at each level. The present invention also replaces the manual gain and offset adjustment procedures used with IR cameras to provide automatic optimized mapping of IR display signals.

IR camera systems produce IR data signals with 12 bit levels of illumination. This means each pixel intensity ranges between 0-4095 levels (since 2.sup.12 =4096). Typical displays used are 8 bit devices, in which each pixel intensity ranges between 0-255 levels (since 2.sup.8 =256).

The present invention provides automatic mapping of 12 bit IR data signals into 8 bit IR display values with a contrast adaptation feature which results from using two separate concepts: mapping and under sampling. Mapping is simply a conversion which allows 12 bit IR data to be converted into an 8 bit representation. Undersampling involves dropping the sparsely occupied pixel levels in assigning the display dynamic range. Undersampling causes an increase in the dynamic range assigned to the background and tends to enhance the contrast of the background regions in a controlled manner. The best features of mapping and undersampling yield an optimum display of objects of interest in an IR scene. The reader's attention is now directed towards FIG. 1, which is a block diagram of an IR histogram projection system which uses the present invention. In FIG. 1, an image 2 is focused through a lens 6 into a 12 bit IR camera 100. This camera can be any commercially available system such as one produced by the David Sarnoff Research Labs of New Jersey. The analog electrical output of the camera 100 is converted into a 12 bit digital signal by the interface 110 for the computer 120. The computer 120 performs a three step process to convert the 12 bit raw data into an 8 bit mapped display signal for the display. First, it determines a binary histogram by counting the occupied levels of the raw data. Second, it maps the 12 bit digital signal from the IR camera into an 8 bit display signal useable by the display 150, and writes a look-up table. Third, the raw data is mapped into the actual display values using the look-up table to produce the display signal for the display.

The reader's attention is now directed towards FIG. 2, which is a schematic representation of how the new algorithm, named, histogram projection (HP) maps the histogram of "raw" scene data (12-bits in present IR cameras) into a resulting histogram of the displayed scene (8-bits for high quality monitors). The procedure simply requires computing the binary histogram of the image (or each frame in live imagery), i.e. which of the possible levels from 0 to 4095 (12-bits) are occupied by at least one pixel. To display the image, these occupied levels only are mapped or projected into the 256 available 8-bit display levels. An example of the mapping from 12 to 8-bits is shown in FIG. 3 for the case where 1000 levels out of a possible 4096 are occupied.

To implement the HP algorithm in software, one computes the binary histogram of the image and converts the result to a function, B[i], where for 12-bit images i goes from 0 to 4095. B[i]=1 for occupied states and=0 otherwise. Each occupied level is then given an order number n which goes from 1 to N with N being the total number of occupied levels. A pixel on the 12-bit scale at occupied level i with order number n is displayed at the gray level=[255.times.n/N] on the 8-bit display scale, where the brackets denote the next lower integer.

Extensive tests have been performed on image frames taken with a 240 by 160 PtSi array Schottky barrier IR camera. These images included day and night, indoor and outdoor scenes and clearly demonstrate the superiority of HP to other possible alternatives for replacing manual adjustment, such as histogram equalization.

As illustrated in FIG. 2, the implementation of the HP algorithm is performed in three parts: 1) determine the binary histogram 210, 2) perform the "projection" operation, namely the mapping from 12 to 8-bits, and write the transformation to a look-up table 220 and 3) map the raw data to the display through the look-up table 230. The entire operation occurs over a two frame duration (1/30 of a second). As the pixel data of a frame is fed into the processor, it is both stored in a frame buffer and used to determine the binary histogram. The processor ceases to compile the histogram after 3/4 of the frame has been read in and begins to perform the projection mapping. The projection or mapping step is complete by the end of the first frame and is used to update the look-up table. During the second frame, data is retrieved from the frame buffer, passed through the look-up table and is displayed (simultaneously the operations just described are also being performed).

As indicated above, the projection mapping is naturally expressed with integer division. However, instead of performing many time consuming divisions, one can use successive subtractions. The process is further expedited by taking advantage of the fact that we need only to calculate a display gray level for each value of n from 1 to N. Hence we can increment n from 0 to N performing a total of (N+256) subtractions.

A flow chart is shown in FIG. 4; a block diagram of the hardware corresponding to this flow chart is given in FIG. 5; and a block diagram of the hardware (including the two RAM look-up tables) which displays the data stored in the frame buffer, is given in FIG. 6. A detailed description of flow of the implementation is given next.

With the start of a new frame, the gray level of each pixel is used as the address to a 4096 by 1 RAM which stores the binary histogram. The RAM is read to determine whether the gray level has been previously identified as an occupied level. If not, then the occupancy counter is incremented. In either case, a one is written to the binary histogram RAM. After a predetermined number of pixel values have been read in, the function of the processor switches over to the projection or mapping step, at which time the number of occupied levels N is stored in the occupancy counter.

In the first steps of projection mapping, the display counter is reset; the gray level counter is reset; and the accumulator is cleared. The gray level counter is used to address the binary histogram RAM. If the data in the RAM is low, i.e. the level is unoccupied, the value stored in the display level counter is written into one of the display RAMs and the gray level counter is incremented. If the RAM data is high, then 256 is added to the value stored in the accumulator latch and the sum is stored in the accumulator. After the data is latched in the accumulator, the number of occupied levels is subtracted from this value. If the result is positive, the new value is latched by the accumulator and the display level counter is incremented and another subtraction of the number of occupied levels is performed. Each time the quantity, (accumulator value - N), is positive, the display level counter is incremented and the resulting new value is latched by the accumulator. Once the resulting value is negative, the process is suspended; the value of the display level counter is written to one of the display RAMs; the gray level counter is incremented and the process resumes. (The reader may find it helpful to trace the specific cases of 100 and 400 occupied levels through the above mapping process. Note that, in essence, the required integer divisions for the 12 to 8-bit mapping are being performed by means of successive subtractions.)

During the frame while the above operations are performed, the data is also being stored in a frame buffer. During the subsequent frame, this pixel data is read from the frame buffer and used as an address for the second display RAM. (Two display RAMs are used so that while one is being written the other can be read). The mapping from the 12-bit pixel data to the 8-bit display value is read from this display RAM. The 8-bit value is then D/A converted to an analog signal and formatted for RS-170 output.

The main advantage of the invention lies in the automatic tracking and optimization of the scene information which replaces the need for constant and laborious updating of the manual offset and gain adjustment as the scene changes or the camera is panned. This is of great value in actual field operation. A secondary advantage is that for some wide dynamic range scenes, all the scene details from the darkest to the brightest regions are better seen at the same time on the automated image than on the manually adjusted image. This reflects both the subtle eye-guided skill required to "tune" the image properly and the linear nature of the tuning which wastes, so to speak, dynamic range on empty levels within the histogram.

The advantage of the HP algorithm for automatic display of IR scenes as compared to the main alternative candidate, histogram equalization, has already been described above. To reiterate, two unfortunate tendencies of histogram equalization are avoided by the new algorithm: the amplification of shot noise through exaggerated contrast at the background levels, and the loss of image detail for smaller objects in the scene.

A final feature and potential advantage of the new algorithm lies in the simple manner that one can gradually and automatically increase the dynamic range assigned to the majority pixel levels (typically the background such as foliage or ground) by incorporating an undersampling switch. In its basic embodiment, the HP algorithm entails sampling each pixel in the image to determine the binary histogram. If instead, one undersamples (for example if one uses every second, or fourth, or eighth, etc., pixel), one gradually increases the dynamic range assigned in the display to majority pixel levels. FIGS. 7 and 8 illustrate the effects of such subsampling. FIG. 7 shows the raw or 12-bit histogram of the image. FIG. 8 shows the display histograms after processing with the HP algorithm with every pixel sampled and with every eighth pixel sampled. Note that the main peak is assigned greater range in the under sampled case. Further software simulations indicate that for scenes with very wide complex histograms, some degree of undersampling every 4th, 8th or 16th) optimizes the display. This could make a virtue out of necessity as the throughput demands of full sampling become problematic with large arrays (512 by 512 or larger) and faster frame rates. The software code for the algorithm is presented below in Table 1.

Many variations in implementation are conceivable such as updating the binary histogram every other frame in displaying each frame. Further, one could implement with a undersampling switch as described above to allow the operator to adopt to the requirements of the scene or application while still retaining the automatic adjustment which is the crux of the invention.

While the invention has been described in its presently preferred embodiment it is understood that the words which have been used are words of description rather than words of limitation and that changes within the purview of the appended claims may be made without departing from the scope and spirit of the invention in its broader aspects.