Solid State Apparatus for Imaging With Improved Resolution

In the past, images (pictures) were taken with cameras utilizing film. In recent years, pictures have also been made by using solid state detectors which are electrically sensitive to various wave bands of energy received through an optical system. The electrical charge from each detector becomes an electronic signal which is eventually changed into the picture element (pixel) of an image. Existing solid state imaging systems which record more than one wave band, depend on the registration of the analogous picture elements of the different wave bands to record and integrate the response from the different wave bands. Such an apparatus is known as a multispectral remote sensor.

Such solid state systems have found wide use in airplane, satellite and space vehicle imaging because the data can be recorded in digital form and transmitted via wireless in near real time.

It is necessary to define several terms:

Detector refers to the smallest solid state element in the apparatus for sensing a particular wave length of electromagnetic energy arriving from the scene. The detector response in turn is processed into an electronic signal which is known as the picture element (pixel) of the resulting image.

Registration is the ability of the solid state image apparatus to merge different sets of electromagnetic energy that is sensed by the detector elements, to form a clear picture.

Resolution is a measure of the information (fineness of detail) that can be displayed in the resulting image.

Systematic offset, as used herein, means that each detector element of a given array of detectors records a signal from the scene which is offset from a signal recorded by an analagous detector in the two cartesian (cardinal) directions by a fractional amount the denominator of which is equal to the number of wave bands utilized. For example, if three wave bands are utilized (one for each array of detectors) the offset will be one third of a detector element which results in a number of subpicture elements equal to the square of the number of wave bands, in this case nine.

As stated, there were many attempts to improve resolution of such solid state imaging apparatus. But there has not been much of an attempt to improve resolution by the multiple use of the same size picture element in a given apparatus. The two exceptions that are noted is, for instance, the solid state camera that was recently developed by RCA which utilizes two green bands in which the detectors are offset to each other. As is known, the resolution in color television transmission has been improved by making one color band of higher resolution then the other two. As a result, this band improves the resolution of the picture. This principal is utilized in the RCA camera as well as the offsetting of the two green electromagnetic bands so as to form some subpicture elements or subpixels. However, this principal in the RCA camera is utilized with only one wave band and as such is of limited utility as compared to the instant invention.

Another solid state imaging apparatus that partially utilizes the concept of overlapping pixels is the multispectral sensor system (MSS) of NASA. This system utilized overlap of the picture elements of the same wave band in one direction. The images are focused by an optical system using a mechanical scanner. However, the overlap is done only in the direction perpendicular to the direction of movement of the satellite and the overlap involves the same wave bands. Accordingly, the resolution of the images could be improved further by utilizing the principal of the instant invention. Further, picture elements have been made smaller and smaller so that there could be more of these picture elements in a given image so as to improve the image resolution but this is a relatively costly way to improve resolution. Nevertheless, it has been found that with the same size of picture elements, the resolution of the picture can be improved markedly by using the principals of the instant invention.

Accordingly, it is the object of the present invention to improve the resolution of multispectral images formed by picture elements of a given size and system cost.

The object of the invention is accomplished by means of apparatus and method set forth in the Figures, and in the information set forth herein below.

FIG. 1 is an illustration of a conventional linear mode array.

FIGS. 2(a) and 2(b) are comparison of conventional 2(a) and systematic offset 2(b) of solid state linear array images.

FIG. 3(a) and 3(b) illustrate a comparison of conventional 3(a) and systematic offset 3(b) of solid state two dimensional arrays.

FIG. 4 is an illustration for the Example.

FIG. 5 is an illustration for the Example.

FIG. 6 is an illustration for the Example.

FIG. 7 is a schematic diagram of the typical equipment necessary along with the offset detector elements to form an image. In this configuration, there is little or no relative motion between the object being viewed and the offset detector elements. Other than the offset detector elements, the equipment represented is conventional state-of-the-art articles which perform the functions indicated.

FIG. 8 is a schematic diagram of the typical equipment necessary along with the offset detector elements to form an image. In this configuration, there is relative motion between the object being viewed and the offset detector elements. Other than the offset detector elements, the equipment represented is conventional state-of-the-art articles which perform the functions indicated.

There is provided by the present invention a solid state apparatus for taking images with improved resolution comprising (a) at least two solid state detector arrays each of which is sensitive to a different wave length of electromagnetic energy from the other in which the analogous detector elements are situated such that they are partially offset from each other so as to form subpicture elements sensitive to the different wave lengths, and (b) a computer program or algorithm (state-of-the-art) that will process the recorded data so that more information (higher resolution) is available than is possible were the analogous detectors of the different sets positioned so as to record the same area of the scene.

The basic function of the instant invention is to improve the resolution of a solid state multispectral images by having at least two detector arrays of different wave lengths of electromagnetic energy which are offset so that they partially overlap each other so as to create a number of subpixels in each picture element. It can be appreciated that the present invention can be utilized with electromagntic energy of different wave bands, and it does not have to be visible light. The figures are set forth in the visible light color bands of blue, green, and red which are most commonly used for display in such systems. Furthermore, there may not be three wave bands, there may be four or more bands, and there may be as few as two bands. FIG. 2b, shows how the wave bands may be offset from each other in one direction where, if they are located in a moving object in the atmosphere or space, the recorded picture element can be offset as indicated in FIG. 2b. FIG. 2a indicates the recorded pixels when conventionally registered pixels are not offset with respect to each other. In other words, in FIG. 2b, the offset is in the direction perpendicular to the movement of the object such that the picture element offset shown is formed by the movement such as from an airplane or satellite.

In the case of little or no relative motion between the scene and the sensor and where close up pictures may be desired, the detector arrays would be offset or overlap as shown in FIG. 3b. There the imaging apparatus or sensor may be stationary, and, accordingly, the picture elements have to be offset in two directions in order to create the necessary subpixels for the desired resolution improvement of the present invention. The present invention can also be utilized by offsetting, for example, three electromagnetic wave bands in the same matter as FIG. 2b and utilized with a mechanical scanner to improve the breadth of the image. The mechanical scanner improves the breadth of the image while the offsetting of the different wave length detectors in accordance with this invention improves the resolution of the image. As stated, bands of more than three as shown in FIGS. 1-3 can be utilized or as little as two can be used. Three were taken in the illustration in the Figures for simplicity of representation.

It can be appreciated that the present invention relies on the basic principal that an image taken with two different geometric aspects will improve the resolution as compared to an image taken with a single geometric aspect. By offsetting the picture elements, it is possible to subdivide one picture element in the case of three different bands of different wave length into nine subpixels which improves the resolution of the image that is formed. Theoretically, with a use of three different color bands offset to each other in the manner shown, the improvement of the resolution of the picture should be equal to the square root of 3. However, because the quality of the information that is obtained is depended upon the spectral difference that is present in the scene that is viewed, and is therefor not fully coherent, the actual improvement of the resolution of the image is somewhat less. According to the present example set forth at the end of the specification, the image would be improved considerably in resolution. A scene without reflective or spatial differentiation proportional to the resolution of the sensor will show little or no information gained by the use of the instant invention. On the other hand, objects or arrays which do show sufficient differentiation such as most earth and planet scenes will be recorded with a considerable increase in information content. There is needed a computer program or algorithm to process the signals obtained by the detectors and create values for the subpicture elements in the appropriate image. There are known algorithms which may be modified for doing this. For example, the Environmental Research Institute of Michigan (ERIM) utilizes such a computer program to create subpixels values for the overlapping areas involved in the multispectral scanner data (NASA's LANDSET sensor). This algorithm is commonly known as restoration or deconvolution. It can be adapted to data created by this invention. There is no claim to the formation of such a program in the present invention.

As FIGS. 3(a) illustrate, the present invention can be applied to a two dimension array for a stationary camera image forming apparatus, and it can be utilized with a mechanical scanner to improve the image that is focused on the detector elements with the different wave lengths of electromagnetic energy that may be utilized. Further, the detector elements that may be present in the apparatus are differentially sensitive to 2 or more different wave bands of electromagnetic energy.

It should be appreciated that the invention of the instant system is that the quality of an image can be improved while the size and number of the picture elements and wave bands, all of which determine cost, are not changed. The concept that is used herein to improve the resolution is to offset the picture elements so as to form sub picture or sub pixels elements in each picture element depending on how many different wave bands of electromagnetic energy are involved. As a result, the invention involves the processing of a larger number of bits of information than for the conventional mode. For example with three wave bands 9 subpixels are produced from each basic pixel thus increasing the amount of data to be processed nine fold. However with modern computer capabilities this is considered to be a small price to pay for the increased information content in a remote sensing system of a given size and cost.

A hypothetical example of how the instant apparatus works is presented below. The example assumes that there is color coherence in the wave bands recorded by the apparatus, and that there are no boundaries very close to each other, but that there is a clear boundary line between the two fields in the scene. The example is one of a simple scene to illustrate the present invention. The calculations were carried out by hand whereas in the actual situation they would be carried out by a computer. There is no intention to set limits and boundaries to the instant invention by the inclusion of the example.

The test area utilized is nothing more than a simple field boundary between a fallow field and growing crops. This boundary is illustrated by FIG. 4 which tabulates the radiometric response for three multispectral wave bands as they might record this area. Note that band 1 response remains the same for fields A and B, band 2 response is lower for B while band 3 is higher for B. To simplify analysis, the sensor path crosses the boundary at 45.degree., and one of the registered pixels (of Z dimension) is bisected by the boundary as shown by FIG. 5. The offset case involves one of the three multispectral pixels (Z dimension) also being bisected as shown. The detectors will record the values shown for the original pixels. In order to develop values for the subpixels (1/3 Z dimension) the following steps must be taken:

1. Determine value for A and B for each of the three bands. This can be done by looking at adjacent pixels of the same band for a commonality which indicates that two values (A & B) and a boundary are involved. The values for areas A and B are thus determined for each of three bands (1, 2, 3). We can then develop radiance values for:

A.sub.1, B.sub.1, A.sub.2, B.sub.2, A.sub.3, B.sub.3 which in this case are:

40, 40, 30, 20, 40, 60 respectively.

The basis pixels are identified by Roman numerals and the subpixels by Arabic numerals on FIG. 5.

2. Determination of subpixel values. The differences in the physical locations and the radiance values of the basic pixels provide the data necessary to develop radiance values for the subpixels. This is accomplished as follows (see FIG. 5):

a. The Z dimension pixels indicated by the three symbols and Roman numeral designators all have given radiance values based on the scene response and the relationship of the pixel to the boundary.

Where a pixel involves two values due to the boundary, we get a value between the A and B values based on the position of the pixel. Therefore, ##EQU1## In like manner we obtain the values for all the pixels as follows:

b. We can see that the offset has created nine response differences in each Z dimension pixel or, in effect, 1/3 Z subpixels. These are here numbered 1 through 48 and each is made up of response from the three (1, 2, and 3) bands. As an example let us compile the value for subpixel 13. To do this we must assume that the three bands do in fact correlate and that they have the same common boundary between A and B. Subpixel 13 contains the responses from pixels II.sub.1, II.sub.2, and II.sub.3, and to compute 13.sub.1 we use the band 1 values for all three pixels. Thus, ##EQU2## For 13.sub.2 and 13.sub.3 we merely substituted the A.sub.2 B.sub.2 or A.sub.3 B.sub.3 values in the same basic formula. ##EQU3## Using this same approach we compute the values for each subpixel in each of the three bands. ##EQU4## The results are shown on FIG. 6 which compares the registered response to the systematically offset response for waveband 2.

c. If one compares the response between the registered and offset pixels created by simple averaging, the differences are not very great except that the alignment (microgeometry) of the boundary is better displayed in the offset case. However, a further enhancement may be applied based on the logical assumption that we are dealing with a single boundary and that any subpixel radiance values within one third of the difference between the two field responses (for any given band) will be adjusted to the field response itself. Using this adjustment (algorithm) we get the responses illustrated by using a boundary algorithm. Here we can see a very significant improvement in the definition to the boundary and the radiance value on each side of it by using the offset approach.

This test is highly simplified and covers only one case by which a boundary between two sizable fields have been defined by conventional and offset multispectral systems. It does, however, illustrate the advantage of using the offset concept with respect to this particular test area. The world is made up of an infinite variety of patterns and responses, and there will be cases, such as areas of high frequency changes, where the elements which create the radiometric patterns are smaller than the pixels. There the improvement by the offset concept may be negligible. However, this test does indicate that, in general, resolution, microgeometry and radiometric fidelity will all be improved by use of this concept. The variety of algorithms that can be applied to the data produced by this system is very large, and the effectiveness of such algorithms will also vary according to the characteristics of the area being imaged. Further testing over a wide variety of objects and areas as expected to develop optimum algorithms for general use.