US 5,065,444 AGrant
Streak Removal Filtering Method and Apparatus
Issue Date:1991-11-12
•10 Claims
•11 Drawing Sheets
Abstract
A filter for decreasing sensor artifacts in an image frame as, for instance, sensor induced streaking in the image frame, has a frame buffer for temporary storing the picture elements of the image frame. A picture element mask is utilized to group related picture elements and statistics are collected with respect to these grouped picture elements. Correction parameters are determined utilizing these statistics and these are then utilized to correct the individual picture elements to remove the sensory induced artifacts. The statistics can be collected with respect to picture element gain and picture element offset with the correction parameters then determined utilizing the picture element gain and picture element offset.
Metadata
Assignee
- Northrop Corporation
Inventor
- David D. Garber
Application Information
Application Number:US 6421334
Filing Date:1991-01-15
Priority Date:1987-07-24
Art Unit:266
Classifications
IPC:
G06K 940
Field of Search:
38235830-34;42;51;54166;167
Patent Drawings (11 sheets)
Description
Background of Invention
This invention is directed to an apparatus and method for processing image frames to decrease sensor artifacts in the image frames, more particularly to decrease sensor streaking artifacts in the image frames.
Many sensors for collecting data have anomalies in the sensors which cause artifacts in images output by the sensors. Typical of sensors which exhibit these properties are infrared sensors, i.e. IR sensors. Multi-detector parallel scanning IR sensors (including common module FLIR sensors and focal plane arrays) can induce sensory artifacts such as streaking into the image output of these sensors. The sensor artifacts degrade the performance of other downstream systems which utilize the output of the sensors.
The artifacts produced by the sensors are of varying origins. Artifacts are very noticeable for IR sensors during `warming up` of the sensors. However, individual elements in a multiple element sensor also change or drift over short periods of time even after the sensor is in a full operational mode. Additionally, response characteristics of individual sensor elements are also dependent upon the content of the scene which the sensor is detecting. For IR sensors very hot objects as, for instance, burning objects, engine compartments of vehicles, exhaust areas of vehicles and the like cause excessive streaking artifacts in data images obtained utilizing the IR sensor.
Temporal induced artifacts and hot spot induced artifacts are very difficult to remove utilizing filters which rely on measuring sensor response or filters which rely on the use of a large set of natural images or uniform test images. Artifacts would also degrade the effectiveness of time averaging filters which might utilize a large sequence of images over time to produce an optimal streak removal filter. Further, this last type of filter would require considerable computational requirements and image storage requirements for the production and storage of a sufficiently large number of images to serve as a basis for the filter. Also, by requiring the use of many sequential images for filtering purposes the ability to operate in real time or near real time is substantially compromised.
For a scanning sensor or a multiple element sensor, in theory it would be possible to have automatic gain circuits or other control circuits for each of the sensor elements and to adjust the circuits such that the output of every sensor element or sensor scan is equivalent. This, however, is not a practical solution because of the hardware demands and/or the inability of hardware or human operator to adjust the output of each sensor element or sensor scan to be totally equivalent to the output of all others.
Brief Description of the Invention
In view of the above it is evident that there exists a need for new and improved filters for the removal of sensor induced artifacts from the output of sensors. Further, there exists a need for real or near real time operating filters for reducing sensor induced artifacts.
It is a broad object of this invention to provide for image filters for decreasing sensory artifacts in the output of sensors. It is a further object of this invention to provide for image filters which operate in real or near real time to decrease such sensory induced artifacts. It is an additional object of this invention to provide filters which are capable of decreasing sensory artifacts in an image frame without utilizing either prior image frames or post image frames in order to operate on a current image frame.
These and other objects as will be evident from the remainder of this specification are achieved in a filter device for decreasing sensory artifacts in an image frame which includes a frame buffer means for temporarily storing the picture elements of an image frame. The device further includes a collection means for collecting correlation parameters between related picture elements in the frame and a correction parameter determination means for determining picture element correction parameters utilizing the correlation parameters. Additionally, it includes an image correction means which utilizes the picture element correction parameters for adjusting the stored picture elements of the image frame to decrease the sensory artifacts in the image frame.
The above described filter advantageously can include a picture element mask means for grouping related picture elements during the collection of the correlating parameters. Further, the collection means advantageously can include a statistics collection means for collecting statistical parameters between related picture elements. The collection means can further advantageously include a relationship sensing means for determining a relationship of a parameter of a selected picture element with a parameter of at least a further picture element.
The relationship sensing means can advantageously include a threshold clipping means for setting a difference threshold between picture element parameters and selectively categorizing picture element parameters according to this difference threshold. Illustrative of the threshold clipping means is an histrograms means for storing picture elements in a histogram and categorizing the picture element parameters in accordance with a clipping threshold established in the histogram.
The correction parameter determination means advantageously includes a picture element gain determining means for determining the differences in image frame gain between related picture elements and for generating a gain correction parameter for these related picture elements and further a picture element offset determining means for determining the difference in image frame offset between related picture elements and for generating an offset correction parameter for these related picture elements.
The objects of the invention are also achieved in a method of decreasing sensory artifacts in an image frame which includes grouping related picture elements of the image frame into sets and then collecting correlational parameters based on relationships between the members of the sets. The method further includes generating picture element correction parameters utilizing these correlational parameters and then adjusting the picture elements in the image frame with the correction parameters to decrease sensory artifacts in the image frame.
The correlational parameters can advantageously be chosen as statistical parameters as, for instance, statistical means and statistical variances. The grouping of related picture elements is advantageously achieved utilizing a picture element mask where the mask selects picture elements which are located in physical proximity with each other in an image frame. The mask can be advantageously moved across the totality of the image frame to collect the correlational parameters for use in generating the picture element correction parameters. In generating the correctional parameters, gain and offset correctional parameters between rows and columns of the picture element in the image frame can be utilized for adjusting the picture element within individual rows or columns for correcting the sensory artifacts in the image frame.
Brief Description of the Drawing
This invention as described in this specification will be better understood when taken in conjunction with the drawings wherein:
FIG. 1 shows a representational view of a typical image sensor, a filter of the invention and further processors and detectors which utilize an image generated by the sensor,
FIG. 2 is a schematic view in partial phantom line showing progression of an image through multiple filters constructed according to the teachings of the invention,
FIG. 3 is a representational view of a typical unfiltered image output by the sensor of FIG. 1,
FIG. 4 is a representational view of the same image of FIG. 3 after it has been processed by a filter of the invention,
FIG. 5 is a schematic view of a typical mask utilized within a filter of the invention,
FIG. 6 is a representational view of an enlarged section of an image frame similar to those of FIGS. 3 and 4,
FIG. 7 is a representational view of an image frame containing a large real object in the frame which crosses the frame in essentially a horizontal direction,
FIG. 8 is a diagram of a histogram related to the image frame of FIG. 6,
FIG. 9 is a schematic flow diagram of certain of the steps involved in processes of the invention,
FIGS. 10, 11 and 12 are schematic flow diagrams of portions of the schematic flow diagram of FIG. 9 further detailing individual portions of the schematic flow diagram of FIG. 9,
FIGS. 13A-13C, taken together, comprise a schematic diagram of device implementing a filter of the invention, and
FIGS. 14A-14G, taken together, comprise a detailed flow diagram of an implementation of the invention.
Detailed Description of the Invention
FIG. 1 shows a typical utilization of a filter of the invention. As is shown in FIG. 1, a filter of this invention is utilized in conjunction with other elements as, for instance, elements which are disclosed and described in detail in application Ser. No. 007,717 filed July 24, 1987 and entitled TARGET ACQUISITION AND TRACKING SYSTEM, the entire contents of which are herein incorporated by reference and of which this application is filed as a continuation-in-part application thereof.
In the application `TARGET ACQUISITION AND TRACKING SYSTEM` a sensor is utilized for obtaining raw input data. This data is digitized in the sensor and then output by the sensor. Alternately analog data from a sensor can be digitized in an independent A to D converter. In any event the digitized sensor data is processed via several global processors and the output of the global processors are utilized in an image detection system for detecting target images in a target field. These target images are then output as a listing of potential targets. As is evident from the very nature of such a system, to be meaningful, image processing must be done in real or near real time.
For the target acquisition system of FIG. 1, a sensor 20 is utilized. The sensor 20 includes an appropriate sensing element therein as, for instance, a scanning IR sensor which is responsive to targets within a target field in the environment which is being scanned. The sensor 20 further includes an internal A to D (analog to digital) converter for digitizing the analog output of its sensing elements and outputing a serial stream of this raw digitized data.
In accordance with this invention the digitized serial data stream output by the sensor 20 is fed to a filter 22 of the invention. The filter 22 of the invention is utilized to remove certain sensor induced artifacts in the data output by the sensor 20. After processing by the filter 22, output from the filter 22 is fed to a global processor 24 and from there to an image detector 26. The global processor 24 processes the digitized data it receives such that the image detector 26 is capable of detecting images within a target field utilizing a variety of parameters as are more fully discussed in the above referred to application entitled TARGET ACQUISITION AND TRACKING SYSTEM. Targets detected by the image detector 26 are then processed by a target lister 28 and are separated and individually acted upon as per other teachings of the above referred to application entitled TARGET ACQUISITION AND TRACKING SYSTEM.
FIG. 3 shows a typical image frame 30 as it would exist without the use of the filter 22 of the invention. As is seen in FIG. 3, within a target field generally depicted by the numeral 32 there is a target 34. In addition to the target 34 there are a plurality of anomalous horizontal sensor induced streaks collectively identified by the numeral 36. It is evident from FIG. 3 that the horizontal streaks 36 generally degrade the totality of the image within the target field 32 and contribute to pixel clutter within the image field. Without the filter 22 of this invention, the global processor 24 and image detector 26 must contend with this pixel clutter while they are processing the image data for identification of the target 34 within the target field 32.
In a similar manner, if the sensor 20 was utilized as, for instance as a night vision sensor, a field of view presented to an observer thereof would be similarly contaminated with the horizontal streaks 36. For the purposes of FIG. 3 the sensor 20 was chosen as a horizontally scanning sensor and as a result thereof artifacts induced into the image field by the sensor 20 are expressed as the horizontal streaks 36. If a vertical scanning sensor was utilized vertical streaking would be present. In any event, the anomalous sensor induced artifacts, i.e. the streaking, degrade the unfiltered sensor image and make target acquisition or other processing of the image frame more complex and less reliable.
FIG. 4 shows the same image as is depicted in FIG. 3 with the exception that the filter 22 of the invention has been utilized to remove anomalous sensor induced horizontal streaking. In FIG. 4 the same target 34 is more clearly seen within the same target field 32 when compared to FIG. 3. It is evident in comparing FIGS. 3 and 4 that any subsequent processing of the target field 32 and the target 34 therein can be more readily done with the image of FIG. 4 because the sensor induced artifacts have been removed.
Further, the image of FIG. 4 has been processed in real or near real time by the filter 22 allowing it to be used by other real or near rear time systems as, for instance, as a night vision device or digitized image frames on a target identification device or the like.
The filter 22 of the invention operates by taking serialized digital data of an individual image frame and processes that data utilizing only the information in that individual image frame to remove anomalies from the frame which have been introduced into the frame as sensory artifacts. The filter 22 of the invention does not require prior frames or subsequent frames in order to process the data content within an individual image frame and remove the anomalies therein. Because of this the filter 22 of the invention does not require large amounts of memory storage and can operate in real or near real time processing each subsequent image frame as it is received from a sensor.
The filter 22 of the invention can be implemented in a variety of apparatus. These include a strictly hardware device hard wired with appropriate circuits to execute the filtering process steps of the invention or a hardware device having a ROM memory attached thereto containing the coded steps as, for instance, in micocode. Alternately they include a software implementation in an appropriate computer device wherein the instruction steps for operating the filter of the invention are in software and are implemented within the computer CPU with the image frames being I/O'ed (input/output) through conventional I/O ports and the like of the computer.
In order to further the understanding of certain details of the invention a discussion of certain generalized concepts will facilitate the understanding of more specific concepts of the invention. If one considers an individual frame, i.e. an image frame, to be made of pixels or pels which are arranged in rows and columns in the picture frame, an individual pixel or pel, i.e. a picture element, can be identified by its value and its row and column position. The image in the image frame can be constructed by assigning gray values as, for instance, an eight bit value in the range from zero to 255 to each picture element. Thus, if such a image frame was displayed on a CRT, an overall scene would be displayed in black, gray and white as determined by the gray values of the individual picture elements.
An individual picture element can thus be identified by its value P and its position in the image frame by its row coordinate i and its column coordinate j. Thus an ideal individual picture element P at coordinates i, j is identified as P.sub.i,j. What is received from a sensor as, for instance the sensor 20, however, is not such an ideal or perfect picture element but, in fact, is an observed element P'.sub.i,j.
This observed picture element P'.sub.i,j can be related to the ideal picture element as, for instance,
This relationship states that an observed picture element is related to the ideal picture element by a gain distortion a.sub.i and offset distortion b.sub.i for the picture element row i and by an unexplained noise or variance element .epsilon..sub.i,j indicative of both the row and column position of the picture element. Simply stated this indicates that the ideal image is distorted by some linear gain factor a.sub.i, an offset factor b.sub.i and a noise factor .epsilon..sub.i,j.
If the .epsilon..sub.i,j terms (the noise or unexplained variance values) are considered to be local, random and uncorrelated, then in constructing a filter of the invention they can be considered as averaging out over an area of the image frame which incorporates a sufficient number of picture elements. If the .epsilon..sub.i,j terms are considered to average out, the above relationship between the observed and the ideal picture element can be rearranged as follows:
If however the error term .epsilon..sub.i,j is included and it is considered that a restored image would be proportional to this term, than an estimated ideal image P*.sub.i,j is related to the observed image as follows:
When .epsilon..sub.i,j is not equal to zero, only a determination of the P*.sub.i,j, i.e. the estimated value for the picture element, can be made instead of a determination of the ideal picture element P.sub.i,j. Given an observed image P'.sub.i,j it is not possible to determine a.sub.i and b.sub.i without knowledge of the ideal image P.sub.i,j. However, a determination of an estimate a*.sub.i and b*.sub.i can be made.
It has been found that adjacent or neighboring pixel elements in an image are normally very strongly related to each other and non adjacent picture elements tend not to be related.
For the filter of the invention an advantage is taken of the finding that neighboring picture elements are related. By considering the homogeneity of a small neighborhood or small set of related picture elements estimates of the gain a*.sub.i and the offset b*.sub.i are possible for restoration of an artifact contaminated image.
If picture elements P.sub.i,j and P.sub.i+1j are from a homogeneous region then
If this above relationship is rearranged as follows: ##EQU1## it is noted that the picture elements P.sub.i,j and P.sub.i+1,j are related by a linear multiplicative factor: a.sub.i+1 /a.sub.i and an additive factor: (1/a.sub.i) (b.sub.i+1 -b.sub.i).
If this same principle is applied between rows (i+1) and (i+2) a similar relationship: ##EQU2## is noted. In equation 6, as with equation 5, P.sub.i+1,j is related to P.sub.i+2J by a linear multiplicative factor: a.sub.i+2 /a.sub.i+1, and an additive factor: (1/a.sub.i+1) (b.sub.i+1 -b.sub.i+1).
If the above relationships are combined row i can be related to row (i+2) as follows: ##EQU3## Again it can be noted that correction factors for row (i+2) relative to row i are a linear multiplicative factor: ##EQU4## that is the product of factors derived using adjacent rows and an additive factor: ##EQU5## which involves individual additive factors, one which involves individual additive factors, one multiplied by a linear multiplicative factor.
If this process is continued it becomes evident that a linear adjustment for row (i+k) relative to row i is a product of k linear factor with these factors determined by comparing adjacent linear line pairs. It is important however, to remember the above teachings with respect to homogeneity of the image. An arbitrary k row can simply not be compared to row i. The rows must be related such that they essentially contain homogeneous data, that is related data. It has been found that this can be achieved by selecting rows and/or columns which are proximal or adjacent to one another in an image frame since the probability of such proximal rows or columns containing related or homogenous data is high.
Utilizing the above concept a gain correction factor, e.g. the a.sub.i factor above, can be selected if it reflects a linear multiplicative factor. Further an offset correction factor, e.g. the b.sub.i factor above, can be selected if it reflect an additive factor.
If related or homogeneous data is utilized a statistical variance can be selected for a gain correction factor since it incorporates a linear multiplicative factor. In a like manner a statistical mean can be selected as an offset factor since it incorporates an additive adjustment factor. However, this requires that these statistical correlation factors be utilized on picture elements which in fact are related.
This invention utilizes a mask to select related picture elements. For use in the illustrative embodiment herein a mask 38 of FIG. 5 is selected. The mask 38 relates picture element P.sub.i,j with respect to both its preceding and post occurrence related elements P.sub.i,j-1 and P.sub.i,j+1 respectively in the same row as element P.sub.i,j as well as 3 elements in the same column positions in a row displaced two rows previously and 3 elements in the same column positions in a further row displaced three rows previously.
While wishing not to be bound by theory it is believed that by ignoring the picture elements in row P.sub.i-1 (the immediate preceeding row) and not including them in the mask 38, sensory artifact factors included within the .epsilon..sub.i,j term of Equation 1 are lessened since the .epsilon..sub.i,j factors tend to be nonhomogeneous and thus do not cancel out with respect to adjacent pixels. Over a larger picture element area however the .epsilon..sub.i,j terms tend to be more homogeneous and thus cancel out.
Utilizing the mask 38, the differences between the picture element P.sub.i,j and the remaining 8 picture elements in the mask 38 when summed across a row in an image frame can be expressed as a relationship as, for instance, the relationship expressed in the following equation: ##EQU6##
Thus across a row d.sub.i,j values (the average difference between the picture element P.sub.i,j and the eight other respective picture elements includes in the mask 38) are summed. These represent the differences between row i and rows (i-2) and (i-3) for the picture elements in those rows as determined utilizing the mask 38.
For the filter of the invention, a picture element correlation collection step as for instance, the masking analysis step of equation 8, is utilized. Equation 8 in the form listed is used when horizontal anomalies such as the horizontal streaks 36 are to be filtered out of an image frame. If vertical anomalies or vertical streaks are to be filtered, of course, the mask 38 would be passed along the columns of the image frame with an appropriate substitution made in the mask analysis equation (and the hardware or software corresponding to it or utilized to implement it) to reflect that the mask is processed across the columns and incremented or stepped along each row during the column processing.
In essence the filter step as implemented by Equation 8 (or an equilivent equation as for instance an equation for moving a mask verticially) measures and collects the difference in gain between succeeding columns in the image frame as analyzed utilizing the mask 38. The mask 38 after stepping across a first set of rows in the image frame is then incremented down to step across a next set of image rows. That is, the mask 38 is first oriented on P.sub.i,j measuring the picture elements in row P.sub.i,j and P.sub.i-2,j and P.sub.i-3,j as it is stepped across these rows and then is incremented to row P.sub.i+1,j and stepped across this row measuring the picture elements in rows P.sub.i+1,j, P.sub.i-1,j and P.sub.i-2,j.
Put in other terms the mask is swept across the image frame and picture elements in row 4 are compared to rows 1 and 2. The mask is then incremented one row to orient it on row 5 and row 5 is then compared to rows 2 and 3. The mask is then further incremented one row and row 6 is compared to rows 3 and 4. This is done for the totality of the image frame.
In progressing the mask 38 across the image frame, stepping across as, for instance, row 4, data for picture elements in rows 1 and 2 is utilized for comparing the picture elements of row 4. Since it is only at row 4 that there are three previous rows, i.e. rows 1, 2 and 3, the mask 38 cannot be used for the first three rows in the same manner as it is used for the remainder of the rows, however, normally there is a border on any image frame which mitigates this difference. This border can essentially be ignored since generally in sweeping across an environment setting targets or other items of interest in that setting tend to become centered in the image frame.
As a practical expedient border columns can be purposely ignored on both the left and right hand side of the image. Further as is the case for an implementation of the invention described below, the picture elements of the upper border of the image of the frame image can be assume to have ideal gain values i.e. a value of 1. The lower border could also be treated in a similar manner. In sweeping an image, a sensor tends to have more distorted data on the edges of the image than in the center of the image and thus ignoring of the borders of an image results in loss of little or no meaningful image data.
In order to utilized Equation 8, for row 4, data for the picture elements in rows 1,2 and 3 are processed but are not used directly in the calculations of adjustments for the remainder of the rows.
If it is desired to correct the picture elements in the totality of an image frame, a first sweep of a mask, such as a mask 38 can be made in a top down manner which results in obtaining complete data for the bottom rows and a second sweep can be made in a bottom up manner resulting in obtaining complete data for the top rows.
The above correlation parameter collection process can be enhanced by taking into account the average differences of the picture elements across an image row. Thus, after obtaining each of the individual differences between the picture elements of a row with the mask 38, the average difference d.sub.i can be calculated as, for instance, using Equation 9 or an equilivent method as follows: ##EQU7## In further computations the individual differences d.sub.i,j are then all made relative to the average difference by substracting the average difference d.sub.i from them.
While neighboring picture elements in an image frame tend to be very homogeneous, when an object is encountered in that image frame the homogeneity of the neighboring elements decreases. In FIG. 6, in processing the mask 38 across row i comparing the picture elements in row i to those of row i-2 and row i-3, at row i, column x+1 the mask 38 encounters a real object 40.
Real object 40 is not an anomaly but in fact is an object or target within the field of view being scanned. Picture elements in row i upto and including column x tend to be homogeneous with respect to picture elements in rows i-2 and i-3 upto and including column x, however those in column x+1 do not because at this point the mask 38 has encountered the object 40.
By incorporating picture elements in row i, column x+1, in the difference determinations as expressed in Equations 8 (with or without utilizing the average row correction factor of equation 9), an effect of the real new object 40 would distort the correction factors being determined for removal of the anomalous streaking artifacts. This, however, can be corrected for by setting a threshold and only utilizing those difference parameters d.sub.i,j which are within or bounded by the threshold. Those difference parameters as, for instance ones occurring because of the presence of the real object 40 in row i, column (x+1) of FIG. 6, which are not within the bounds of the threshold, that is they are extrinsic of the threshold, will clipped about the threshold and will not be utilized further.
FIG. 8 shows a histogram of a sweep across a scene as for instance across the row i of FIG. 6. The majority of the picture elements fall in the major area 42 of the histogram clustered about the average deviation d.sub.i. However, those picture elements which occur because of the appearance of the object 40 in the scene of FIG. 6 fall into the area 44 of the histogram.
A threshold value t.sub.i can be chosen such that only those d.sub.i,j values which lie in the area 42 which is less than the absolute value of the threshold chosen are further processed. As such the threshold essentially clips or eliminates from consideration or filtering purposes those picture elements which occur in local nonhomogeneous neighborhoods as the result of the appearance of a real object. Since real objects destroy the homogeneity of a local neighborhood, sets of picture elements which include the picture elements from the real object are clipped and are not used for further correction parameter determinations.
Normally the object size of any real objects in an image frame are sufficiently small compared to the overall size of the frame itself that any real object is encountered in sweeping across a row or a column for only a portion of that row or column. This is evident in FIG. 6 for the object 40. To eliminate the real objects, the threshold cutoff value can be chosen arbitrarily, however, for typical image frames a threshold cutoff value of 20% is normally sufficient to eliminate any effect of real objects during correction of the image frame to remove anomalous artifacts. Depending upon the image frames one might select threshold values of from about 0% to about 75% or typically at from about 5% to about 50%. Thus if very small objects are expected within an image frame lower threshold values would be selected and if large objects are expected within an image frame larger threshold values would be selected.
While it is theoretically possible as is seen in FIG. 7 that a real object 46 will extend in a perfect straight line across the field of view of an image frame parallel to a frame edge, for all practical purposes such large real objects such as the object 46 will either not have perfectly sharp edges or will not have edges which are parallel to one of either the vertical or horizontal axis or both. Thus the probability of encountering such a real object 46 within an image frame is very low. Because of this the probability of such a real object 46 distorting the collection of correlation parameters is also low.
After the difference values d.sub.i,j are obtained by processing the mask 38 completely over the image frame, means and variances of the collected d.sub.i,j 's can be determined as, for instance, using the following equation 10 for the mean and equation 11 for the variance. The d.sub.i,j 's are made relative to the overall characteristics of the image row by making each d.sub.i,j relative to the overall row by substracting the average difference d.sub.i from each individual d.sub.i,j prior to summation and threshold processing as per equations 10 and 11. ##EQU8##
Equations 10 and 11 and the process steps they represent take into account the utilization of a percentage threshold `perot` to eliminate the effect of real objects being included in collecting the parameters which are used to eliminate sensor induced artifacts. In Equations 10 and 11 the differences for the picture elements P are summed across the image frame for only those elements whose absolute value is less than a selected threshold t.sub.i.
The percentage threshold `perot` is globally utilized over the totality of the image frame to eliminate the effect of any real object in calculating the means and variances for use in making a.sub.i * and b.sub.i * parameter estimations. Thus, only picture elements as, for instance picture elements in major area 42 which are not part of a real object are utilized in calculating the correlation parameters, i.e the means .mu. and variances .sigma..sup.2.
Referring for a moment to FIGS. 9 and 10, an image frame is received from a sensor element or other system component as a digital serial data stream and is temporarily stored in a buffer 48. A mask is processed sequentially across rows of the image. The rows are stepwise or otherwise retrieved from the buffer 48 during this process and certain correlation parameters are collected. These correlation parameters are generated in the correlation collection step indicated at 50 of FIG. 9, which is also shown and expanded at 50 of FIG. 10 to indicate more detailed steps in FIG. 10.
In FIG. 10 the correlation collection step indicated at 50, is broken down into the steps of collecting the differences d.sub.i,j along each row at 52, determining the average d.sub.i along each row at 54, creating a histogram of d.sub.i,j -d.sub.i for each row at 56, clipping the histogram at a threshold t.sub.i at 58 followed at 60 by determining the means, .mu., and variances, .sigma..sup.2 in each row for all pixel elements wherein the absolute value of the difference of d.sub.i,j is below the threshold t.sub.i.
For use in implementing the filter 22 of the invention, as noted above, the picture elements of the image frame can be stored in a suitable buffer 48 as, for instance a two dimensional array and the remainder of the computational data can be stored in temporary and refreshable buffers as, for instance one dimensional arrays.
The data now generated in the correlation collection step at 50 is now utilized in a correction parameter determining step shown both at 62 in FIG. 9 and in an expanded form also at 62 in FIG. 11. Estimates for the gains a.sub.i * are obtained utilizing Equation 12 for the mask 38. ##EQU9##
It is of course, realized that if a different mask was utilized an appropriate equation equivalent to Equation 12 would be utilized to mimic the mask parameters. In any event, the normalized standard deviations of the variances across the rows which represent linear multiplicative factors are utilized to obtain an estimated value a*.sub.i of a gain correction parameter. This is identified at 64 in FIG. 11.
As is evident from Equation 12 the derivation of the linear gain correction factor of each subsequent row includes the previously determined linear gain factors of previous rows. In order to insure that the overall gain of the image across the entirety or totality of the image frame does not change because of progressively incorporating increases in gain between subsequent rows, the product of all the a.sub.i * is determined in a suitable manner as, for instance equation 13, and then each individual a.sub.i * is adjusted by a factor of the product as per Equation 14. ##EQU10##
In reference to FIG. 11, at 66 the a* sub product as is exemplified by Equation 13 is determined and at 68 the a*.sub.i 's are adjusted with this product as per Equation 14.
Estimates for the offset correction factors b.sub.i * are also obtained in a suitable manner as, for instance as follows:
As discussed generally above, correction offset factors can be estimated utilizing a procedure which takes into the account the linear multiplicative factor and individual additive factors as generally discussed with respect to Equations 6 and 7 above.
The offset factors are determined at 70 in FIG. 11 utilizing Equation 15. However, again in order to insure that the overall bias of the image does not change across the image frame, the overall mean of the image offset adjustment, i.e. the b*.sub.sum, is determined in a suitable manner as, for instance utilizing Equation 16 at 72. Then each of the b*.sub.i 's can be adjusted with the b*.sub.sum in a suitable manner as, for instance utilizing Equation 17 at 74 in FIG. 11. ##EQU11##
It has been found that generally the offset factor has a greater effect on decreasing sensory artifacts than the gain factor. Generally, the gain factors, the a*.sub.i are very close to unity (1). Utilizing these findings other suitable correction parameter determinations as for instance the processes exemplified by Equations 18 and 19 can be substituted as alternates for Equations 16 and 17. ##EQU12##
These alternate correction parameter determinations differ from Equations 16 and 17 in that they take into account the finding that the general mean of a particular line in certain instance would be very close to 1 and the linear factors can therefore be essentially ignored. However, for greater accuracy in correcting and removing sensory artifacts in an image frame, preferredly the linear factors would be taken into account when making the offset corrections.
Referring now to FIGS. 9 and 12, once the correction parameters have been determined image correction as shown at 76 in FIGS. 9 and 12 can be effected. In image correction at 76 in FIG. 12 computed correction parameters from 62 of FIG. 11 are utilized at 78 to adjust the picture elements of the image frame stored at 48 to produce the corrected image frame picture elements. The corrected image frame picture elements can then be clipped to a range as, for instance, to an 8 bit gray scale range of 0 to 255 in range clipper 80.
The correction step at 78 is shown in Equation 20 wherein the adjusted picture element P*.sub.i,j is determined from the observed picture element P'.sub.i,j by multiplying it by the gain factor a*.sub.i and adding to it the offset factor b*.sub.i as follows:
As per the process described in FIG. 9, an image frame has been corrected utilizing a filter 22 of the invention to remove horizontal sensory artifacts, i.e. streaks which were introduced into the image frame as anomalous data by the sensor 20. Depending upon the sensors utilized and the desired degree of correction necessary for a particular image frame, one or more image corrections will be effected. As per FIG. 2 a horizontal filter 82 as per the filter described in discussing the apparatus and process of FIG. 9 and the related figures, has been utilized to remove horizontal streaking from an image.
The output from the horizontal filter 82 could be utilized directly as an adjusted image or alternatively it could serve as input data to a vertical filter 84. In the vertical filter 84, the image data would be processed in a manner equivalent to that seen above for the horizontal processing except that vertical anomalies would be removed from the image. This image could then be utilized as an adjusted image or the image could be processed even further by passing it through a further horizontal filter 86 and a further vertical filter 88.
Since the filters of the invention process or filter the images utilizing only one image frame at a time and do not require either a proceding or subsequent image frame to effect this filtering process, as noted above they operate in real time or near real time. For a sequentially filtering process as is depicted in FIG. 2, individual image frames could be pipelined through the successive filter stages, that is from filter 82 to filter 84 to filter 86 to filter 88 etc., also in real or near real time.
Since each image frame is filtered independent of any other image frame, the serial filters of FIG. 2 operating in a pipeline can independently operate on successive image frames in an expedient manner. No one of the filters 82, 84, 86 or 88 is depending upon any of the other of the filters or the image frame currently being processed in the other filters.
A device of the invention is illustrated in FIG. 13. Digitized data from a sensor 90 is stored in frame store 92. Mask device 94 processes the store picture elements. Output from mask device 94 is processed to obtain the average deviation of each frame row by device 96 and the output of this device and that of device 94 are fed to device 98 to obtain relative picture element deviations. A threshold cut off and the output of device 98 are input to device 100 for creating a histogram of this input data. Device 102 determines the threshold cut off in the data and device 104 process the data for the means and variances of the differences of the picture elements in the image rows.
The row difference values in storage arrays 106 are processed in multipliers 108 and 110 with the respective variances from device 104. The output of the multipliers 108 and 110 is fed to device 112 for summing the two sets of data and this is output to divider 114 for division by the variance of the test row. The root of the output from the divider 114 is processed in root device 116 with the output of this device stored in the storage arrays 106 of a gain storage buffer 118.
The adjusted data in storage arrays 106 is also fed to multipliers 120, 122 and 124 for processing with respective means from device 104. Unadjusted data in storage arrays 126 are combined with the output from the multipliers 120, 122 and 124 in device 128. Device 128 outputs uncorrected (corrected for the total frame) offsets to storage arrays 126 of an offset storage buffer 130.
Mean data and mean data processed with data from storage array 106 in multiplier 124 is combined with data from buffer 130 and feed back data in device 132 and the output of this device is accumulated in accumulator 134. The uncorrected offset data is corrected with the output from device 134 in device 136.
Original picture element data in buffer 92 is processed with data from the gain buffer 118 in multiplier 138 and the output of this in turn is processed with the output from device 136 in picture element correction device 140. The output from the correction device 140 is fed to limiter 142 for clipping the corrected image frame picture element to a particular gray scale. A filtered frame image is output from the limiter 142.
The device of FIG. 13 does not adjust the gain correction parameters for the totality of the picture frame as is illustrated in the process steps 66 and 68 of FIG. 11, however, if desired these steps could be implemented in the the device of FIG. 13. The correction for the offsets over the totality of the image frame corresponding to steps 72 and 74 of FIG. 11 are however implemented in the device of FIG. 13 in devices 132, 134 and 136.
In FIG. 14 a detailed flow diagram is shown detailing an implemention of a filter of the invention. A histogram cutoff value is obtained at input 144. The first four rows of an image frame are then processed at loop 146. These first four rows are read into a storage array at row store 148 and loop variables are zeroed out at 150. Loop 152 is then iterated reading across the columns and the sums and the sums.sup.2 of the picture elements of the first four rows are then determined at 154. These sums and sums.sup.2 are then utilized to compute the means and variances for the first four rows at 156.
The remainder of the rows of the picture elements starting at row 4 are then processed in loop 158. A row is stored into the array storage element at 160 and variables zeroed out at 162. Nested loops 164 and 166 are then iterated to get the mask picture elements at 168. The differences d.sub.i,j are calculated at 170 and the overall sum of each row d.sub.i are calculated at 172.
Average differences for each row are then computed at 174. Variables are zeroed out and the histogram bin size is set or reset at 176. The columns are read at loop 178 and the pixel differences are made relative to the average difference at 180. The appropriate histogram bin for the pixel is selected at 182 and the histogram bin is incremented at 184.
The histogram bin cutoff is then evaluated with respect to the filled histogram bin at decisions 186a and 186b and if it is inappropriate with respect to the histogram (186a) and if the maximum bin size has not been reached (186b), the bin is incremented at 188. If the histogram is within the bin size or the maximum bin size has already been reached, the process continues wherein the largest histogram bin is selected. Threshold cutoff is set at 190 and the sum and sum.sup.2 counters are zeroed at 192.
Loop 194 is iterated and the relative differences d.sub.i,j are examined to see if they fall within the histogram cutoff range at decision 196. If the relative differences fall within the histogram range then sums and sum.sup.2 are determined at step 198. If the differences are outside of the histogram range the loop is iterated and these values are ignored. For those values within the histogram range the values are then utilized to compute the means and variances at 200 for the fourth row through the maximum row.
The above steps correspond to processing step 50 of FIG. 9 as was detailed in FIG. 11 at 52 through 60 respectively.
Having determined the correlation parameters for the image frame, loop 202 is now iterated and the a* value for the top rows of the image frame which were not analyzed with the mask is set to 1 at 204. Gain parameters are set at 206 and the loop 208 executed. Within the loop 208 the a* for rows 4 through the maximum row are computed at 210. The parameters are zeroed at 212 and loop 214 executed. In loop 214 the a* product for rows 4 through the maximum row is found at 216. A computed gain multiplier is then calculated at 218 and range variables are set at 220.
A loop 222 is executed and the a*.sub.i values are adjusted for the total gain of the image frame at 224. Range parameters are set at 226 and a loop 228 executed wherein the b* parameters are zeroed for rows 1 through 3 at 230.
A loop 232 is executed and the b*'s for rows 3 through the maximum row are computed at 234. At 236 range parameters are set and loop 238 is then executed wherein the b* sum is computed at 240. The mean of the sums is computed at 242 followed by setting range parameters at 244. A loop 246 is executed and computed adjusted b* offset parameters are calculated at 248.
The above referred to steps 202 through 248 correspond to the process steps of FIG. 9 indicated at 62 as detailed in FIG. 11 at 64 through 74 respectively.
After calculating the correction parameters for the totality of the image frame, nested loops 250 and 252 are iterated and the picture elements adjusted with the correction parameters at 254. The adjusted picture elements are then clip at step 256 between 0 and 255 to remain within the gray scale of the image frame. A corrected image frame is then output at 260 for further use as, for instance, by the global processor 24 of FIG. 1 or for display storage or other process depending upon the implementation which the filter of the invention is being utilized for.
As noted the filter of the invention can reside in hardware and can utilize an appropriate micro code or the like for implementing the hardware filter or alternately it can be utilized utilizing a CPU which processes image frames from tape input, other memory input or from direct input from an appropriate sensor under the control of software.
An implementation of the filter of the invention is given as a Fortran language program which corresponds to the flow diagram of FIG. 14. Certain more trivial routines of this code have been left out for brevity of the code as, for instance, filling arrays and the like and I/O (input/output) routines which would be device dependent. This code is as follows. ##SPC1##
Claims
What is claimed is:
1. A streak removal filtering method for estimating an image of pixels P.sub.ij from measured pixel intensity values P'.sub.ij in an image frame, comprising, the steps of assuming that pixels in a predetermined region near the pixel P.sub.ij are strongly related to each other and to P'.sub.ij and that a correction based on P*.sub.ij =a*.sub.i P'.sub.ij +b*.sub.i can compute streak removal information, where a*.sub.i is the gain correction, P'.sub.ij is the measured intensity of the pixel, b*.sub.i is an offset correction, and P*.sub.ij is the estimated value of the pixel P.sub.ij of the actual image, as corrected; performing the following steps in a computer: computing the differences d.sub.ij between each pixel in said region and the measured pixel P'.sub.ij ; thresholding d.sub.ij for values lower than predetermined values above which the difference may be assumed to be due to real object changes in the image; computing the mean and the variance of the intensity values of the pixels P'.sub.ij in said region for which d.sub.ij is less than said threshold; averaging the product of the gain corrections and the variances of the pixels preceding the measured pixel, P'.sub.ij, ratioed by the variance of the immediately adjacent pixels to estimate said gain correction, a*.sub.i ; averaging the corrected pixel values from regions of pixels preceding the measured pixel, P'.sub.ij, less the computed means pixel intensity value times the gain corrections of the adjacent pixels to determine said offset correction, b*.sub.i ; computing the estimated value P*.sub.ij of the intensity value of P.sub.ij by adding the offset b*.sub.i to the measured intensity P'.sub.ij from said relation P*.sub.ij =a*.sub.i P'.sub.ij +b*.sub.i ; processing successive regions of pixels of the image, in seriatim, in accordance with the above steps, to develop a streak corrected image formed from the corrected estimated values of pixels P*.sub.ij ; and, thereafter outputting said corrected image of said image frame.
2. The method as set forth in claim 1 further including the steps of selecting n gain correction values for n respective regions in said image, ratioing the gain correction value of the present region by the n.sup.th root of the product of the n gain corrections.
3. The method as set forth in claim 2 further including the step of: reducing the offset correction by the average of all offset corrections after correcting the means of each region according to the computed gain corrections.
4. The method as in claim 1 in which said processing step includes masking to define said region of pixels about P.sub.ij to consist of P'.sub.ik, P'.sub.i-3,k and P'.sub.i-2,k for k=j-1, j,j+1.
5. A streak removal filtering method for estimating an image of pixels P.sub.ij from measured pixel intensity values P'.sub.ij in an image frame, comprising, the steps of assuming that pixels in a predetermined region near the pixel P.sub.ij are strongly related to each other and to P'.sub.ij and that a correction based on P*.sub.ij =a*.sub.i P'.sub.ij +b*.sub.i can compute streak removal information where a*.sub.i is the gain correction, P'.sub.ij is the measured intensity of the pixel, b*.sub.i is an offset correction, and P*.sub.ij is the estimated value of the pixel P.sub.ij of the actual image, as corrected, and assuming the gain a*.sub.i can be set to unity, as an approximation; performing the following steps in a computer: computing the differences d.sub.ij between each pixel in said region and the measured pixel P'.sub.ij ; thresholding d.sub.ij for values lower than predetermined values above which the difference may be assumed to be due to real object changes in the image; computing the mean and the variance of the intensity values of the pixels P'.sub.ij in said region for which d.sub.ij is less than said threshold; averaging the corrected pixel values from regions of pixels preceding the measured pixel, P'.sub.ij, less the computed mean pixel intensity value times the corrected gain correction of the adjacent pixels to determine said offset correction, b*.sub.i ; computing the estimated value P*.sub.ij of the intensity value of P.sub.ij by adding the offset b*.sub.i to the measured intensity P'.sub.ij from said relation P*.sub.ij =P'.sub.ij +b*.sub.i ; processing successive regions of pixels of the image, in seriatim, in accordance with the above steps, to develop a streak corrected image formed from the corrected estimated values of pixels P*.sub.ij ; and, thereafter outputting said corrected image of said image frame.
6. Apparatus for removing streaks from an image of pixels P.sub.ij from measured pixel intensity values P'.sub.ij in an image frame by assuming that pixels in a predetermined region near the pixel P'.sub.ij are strongly related to each other and to P.sub.ij and that a correction based on P*.sub.ij =a*.sub.i P'.sub.ij +b*.sub.i can compute streak removal information, where a*.sub.i is the gain correction, P'.sub.ij is the measured intensity of the pixel, b*.sub.i is an offset correction, and P*.sub.ij is the estimated value of the pixel P.sub.ij of the actual image, as corrected; comprising means for computing the differences d.sub.ij between each pixel in said region and the measured pixel P'.sub.ij ; means for thresholding d.sub.ij for values lower than predetermined values above which the difference may be assumed to be due to real object changes in the image; means for computing the mean and the variance of the intensity values of the pixels P'.sub.ij in said region for which d.sub.ij is less than said threshold; means for averaging the product of the gain corrections and the variances of the pixels preceding the measured pixel, P'.sub.ij, ratioed by the variance of the immediately adjacent pixels to estimate said gain correction, a*.sub.i ; means for averaging the corrected pixel values from regions of pixels preceding the measured pixel, P'.sub.ij, less the computed mean pixel intensity value times the corrected gain correction of the present row to determine said offset correction b*.sub.i ; means for computing the estimated value estimate P*.sub.ij of the intensity value of P.sub.ij by adding the offset b*.sub.i to the measured intensity P'.sub.ij from said relation P*.sub.ij =a*.sub.i P'.sub.ij +b*.sub.i ; and means for processing successive regions of pixels of the image, in seriatim, in accordance with the above steps, to develop a streak corrected image formed from the corrected estimated values of pixels P*.sub.ij.
7. An apparatus as in claim 6 further including means for selecting n gain correction values for the n respective regions in said image, means for ratioing the gain correction value of the present region by the n.sup.th root of the product of the n gain corrections.
8. An apparatus as in claim 7 further including means for reducing the offset correction by the average of all offset corrections after correcting the means of each region according to the computed gain corrections.
9. An apparatus as in claim 8 further including a mask for defining said region of pixels about P.sub.ij to consist of P'.sub.ik, P'.sub.i-3,k and P'.sub.i-2,k for k=j-1, j,j+1.
10. Apparatus for removing streaks from an image of pixels P.sub.ij from measured pixel intensity values P'.sub.ij by assuming that pixels in a predetermined region near the pixel P.sub.ij are strongly related to each other and to P.sub.ij and that a correction based on P*.sub.ij =a*.sub.i P'.sub.ij +b*.sub.i can compute streak removal information where a*.sub.i is the gain correction, P'.sub.ij is the measured intensity of the pixel, b*.sub.i is an offset value, and P*.sub.ij is the estimated value of the pixel P.sub.ij of the actual image, as corrected, and assuming the gain a*.sub.i can be set to unity, as an approximation; comprising, means for computing the differences d.sub.ij between each pixel in said region and the measured pixel P'.sub.ij ; means for thresholding d.sub.ij for values lower than predetermined values above which the difference may be assumed to be due to a real object changes in the image; means for computing the mean and the variance of the intensity values of the pixels P.sub.ij in said region for which d.sub.ij is less than said threshold; means for averaging the corrected pixel values from regions of pixels preceding the measured pixel, P'.sub.ij, less the computed mean pixel intensity value times the gain corrections of the adjacent pixels to determine said offset correction b*.sub.i ; means for computing the corrected estimate P*.sub.ij of the intensity value of P.sub.ij from said relation P*.sub.ij =P'.sub.ij +b*.sub.i ; and means for processing successive regions of pixels of the image, in seriatim, in accordance with the above means, to develop a streak corrected image formed from the corrected set of pixels P*.sub.ij formed from the corrected estimated value of pixels P*.sub.ij.