Photometric Measurement Apparatus

This invention relates to an apparatus for capturing a digital image of a scene with a wide intensitive dynamic range and for processing information contained within all or part of the captured image.

There are many situations where a need arises to make photometric measurements over a scene. For example, lighting installations (such as airport runway lighting road lighting, building lighting, aircraft cockpit lighting, sports arena lighting) are often designed to meet or exceed minimum criteria for luminance, luminous intensity or illumination. The minimal criteria are typically specified in the appropriate regulations.

The traditional approach to light measurement is to take photometric measurements of small areas with a spot photometer. When measuring the light intensity distribution within entire scenes one must provide a means of scanning the scene in a stepwise manner, one spot measurement at a time. This requires moving the measurement aperture of the spot photometer in relation to the scene--a method that is imprecise and slow at best. Only through the use of additional precision positioning equipment can the user make repeatable measurements with this technology.

An alternative approach is to capture an image of the scene with a camera. However, in order to be able to make photometrically valid measurements every location within the captured scene has information regarding the light intensity emanating from the corresponding location in the original image. This approach would not only speed up the data acquisition process but would also promote greater accuracy of measurement, since the spatial relationship among all elements in the scene are precisely preserved.

Unfortunately, the intensity dynamic range of many real scenes of interest far exceeds the capture range of all but the most expensive electronic camera systems. Consequently, useful measurements with a conventional video camera are only possible for images containing relatively small luminance ranges.

An object of the invention is to overcome this problem.

According to the present invention there is provided a photometric measurement apparatus, comprising a digital video camera for generating digital data representing the image of a scene of interest; means for stepwise changing the exposure duration of the camera; means for controlling said camera to create automatically a series of digital images of the same scene over a range of different exposures; means for storing the digital images; means for assembling data from said series of digital images to create a single composite digital image containing information from at least some of said digital images and representing at each picture element an intensity dynamic range greater than that obtainable from any single image; and means for storing said assembled data representing said composite digital image.

In this specification the expression "photometric" may include the measurement of electromagnetic radiation outside the visible spectrum. "Exposure" refers to the total number of quanta incident upon the camera's photosensitive detector array. In the most general sense, exposure can be varied by altering the optical aperture of the camera's lens or by varying the frame sampling period of the detector array. In the preferred embodiment, exposure is varied electronically.

Using the present invention, an operator can instruct the system to record one or more scenes and store them as digital files. He or she can then perform a detailed photometric analysis of the captured scenes by loading their digital image files into a computer running the system software. An image of the scene is shown on the computer's visual display and areas of interest can be identified with a pointing device such as a mouse or light pen. Spot photometric measurements can be made by locating the pointing device on appropriate points of interest.

The intensity information is normally stored in a digital file that contains more information regarding the intensity at every sampled point in the image than can be shown on the computer visual display. This makes it necessary to produce (1) a compressive transformation of the photometric data range in the original scene for use in generating the image on the visual display monitor of the computer and (2) a means of relating each spatial point representing values in the displayed image to a corresponding point in the image intensity information file.

Since complete information is stored in the image intensity information file, complex photometric analyses can be performed to produce results in any form desired. Using the invention the operator can make measurements from the image on the computer's visual display as if he or she were actually present at the location where the original scene was captured. This is because all of the relevant intensity information is stored in a digital image file even though only a fraction of it can actually be represented in the displayed image.

The camera is normally a digital still-frame device employing a charge-coupled device (CCD) array as the image detector outputting a series of digital words, each representing the intensity of one spatially quantized picture element (pixel) in the captured image. Each of these digital words is the result of an analog to digital (A/D) conversion process, whereby the signal charge on a given pixel photosite is converted to one of a number of discrete digital words, followed by the application of a calibration transformation equation. The calibration transformation equation is a function of several variables, including the location of the pixel in the spatial array, the lens aperture, focal length and focus distance, the frame sample and hold period and the spectral correction filter employed in the system at the time of image capture.

Since the analog to digital conversion process is not linear over its full dynamic range, only a subset, typically the middle portion, of the conversion scale applied to any single captured image is used as valid data for assembly into the composite image used for measurement purposes. This necessarily reduces the number of discrete digital steps that can be created from any given frame from that of an optimal A/D converter. In a preferred embodiment of the invention, the useful dynamic range of the A/D conversion process is estimated iteratively by presenting the camera with a large series of controlled spatially homogeneous images of different average intensity and measuring the digital response to each. By fitting a polynomial equation to each set of data obtained in this way, the coefficients of each fitted polynomial can be used as a description of the operating characteristics of the camera in response to lights of different intensities. The intermediate values of the function represent the region of response whereby each small incremental change in the number of quanta incident upon any given spatial region of the detector is represented by a small increment in the associated resulting digital value in the captured image.

The range of intermediate values on the A/D conversion scale will define the resolution accuracy of each individual pixel intensity value in the temporary composite image array. The image intensity information file results from the process of archiving this temporary array file upon a mass storage device.

The invention also provides a camera for performing photometric measurements from images captured from scenes, comprising a sensor array for transducing information carried to the camera from the scene by photons into an electrical signal and from that signal to an array of digital words; means for stepwise changing the rate of exposure of quanta incident upon the array from a scene of interest; means for controlling said exposure changing means to crease automatically a series of images of the same scene over a range of exposures; means for storing said images as digital files; means for transmitting said digital files for further processing.

The invention will now be described in more detail, by way of example, only with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of the hardware components of one embodiment of the invention, including a diagram of an intelligent camera containing components to perform some of the functions for obtaining and temporarily storing, on board, an array of images of the same scene, each with a different exposure;

FIG. 2 shows the functional components of the camera system;

FIG. 3 provides an overview of a photometric image analysis system in accordance with the invention;

FIG. 4 is a schematic of the system components involved in image capture;

FIG. 5 illustrates the electronic bracketing system;

FIG. 6 shows the relationship between the image capture sequence and the assembly of the composite image array, leading to the production of the image intensity information file;

FIG. 7 is a schematic diagram showing the relationship between the image intensity information file and the image display bitmap file for quantitative image analysis and reporting functions;

FIG. 8 illustrates the variations in the quantitative image analysis depending upon the number of spectral correction filters used during the image capture process;

FIG. 9 shows the range of interactive and automatic analysis tools available within the system that can be applied to a captured image file;

FIG. 10 shows the processes for defining an object or subset of the pixels within a captured image and the background against which it is viewed;

FIG. 11 illustrates the procedures associated with archiving a composite image and the subsequent analysis applied to it, including a means of exporting these files to third party software applications;.

FIG. 12 illustrates the interactive and automatic reporting tools offered by the system;

FIG. 13 shows the automation process whereby all stages of image capture, analysis, archiving and reporting are combined in the a single process.

FIG. 14 shows the processes involved in calibrating the digital camera for use with the system.

Referring first to FIG. 1, the two major hardware components of the system are the digital camera 19 and host computer 20, typically a Pentium-based personal computer running a windows.TM. operating system. Light from a scene 1 is collected and attenuated by the variable aperture of the camera lens 2 and passed through a spectral correction filter 3 before being imaged upon the camera's CCD array detector 4. The scene 1 can be any large scale or small scale scene over which it is desired to make photometric measurements. For example, a view of the earth from space, the lighting distribution in a theatrical attraction, the visual computer display on a desk in an open-plan office, a lit roadway, an airport taxiway guidance sign, the legend detail within a visual display from an aircraft cockpit, the floor of a tennis court, the beam pattern from an automobile headlamp, a self-luminous emergency exit sign, etc.

The lens 2 can be adapted to provide an image upon the CCD array 4 from scenes of arbitrary distance from the camera. The spectral correction filter 3 can be one of a set of spectral correction filters that can be separately positioned into the optical path of the camera. Each filter of the set of spectral correction filters can be used to produce a different overall effective spectral sensitivity to the camera.

The exposure duration of the CCD detector 4 is controlled by a sample and hold circuit 5, itself under control of a microprocessor 8 whose operating instructions are stored in an electrically programmable read-only memory 9. The microprocessor 8 processes digital information obtained from the analog to digital converter and generates temporary image files which are stored in local random access memory 7 on board the camera. Temporary image files can also be stored on mass storage devices 10 if transmission to the host computer is not immediately feasible.

The digital camera 19 is typically connected to a host computer 20 through a communication port such as a high speed bidirectional parallel interface. The host computer typically consists of a microprocessor 13, one or more mass storage devices 12, a human interface consisting of a keyboard 17, a pointing device 16 and visual display 18, a machine interface consisting of a modem 14 and printer 15.

In order to capture the full intensity dynamic range of a scene 1, the operator mounts the digital camera 19 on a stable platform and connects it to a host computer 20. He or she then aims and focuses the camera upon the scene of interest with the aid of a bitmap of the output of the camera displayed in a window upon the visual display 18 of the host computer system 20. Once the user has focused and set the camera to his/her satisfaction, the system proceeds to automatically compute the optimal single exposure that will be used as the display bitmap image. The image capture process then proceeds as follows:

FIG. 2 shows a schematic of the image capture process of a preferred embodiment of the digital camera. As is known, filtered and focused light made incident upon the imaging area of a CCD array 4 causes displacement of electrons in the crystalline structure of silicon. This displacement results in the accumulation of charge carriers within the localized regions near each photosite in the array. The initial spatial distribution of charge across the detector array substrate is replenished by charging the array via the sample and hold timing circuit 5. The charge distribution process is under timing control of the digital camera control unit 112. At the end of the sample and hold period controlled by timing circuits 5, the image is represented on the CCD imaging area by a spatially differential charge distribution within the detector array 4. The time period between replenishment of charge carriers in the substrate is known as the frame sampling period. The frame sampling period is the time during which the action of light within the image focused upon the array is able to cause a differential spatial charge upon the detector and is, thus, analogous to exposure duration for conventional photographic film. Additional circuits cause the charge distribution located in the detector array 4 to be shifted into an adjacent storage array 206 which is shielded from light.

From the storage array the image is scanned pixel by pixel and output through the analog output register 207 to a video amplifier 208 whose gain is programmable under control of the digital camera control unit 212. At this stage the amplified signal is converted from analog to digital form 209, formatted into a digital array and stored in digital array storage memory 211. From this memory the digital image remains available to be reconverted and conditioned into an analog signal 210 suitable for displaying on a television monitor 214 or formatted by the digital camera control unit 212 for communication as a digital file string through a communication port 11 to a host computer 20. The digital camera control unit 212 also provides a trigger signal 216 synchronous with the initiation of a frame sampling period. The unit 212 also accepts remote trigger inputs 217 for initiating the picture taking sequence from an asynchronous external source.

FIG. 3 shows an overview of the functional components of a photometric image analysis system in accordance with the invention. The initial task for the user is to log on to the system 301. This process has two purposes: to ensure that only authorized users have access to the system and that pertinent information is available to the system regarding the identity and contact data (telephone numbers, etc.) of individuals who capture images and/or perform quantitative analysis upon them. After the user has logged onto the system 301 he or she has four initial options: to capture digital images at 302 with the camera 19; to perform quantitative image analysis upon captured files 310; to produce reports on the results of previous analyses 313 and to perform calibration procedures on the camera 305. The first three of these options, 302,310 and 313, lead the user to the use of associated interactive and automatic tools 303,311,314, that assist in the prosecution of the various associated information processing tasks. If the user opts for digital image capture 302 the system provides a set of interactive and automatic image capture tools 303 that assist the user in programming the system to capture the full intensity dynamic range of a scene.

FIG. 4 shows more detail regarding the image capture tools. The user first decides if a new image is to be acquired at decision 402. If a previously acquired image is to be loaded into the host the system recalls a digital image file from the image archive 405. If new images are to be acquired the system invokes the electronic viewfinder mode 403 which provides the user with visual feedback from the camera on the computer visual display that can be used to focus and align the image 404. Once this process has been completed the user enters the lens settings and other information 406 into the host computer, including the acceptance criteria for the number of invalid pixels within the captured scene. This criterion is used by the system to decide when to stop collecting intensity data from a scene, as shown in FIG. 5.

The system then establishes a temporary file 409 that includes the various entered parameters and data obtained from the viewfinder mode 3. The user then decides if multiple pictures are required 411. If so, the user is then given the choice 408 of having the multiple pictures taken by an automatic time lapse process 407 or on manual command through keyboard entry on the host computer (FIGS. 1, 17) or through an electronic trigger input to the camera (FIGS. 2, 17). An additional decision is made by the user regarding the dynamic range of the image to be captured. If the user opts for electronic bracketing 412 an image capture sequence is initiated which culminates in the assembly of a composite intensity information file. If the user does not opt for electronic bracketing, then the system sets the exposure frame sampling period 413 and captures a single image for downloading into the host computer 414.

FIG. 5 shows the details of the electronic bracketing procedure. FIG. 6 shows how electronic bracketing is implemented in a preferred embodiment as two sequential processes--one involving the capturing of a sequence of images and the other involving the production of an electronically assembled composite image. The electronic assembly process results in an image intensity information file containing data from the pixels locations of at least some of the images obtained in the image capture sequence and stored in image array memory. Referring to FIG. 5; first, an image capture sequence is programmed 505 and used to initiate an image capture sequence 502 resulting in a series of images being taken of a scene, each captured at a different frame sampling period. The data from the first captured image are downloaded into the host computer 503 where all pixels in the image are evaluated for valid pixels 504. The intensity of each valid pixel is computed by referring to one of a set of polynomial equations chosen on the basis of information regarding the lens settings entered in the Viewfinder mode and stored in a temporary file (FIGS. 4, 9). From this selected polynomial equation the A/D converter values are transformed into actual photometric measurement units. These units are then stored in a temporary composite image file 511. The system continues this process until all valid pixels have been converted in this way. It then searches for the presence of invalid pixels 513.

If invalid pixels are found in this image, it is determined if there are any additional images are present in the image capture sequence 510. If there are, the next stored image from the capture sequence is searched for valid pixels 507 and the conversion process is repeated on them. Following this the system queries for remaining invalid pixels and repeats this process until either no more invalid pixels are found or invalid pixels are found but no more images are present in the capture sequence. If the latter is true, the system programs a second image capture sequence, but with a different aperture setting on the camera lens 506. Once this second sequence is captured the process of converting valid pixels to photometric data begins again. This process iterates until either (1) all pixel addresses have been entered with valid data or no additional lens aperture settings can be found that will generate a reduction in the number of remaining invalid pixels. At this point the image is optionally archived 512.

Referring to the overview FIG. 3: The user may also optionally perform certain digital camera calibration procedures 305, which are related to but different from the factory calibration procedures that are described later and schematized in FIG. 14. The user digital camera calibration procedure 305 results in the production of numerical correction factors that can be applied to all intensity measures for all pixels in a given captured scene. This process is useful when referring photometric data to an in-house absolute or relative calibration standard or when specific types of illumination sources are used to light a scene.

FIG. 7 shows the relationship between the image intensity information file 707 and the image display bitmap file 705. The image intensity information file 707 is available to the computer's calculation engine 704 for computing various photometric functions of the original image scene. The user is only able to visually appreciate an image of the captured scene through the translation of the image display bitmap file 705 back into a visual stimulus, through the use of the human.backslash.computer interface 702 and display 708.

The various options for performing quantitative image analysis are shown in FIG. 8. The user must determine if an image had been captured using a single spectral correction filter 4. If a single correction filter has been used to capture an image 2, the user then determines whether the image was obtained from a luminance distribution or from a beam pattern distribution (used to measure radiant intensity 10 and luminous intensity 11). If the light distribution from a beam projection pattern was not captured 5, then the user has the option of performing analysis with radiance 3, luminance 4, or analysis with an arbitrary spectral correction filter 9, such as a filter designed to correct the CCD array for the scotopic or nighttime spectral sensitivity of the human eye, rather than the photopic or daytime spectral sensitivity. If an image has been captured with more than one spectral correction filter 7 the user has the option of quantitative analysis of images with either arbitrary functions of multiple filters 8 or calorimetric analysis with specified calorimetric filters 12.

FIG. 9 shows a schematic diagram of the available processes for interactive and automatic analysis of intensity data within an image. By centering the pointing device cursor on a location within the image the system will compute various descriptive statistics of intensity for all the pixels within a series of circular areas concentric to the cursor position and report them in the appropriate measurement units 902. Various grids can also be applied and aligned to the image and the system will compute these descriptive statistics for all pixels within each cell in the grid 903. Since it is typical of the photometric measurement process to be interested in objects within a scene, the system provides the user with various means for defining one or more objects within a scene and for defining the backgrounds against which these objects are viewed 904. This process is critical to the ensuring accurate, valid and reliable measurements of the intensity of objects within the scene and for ensuring that the relationship between object and background is consistently defined.

The various methods that are available to define both objects and the backgrounds against which they are viewed are shown in FIG. 10. If the object can be easily defined by its outline within the scene, the user may opt to outline the shape of the object with the pointing device 1002. For more complex objects (for example, the alphanumeric characters in the captured image of a visual display), the user may define the object by the range of intensity values that all pixels in the object have in distinction to those of the background against which the object is viewed. This can be done by bracketing the intensity range of the object subset of the pixels within the scene 1003. If the scene have been captured with more than one spectral correction filter (as in FIGS. 8, 7) the user may choose to define the pixels in the scene belonging to the object in terms of a function of more than one pixel intensity variable 1004. The user may also choose to define the full image as the object 1005.

Referring to FIG. 10, the background against which an object is viewed is also definable in a similar fashion to the methods used for defining the object. The background can be defined by the user by outlining with the host computer pointing device 1007. Using this method the polygon defining the object outline always remains inside the polygon defining the outline of the background. Having defined the object by outlining the user may opt for defining the background as a multiple of the area of the convex hull of the object outline 1008. In this method of defining the background the system computes the convex hull of the polygon outline of the object and iteratively expands this shape while computing the solid angle of the background. This area is compared to that of the interior of the polygon defined by the object. The iteration proceeds with the comparison of the two areas until the area of the background reaches some user-defined integer multiple of the area of the object.

Three additional optional methods of defining the background are also available in the system, as shown in FIG. 10. In the first, the user may select all pixels in the image that are not defined as part of the object 1009. In the second, the user may define the background by bracketing the intensity range of the remaining pixels in the scene not defined as those of the object 1010. Finally, for images captured with more than one spectral correction filter, the user may define the background in terms of a function of more than one pixel intensity value 1011.

Since both process 1004 and 1011 refer to defining objects in terms of more than one pixel intensity value, it can be appreciated that this includes the process of defining both objects and backgrounds by their chromaticity. One such embodiment of this process is through the use of a set of filters through which transformations can be computed resulting in chromaticity measurements of the pixels in the image as taught by the Commission Internationale de l'Eclairage or CIE. The data can then be represented in a chromaticity diagram such as the 1931 CE x,y chromaticity diagram or the 1976 CE u',v' chromaticity diagram.

Once one or more objects and backgrounds have been defined these definitions can be optionally saved in an electronic data file 1012 for future recall and use.

Referring again to FIG. 9; following the object/background definition process 904 the user can make quantitative measurements of two types: measurements based upon the photometric intensity of objects and backgrounds 905 and object dimensional analyses, including measurements of linear extent and of area 906. Descriptive statistics, such as the mean, median, range, variance, etc. of the intensity values of the defined object(s) and background(s) are automatically computed by the system 907 as are various functions of the relationship between defined object and background, called contrast metrics 908. Several different contrast metrics are automatically computed. The different metrics. which are all derived from measures of defined objects and defined backgrounds, provide meaningful quantitative measures to various interest groups and individuals with various areas of expertise within lighting measurement community.

The user may also optionally define a reference area within the scene and the system will compute the ratio of every pixel intensity in the scene that is outside this reference area to the average of the pixels within the defined area 909. This measurement procedure can be visualized through the use of pseudo-color mapping of the intensity scale within the image. Using this method the intensity range within the pixels of the image is quantized into n discrete intervals or "bins" and each bin is assigned a perceptually discrete color. Thus, when the image is viewed the color of a given pixel or groups of pixels provides the user with a convenient means of visualizing the intensity ratio defined above.

Two defined objects from either the same image or from different images can also be compared 12. The differences between the objects and between their defined backgrounds are computed for both intensitive descriptive statistics and for dimensional measurements. Arithmetic operations can be applied to images using both unary and binary operators and transformations 904. A typical example of such operations is the subtraction of one image from another, as is often done to determine what has changed in a scene between two sample periods.

Various visual performance metrics that depend upon both the intensitive contrast and linear and/or area extend measures are automatically computed by the system in process 915. Through the use of visual ergonomic data stored in a database 913 predictions of the visual performance of individuals can be estimated 916. The visual range of objects can be automatically computed in process 917, which incorporates mathematical models of light scatter to the transmission of contrast metrics calculated in process 908. The visual range of an object is thus defined as the distance from the observer at which the contrast falls below a specified critical value.

The results of various quantitative analyses applied to defined objects and backgrounds are automatically displayed in graphical and tabular form within the visual display and/or printed as hardcopy reports. Additional hardcopy generation capabilities are described in a later module.

After image capture or once the quantitative analysis has been carried out on a given image the captured image and quantitative results can be stored in a digital image file archive as shown in FIG. 11. The user enters various pieces of information 1106 regarding the details of the pictures analyzed through the host computer keyboard and the host computer appends the date and time of image capture to the file in the archive 1107. The archiving process 1101 generates two related files for storage: the image intensity information file 1109 and the image display bitmap file 1110. This process optionally permits lossless compression to be applied to the image intensity information file 1108.

Image intensity information files can be optionally exported to 3rd party software applications 1102. This process permits several file formats and export/import protocols to be developed within the system that conform to Microsoft Windows file transfer protocols.

FIG. 12 schematizes the various options available for reporting the results of the quantitative analyses performed upon captured images. The term reporting herein refers to the process of producing the report as output on either a hardcopy on a printer 1205 or an electronic file transmitted to another site or facsimile machine via a modem 1203. Interactive and automatic reporting tools 1201 permit the user to instantly output a series of preformatted single-page reports at the conclusion of each measurement process from the host computer 1202 or to append each report to an accumulation file and output them all at once at a later time 1204. Stored report files can also be exported to a 3rd party application. The image bitmap file 1207, the image intensity information file 1208 and the file containing the results of the image analysis 1209 can be exported separately.

The image capture, analysis, archive and reporting functions as separately outlined above, can also be incorporated into a process whereby all functions are automated. FIG. 13 shows how the quantitative image process sequence generator 1302 receives programming inputs from each of the four processes, image capture 1301 image measurement 1303 archiving 1305 and reporting 1307. The sequence generator produces a control sequence from these inputs to cause the four processes image capture 1304, image measurement 1306, archiving 1308, and reporting 1309 to be iteratively executed. This is useful where repetitive analysis of the intensitive and dimensional properties of objects within images must be carried out. An example of this requirement is where quality inspection is performed on a lighting product at a production stage within a manufacturing process.

A schematic of the factory calibration process applied to the digital camera is shown in FIG. 14. A number of calibration procedures are carried out on a given digital camera/lens combination. The tests include linearity of the intensity scale 1407, spatial resolution 1417, spatial extent 1413, lens and field uniformity 1415 and detector noise 1414. For each test the camera is mounted on a stable platform in the calibration facility where it is aligned and focused the stimulus target(s) appropriate for a given test. For the linearity test the camera is presented with a large series of spatially uniform targets produced by the output of a stimulus generator 1410 under control of a dedicated computer system. The intensity of the stimulus from this generator is monitored by an optical intensity monitor 1411 that has, itself, been calibrated with reference to a secondary standard light source traceable to a National Standards Laboratory.

The intensity linearity calibration procedure 1407 involves capturing a very large number of images for each lens focal length and aperture setting, where the control commands for the production of each image are obtained from a calibration file in the intensity linearity test generator 1407. For each calibration image presented at a fixed intensity, a series of images is captured where each image in the series is obtained at a different frame sampling period. The full range of sampling periods is represented in the series. From these data the arithmetic mean of digitized intensity values as a function of pixel position are obtained and used as input to a polynomial curve fitting procedure 1408. The output of this procedure is used to generate a set of calibration coefficients 1404 which are used to estimate the intensity of any arbitrary stimulus input to the camera.

Referring again to FIG. 14, the accuracy of the intensity coefficients developed in the linearity calibration procedure 1407 are tested through the use of a linearity performance test 1402. This test introduces a large number of calibrated intensity stimuli back into the camera which captures the images and computes the average intensity of local regions of the array. The system then computes the difference between the estimated intensity and the measured intensity from the optical intensity monitor 1411. After the test is complete a decision is made as to whether the calibration is acceptable. If the calibration is acceptable the coefficients are stored in the camera calibration database 1403 from which the camera calibration files are generated 1406. Copies of this calibration file are then made, along with a hardcopy report of the results of the performance test 1409. If the calibration test fails 2 the calibration linearity test process 1407 is repeated whereupon the linearity test is repeated. Cameras that fail the performance test 2 twice are removed from production and rebuilt.