Adjusting an electronic camera to acquire an image

[0002] This application is a continuation of application Ser. No. 09/444,770 filed Nov. 22, 1999 and now U.S. Pat. No. 6,366,680.

[0003] This invention relates to electronic cameras and more particularly to the control of electronic cameras.

[0004] Electronic cameras are commercially available and in widespread use. The basic technology used in electronic cameras is well developed. Electronic cameras generally include CCD or CMOS light sensing elements. The exposure time of the light sensing elements and the gain of the readout circuitry can be set to control the quality of the images which are acquired by the camera.

[0005] Some electronic cameras have a built in exposure meter that detects the amount of light reaching the camera. The appropriate exposure time and gain of the sensing elements is then automatically set at values determined by the signal generated by the exposure meter.

[0006] Some electronic video cameras which are designed for integration into a personal computers provide the user with a menu such as that shown in FIG. 1 which can be used to set various parameters to control the operation of the camera. A user typically sets the various parameters so that the camera will produce an image that is visually appealing to a human observer. A program in the computer adjusts the actual settings in the camera in a direction which attempts to have the images meet the specifications set by the user. This type of adjustment may take place over several sequential frames. Furthermore as ambient conditions change, the actual exposure time and gain of the sensing elements is varied between frames in an effort to meet the specifications set by the user.

[0007] Electronic cameras can be used as part of systems such as that shown in copending application Ser. Nos. 09/314,648, 09/343,104, 60/158,015, and [attorney docket 60037, filed Nov. 3, 1999]. The systems shown in the referenced copending applications are designed to detect a digital watermark in an image acquired by a video camera. In such systems after a video camera is used to acquire an image, the acquired image is digitally processed to detect a digital watermark that is contained in the image. It has been found that the optimal settings for a camera used in such systems is not necessarily the optimal settings that are established to provide a pleasing image to a human viewer.

[0008] The type of parameters that can be established using a menu such as that shown in FIG. 1 do not always produce optimal conditions for detecting a digital watermark in an acquired image.

[0009] The present invention provides a mechanism for automatically setting the exposure time and gain of an electronic camera so that the acquired image has improved characteristics for the detection of a digital watermark. With the present invention the exposure time and gain of an electronic camera are set by first directing the camera at an image with the camera's controls set to a set of default or initial values. The system then calculates and stores data which represents a histogram of the luminance values of the pixels in the acquired image. Next characteristic values from the histogram data such as the peak value, the centroid value, the RMS value, the median value, etc., are calculated. One or more of the characteristic values of the histogram data are then applied as an index or pointer to values of pre-established settings that are stored in a table. Values from the table are used to set the exposure time and/or gain of the camera. In some embodiments, the variance of the luminance histogram data is also measured and used to index data from the table. In some embodiments, a test target is used to acquire an initial image. In other embodiments, the initial image is made by pointing the camera at the image which contins the watermark data.

[0010] FIG. 1 illustrates the screen presented to a user to enable the user to set the values that control a camera.

[0011] FIG. 2 is an overall diagram of a first embodiment of the present invention.

[0012] FIG. 3 is a diagram that illustrates a luminance histogram.

[0013] FIG. 4 is a diagram illustrating a table which stores values for exposure time for a camera's shutter.

[0014] FIG. 5 is a flow diagram of the control program executed by the computer embedded in the camera.

[0015] FIG. 6 is a flow diagram of an alternative embodiment.

[0016] The overall system shown in FIG. 2 includes a conventional digital camera 21 and a conventional control computer 22. As is conventional the camera 21 includes a CCD detector 23 and a lens 24 which focuses an image on the CCD detector. The CCD detector 23 has a gain control line 23aand an exposure control line 23b. The control computer 22 controls the values on the lines 23aand 23b. While for convenience of illustration and explanation the camera 21 is shown in FIG. 2 as an individual unit separated from control computer 22, it should be understood that in many commercially available systems the camera 21 is mounted within (or attached to) the computer 22 and lines 23aand 23brepresent logical control lines rather than physical wires.

[0017] As is conventional the computer 22 includes memory 22aand programming 22b. For convenience and clarity of illustration and explanation these are illustrated as separate boxes in FIG. 2. The other conventional parts of the camera 21 and the computer 22 are not illustrated in FIG. 2 and not discussed herein since they are not particularly relevant to an explanation of the invention.

[0018] In the first preferred embodiment of the present invention, the camera 21 is first directed at a test target with the exposure time and gain of the camera set to the default values. When the camera is activated, the camera will detect an image, each pixel of which has a certain luminance value. The luminance value of each pixel is a particular numeric value between 0 and 255. The number of pixels at each luminance value is counted. The count values form a histogram such as that shown in FIG. 3. Naturally in computer 22, the histogram is stored as data values. FIG. 3 is merely an illustration of an example of such data.

[0019] The test target used in the first preferred embodiment is a standard photographer's “gray card”. That is, a card that has an image consisting of a gray background that is eighteen percent gray.

[0020] Various mathematical parameters can be calculated from a histogram (or from the data that can form a histogram). In first the preferred embodiment, the centroid value of the luminance data is calculated and used for subsequent operations. However, it is noted that in alternative embodiments, other values are used including:

[0021] a) The peak value of the histogram,

[0022] b) The root-mean-square value of the points that form the histogram,

[0023] c) The median value of the points that form the histogram.

[0024] d) The deviation value of the curve.

[0025] e) etc.

[0026] The value of a parameter from the histogram data (in the first preferred embodiment that is the centroid value) is used as an index or pointer to values in a table which contains exposure values. An example of such a table is given in FIG. 3. Naturally it should be understood that in computer 22, the data is stored as a conventional data table. FIG. 4 is merely an illustration of such a table.

[0027] FIG. 5 is a flow diagram illustrating the steps in the operation of the system shown in FIG. 2. First, as indicated by block 501, the control parameters of the camera are set to a set of default values. In particular the exposure time is set to {fraction (1/90)} second. Next as indicated by block 502, the camera is pointed at a test target and an image is acquired. The test target is a conventional photographers gray scale that is eighteen percent gray.

[0028] The luminance of the pixels in the image are measured (as indicated by block 503) and a histogram of the luminance values is generated. In the particular embodiment, each pixel has a luminance between 0 and 255 (i.e. an 8 bit representation) and the histogram has an appearance somewhat like the histogram shown in FIG. 3. Naturally, the actual values will be dependent on various factors such as the amount of ambient light present. Furthermore, if a 16 bit system were used, the luminance would have values between 0 and 65535.

[0029] Next (as indicated by block 504) the centroid of the histogram is calculated in a conventional manner. That is, the centroid is calculated from the luminance data directly. There is no need to actually construct or calculate the histogram. The histogram is merely discussed herein to principles of the invention. A centroid is calculated directly from the luminance data by adding together each luminance value times the number of pixels at that value. For example: assume that there are:

[0030] 3 pixels with value 1,

[0031] 2 pixels with value 2

[0032] 6 pixels with value 3

[0033] 5 pixels with value 4

[0034] etc

[0035] The centroid is then calculated as:

[0036] (3×1+2×2+6×3+5×4+ . . . etc.)/(3+2+6+5 . . . etc.)

[0037] The value calculated for the centroid is used as an index or pointer to select a value in a table such as the table shown in FIG. 4 as indicated by block 506).

[0038] Next (block 506) the exposure time for the camera is set to the value obtained from the table and an image is acquired using that exposure time (block 507). Finally (block 508) a watermark is read from the acquired image in a conventional manner.

[0039] A flow diagram for a second preferred embodiment of the invention is shown in FIG. 6. In this alternative embodiment, no test target is used. Instead, the camera is pointed at the image which contains a watermark and repeated frames containing the image are acquired.

[0040] Between each frame a new setting for the camera is calculated using luminance values as previously described; however, in this embodiment, the luminance values are the luminance values of an image that contains a watermark. After each image is acquired an attempt is made to read the watermark, if the watermark can not be read, the new camera setting are used to acquire another image. The process repeats until the watermark can be read.

[0041] FIG. 6 is a flow diagram of the operations. The camera is set to a relatively slow frame rate (such as five frames per second) so that there is sufficient time between frames to process the data to detect a watermark. This operation is conventional. The calculations to determine a new setting for the camera take much less time than do the calculations to detect a watermark.

[0042] The process begins as indicated by block 601 when the camera is pointed at an image containing a watermark and an image is acquired. As indicated by blocks 602A and 602B, two operations proceed in parallel. First an attempt is made to read the watermark. If that is successful (block 603) the operation goes to block 605 where the watermark data is used for purposes such as those described in the previously referenced co-pending applications. The second program that operates after each image is acquired in the program indicated by block 602B. This program assembles data that forms a histogram of the luminance of the pixels in the image, the centroid is calculated and a new exposure value is obtained from a table (similar to the operation of the previous embodiment). Next the camera is set to the new exposure value as indicated by block 604.

[0043] Finally if the watermark reading operation was not successful another image is acquired and the process repeats.

[0044] The following is a more a specific description of a program that implements the operations shown in FIG. 6. This embodiment of the invention utilizes a commercially available “3Com HomeConnect” PC Digital Camera. The camera as commercially available provides a software interface, which allows many parameters to be queried and modified. The available parameters and their value ranges shown in parenthesis are:

[0045] Enable/disable the interface (e/d)

[0046] Enable/disable Auto White Balance (AWB) tracking (e/d)

[0047] Red balance (0-255)

[0048] Blue balance (0-255)

[0049] Color saturation (0-255)

[0050] Set the mode of Auto Gain Control (AGC) operation (disabled, average mode, average center window mode, peak mode)

[0051] AGC bias (0-100)

[0052] Shutter value (0-3900)

[0053] Video gain (0-255)

[0054] Gamma value (0-255)

[0055] Image sharpness (0-255)

[0056] The only parameter that the program which implements the present invention changes dynamically in its automatic operation is the shutter value. This value relates to a shutter speed and has a non-linear range of 0-3900. In this scale, 0 is a shutter speed (or exposure) of {fraction (1/31,500)} of a second and 3900 is a shutter speed of ¼ of a second.

[0057] When the program starts, the first thing it does is to enable the software interface to the camera. The program saves the current values of all the camera parameters and then puts in its own set of values. The camera's initial values will be restored when the program shuts down. The following are initial values for various parameters that have been found to operate satisfactorily in this particular embodiment of the invention.

[0058] When each image is acquired the program starts examining the video frame for watermarks as indicated by block 602A. As each frame is received, its luminance centroid is calculated and compared to our target luminance. The program dynamically adjusts the shutter value in order to move the centroid towards the target luminance as indicated by block 604.

[0059] Values used in the algorithm:

[0060] Target luminance: 110 (range is 0-255)

[0061] Distance is the distance between Centroid and Target

[0062] Midpoint is one half of Distance

[0063] Direction flag: set to 1 if Centroid <Target, otherwise set to −1

[0064] LastShutter is the shutter value from the last frame we received

[0065] Shutter is the current shutter value

[0066] LastLuminance is the centroid for the last frame we received

[0067] DeltaLuminance is the difference between LastLuminance and Centroid

[0068] AdjustmentAmount is the change that we will be making to the Shutter value for this frame

[0069] LastAdjustment is the AdjustmentAmount we made on the last frame we received

[0070] Adjustment table (defined below)

[0071] The adjustment table lays out ranges of luminance values along with adjustment factors and a scale flag. The adjustment factor is a multiplier which is used along with the distance from Centroid to Midpoint. If the scale flag is set for a particular range then the Current Shutter value is also factored into calculations involving that range.

[0072] The following algorithm is implemented by the program:

[0073] If Centroid has not changed since the last frame and the distance from Centroid to Target <20 then don't make any adjustments

[0074] Set CurrentAdjustment to 0

[0075] Walk through the Adjustment Table. For each entry in the table, if any part of the Area, defined as the area between Centroid and

[0076] Midpoint falls within the range of Lower Bound to Upper Bound then calculate Scale and AdjustmentIncrement as follows:

[0077] If Flag is Off then

[0078] Scale=Factor

[0079] Else

[0080] Scale =Factor*Log (Current Shutter)

[0081] AdjustmentIncrement=(amount of Area that falls between Lower Bound and Upper Bound )*Scale*Direction

[0082] Each AdjustmentIncrement is added to CurrentAdjustment.

[0083] Compare DeltaLuminance to LastAdjustment. If DeltaLuminace is a positive number while LastAdjustment is negative or vice-versa then it means that movement of Centroid is going in the opposite direction from our adjustments. We will continue to adjust until Centroid movement switches directions, then we will slow down and wait for Centroid to catch up. At the same time we will limit CurrentAdjustment changes to be no more than 25% of the Shutter value.

[0084] It is noted that in the preferred embodiments described above, the invention is used to only set the exposure time of the camera. In other embodiments, the table shown in FIG. 4 is expanded to also include the value of the gain and the gain of the camera is set utilizing the present invention. In such an embodiment both exposure time and gain can be set using one parameter from the histogram data. Various rules could be used to construct the table used to obtain values of the exposure time and gain, that is, to map the one dimensional parameter value into a two dimensional exposure/gain value. In similar fashion other parameters of the camera can be set utilizing the present invention.

[0085] While the invention has been described with respect to preferred embodiments thereof, it should be understood that various other changes in form and detail are possible without departing from the sprit and scope of the invention. For example, while the invention is particularly illustrated with reference to a camera employing a CCD detector, any other optical sensor technology can likewise be employed. Similarly, while the invention is particularly described with reference to a video camera, the same principles are equally applicable to watermark detection with non-video image acquisition device. Yet further, while a table was used in the illustrated embodiment to generate camera settings, other known structures and approaches (e.g., formulas) using the same or similar input information can be readily substituted. Still further, while a histogram was described as a preferred analysis technique, other statistical analyses for characterizing the camera's response can similarly be utilized. Accordingly, the scope of applicant's invention is limited only by the structure and methods described in the appended claims and equivalents thereto.