Pattern Optimization When Measuring Depth to a Surface Using Lens Focusing

The present application is a continuation-in-part of the parent application Ser. No. 566,687 filed Dec. 29, 1983.

In any camera system, range to the object may be approximated for objects closer than the hyperfocal distance by observing the sharpness of focus of the image of the object (or a portion thereof) as the camera lens is moved or indexed across its focusing range. At the position of sharpest focus the range of the object (or portion thereof) may be read directly from a calibrated focus adjustment. This can be done whenever there is sufficient detail on the object surface or at its boundary to permit the focusing to be readily apparent to the human eye or to a processor which can determine sharpness of focus based upon fine detail. Sharpness of focus is determined when the spatial high frequency content of a region is maximized or when the percentage modulation at a region is maximized, or both. This can be achieved via many well known digital processing schemes which use the digitized output from a TV camera as input or alternatively, via the use of instantaneous holographic fourier transform techniques which instantly yield spatial frequency information. However, both techniques fail when the object has insufficient surface detail or its edges are gently curving so as to defy the requirements for the presence of some innate structure upon which to judge proper focusing.

The methods and arrangements described heretofore are for obtaining three-dimensional measurements of surfaces by observing the point at which a projected pattern was most sharply imaged. The projected pattern provides detail on a surface lacking sufficient surface detail to determine when the surface lies at the exact distance to which the projector and camera are focused. However, a surface may exhibit high curvature in certain areas. If linear patterns are projected such that the curvature is in the direction perpendicular to the projected lines, and an insufficient number of lines are able to be focused, then the distance measurement fails.

It is an object of the present invention to overcome the prior art disadvantages.

More particularly, it is an object of the present invention to measure the distance from a sensor to a surface which may be devoid of surface detail and provide the three-dimensional co-ordinates of points on the surface relative to the sensor.

A further essential object of the present invention is to measure the distance from a sensor to a surface with rapidly changing surface depth relative to the sensor, and to provide the three-dimensional co-ordinates of points on the surface relative to the sensor.

In keeping with this object, and with still others which will become apparent as the description proceeds, the invention comprises projecting a pattern of light (not necessarily restricted to visible electromagnetic radiation) onto the surface to be measured; receiving the reflected light from the surface through the same lens used for projection; imaging the reflected pattern on a light sensitive surface; reading out the pattern detected by the light sensitive surface; determining the location in the pattern that maximally correlates with a reference pattern or provides maximum percentage modulation; and reporting the co-ordinates, relative to the sensor, associated with that location within the pattern.

An alternate method of the invention separately projects a pattern of light upon the subject surface with a large depth of focus.

A third method of the invention separately projects a pattern of light upon the subject surface with a narrow depth of focus, the distance to focus being controlled to match the imaging system focal distance.

The present invention comprises, furthermore, optimizing the light for expected surface contours.

The invention will hereafter be described with reference to an exemplary embodiment, as illustrated in the drawings. However, it is to be understood that this embodiment is illustrated and described for the purpose of information only, and that nothing therein is to be considered limiting of any aspect of the invention.

FIG. 1a is an optical schematic of a plan view of an exemplary embodiment of the invention;

FIG. 1b is a side view of the embodiment of FIG. 1a;

FIG. 1c is an axial view of the embodiment of FIG. 1a;

FIG. 2 is a block diagram of the processing for an exemplary embodiment of the invention;

FIG. 3 illustrates an exemplary projection slide pattern and typical image received;

FIG. 4 is a block diagram of one method of processing the received image signal;

FIG. 5 illustrates typical voltage waveforms in processing the image;

FIG. 6 shows the unique dimensional mapping;

FIG. 7 shows a pattern applicable to the problem encountered with high surface curvature or slope relative to the sensor, in accordance with the present invention;

FIG. 8 shows an imaged pattern on a flat surface;

FIG. 9 shows an imaged pattern on a surface with high curvature orthogonal to the pattern;

FIG. 10 shows an imaged pattern on a surface with high curvature parallel to the pattern;

FIG. 11 shows a flat surface in a side view with the focal plane of the projected pattern intersecting the surface;

FIG. 12 shows a side view of a surface with high curvature.

The present invention provides a solution to the failure of prior art devices to measure distance to objects lacking surface detail. The requirement for object detail is obviated by projection of a light pattern containing detail onto the object surface. FIG. 1b shows the addition of projected pattern 125 to the prior art system that only used movable lens 13 with image sensing surface 17. The pattern 125 on slide 127 to be projected is illuminated by light source 120 with condensing lens 19. Light passing through transparent portions of pattern 125 is partially reflected towards lens 13 by beam splitter 121, the remainder of the light being absorbed by light absorbing surface 14. Lens 13 focuses the image of pattern 125 at a known distance from lens 13. An object 11 at that distance with a sloping surface 12 facing the device will have a sharp image of pattern 125 irradiating its surface at the in-focus distance 123 and a blurred image at further distances 122 or nearer distances 124 as illustrated in FIG. 1a of the system. Reflected light from the pattern projected on surface 12 is then imaged by lens 13 on light sensitive surface 17 of detection device 15. A portion of the light is lost due to the beam splitter 121. An axial view (FIG. 1c) of detector 15 is shown with the received image 18.

It is not necessary to use a single lens 13 and beam splitter 121 to project a pattern 125 upon object 11 though many important advantages are lost. A separate projector working in conjunction with a prior art device can add detail to a featureless surface 12 and thus enable the measurement of distances to points on its surface. The projector may be fixed focus with a depth of field adequate to encompass the measurement range of the distance measuring device or operate cooperatively with the distance measuring device to provide a sharply focused projection image at a distance equal to the in-focus distance of the measuring device. Axial motion 126 of lens 13 provides this simultaneous adjustment for both pattern projection and image reception in the embodiment shown in FIGS. 1a to 1c. This greatly simplifies this simultaneous adjustment and provides the increased accuracy (sensitivity) obtainable by using a sharply focused projected pattern. This sensitivity of focusing is increased because motion 126 of the lens 13 away from its true focus simultaneously defocuses the projection of detail 123 on the object 11 and defocuses the imaging 18 of the object on the pickup device 15. Thus, the sensitivity is doubled. It should be noted that the optical distance of the projection slide 127 from the beam splitter reflecting surface must exactly equal the optical distance from the beam splitter reflecting surface to the detection device surface 17 to ensure that optimum focus is achieved simultaneously for both the projection and imaging paths. Also, the internal optical elements, including the projection slide 127, must be carefully coated and internal structures treated to avoid internal reflections since internally scattered projector light reaching the pickup means 15 will degrade image contrast.

When using the described single lens system of FIG. 1, a processing advantage is obtained because the process of correlation against a known image or alternatively, spatial synchronous detection may be used to process the received image. This occurs because of the reciprocity properties of the optical path from pattern 125 through lens 13 to object 11 and back through lens 13 to detection surface 17. For example, if a flat white screen is set perpendicular to the lens optical axis at the in-focus distance 123, a sharp image of pattern 125 will appear on the screen. Due to lens effects, however, the image will be slightly geometrically distorted. However, since the image of the pattern 125 on the screen is imaged back through the same lens onto the detection device 15, it can be seen that the image on the detection surface 17 must be an exact duplicate of the pattern 125 except for any lens 13 defects that cause a blurring of the image. This occurs because sending the distorted image back through the same lens 13 cancels the distortion. Thus, the image on surface 17 from a flat screen at distance 123 is essentially identical to the projected pattern 125. For any position of lens 13 in the system's working range 126, the screen may be placed at the corresponding in-focus distance 123 and the pattern image onto the detection surface 17 would be invariant. This occurs because the magnification of the projection part of the system is equal to the demagnification of the detection part of the system.

Therefore, the image at the detection surface 17 is invariant except for the blur due to defocusing regardless of where an object is located in its field of view. For maximum distance sensitivity, the system is normally used with the aperture as wide open as possible. The preferred processing required to find regions of sharp focus consists of looking for fixed known patterns in each region of interest using correlation against the known pattern for that region or synchronous detection against the known pattern. Even if the object 11 reflecting surface 12 is not uniformly reflective or featureless, the same processing may be used to assist in focus determination since the image on detection surface 17 will consist of the spatial product of the object surface reflectance and the projected pattern. Thus, except for completely non-reflective object regions, the projected pattern (as modulated by object reflectance) will always appear at the detection surface 17 whenever the region in question is in focus with respect to the current lens 13 adjustment 126. The percent modulation found in each local region may also be used to determine optimum focus.

The projected pattern 125 may consist of any pattern that has mathematical properties providing ease of processing. Such patterns as dot arrays, horizontal bars, vertical bars, angled bars, concentric circles, etc. may be employed.

Similarly, to avoid the contrast diminishing effects of ambient light when using this process, it is anticipated that light source 120 will be a strobelight or a narrow band illumination device (e.g. a laser) with corresponding bandpass filter 129 in the detection path. A shutter 128 may also be used synchronized with light source 120.

Additionally, when a vidicon device which may introduce distortion in readout is used as detector 15, it is expected that a reference pattern will be generated by placing a flat screen at the system's focal distance 123 to derive the actual pattern required when performing the cross correlation or synchronous detection process.

FIG. 2 is a block diagram of one method of the processing required to determine the distance from the sensor 21 to the object 27. The measurement system consists of sensor 21 which has been detailed in FIGS. 1a to 1c, an analog to digital converter 23, memories 24 and 25 for the measured and reference scene data, computer 26 and controller 29. A pattern is projected from sensor 21 onto object 27 and the reflected light converted to a video signal 22 by a TV camera performing the detection function of sensor 21. A/D converter 23 digitizes the video signal which is stored in memory 24. During calibration, memory 25 stores a reference pattern as previously described. Computer 26 cross-correlates the data from memories 24 and 25 and provides output 28 to report the three-dimensional location information of all in-focus surface areas on object 27. Computer 26 also directs controller 29 to focus at a new distance to repeat the process until all surface areas have been measured. If percent modulation is used instead of cross-correlation then memory 25 is not required.

A more detailed explanation will be given using an analog processing method as shown in FIG. 4. Although an analog processing scheme is shown it should be understood that all of the processing may as easily be performed by computer 26 shown in FIG. 2. For the example given, the pattern 125 projected will be assumed to be horizontal bars as shown in (a) of FIG. 3. The small region 123 in focus on object 11 provides image 18 on detector 15, shown enlarged in (b) of FIG. 3. Pattern (a) will have a Video reference signal 41 that will look like waveform 52 in FIG. 5. The reference signal 52 is shown normalized to zero voltage level 51. Video Object signal 42 will have a waveform 54 corresponding to the narrow in-focus region 18 of the image and level 53 and 55 in the blurred, out of focus regions either side of 18. The processing shown in FIG. 4 is typical of both correlation and synchronous detection. The regional cross correlation process 43 does not require sliding the two signals relative to each other because the desired component of the input signal 42 is known to be synchronized to the reference 41. The product 44 of the signals will have a waveform 57 corresponding to the in-focus region 54 and waveforms 56 and 58 corresponding to regions 53 and 55, respectively. Low pass filter 45 output 46 will reveal the location of the in-focus region 18 as an increased level 520 relative to surrounding regions 59 and 521. A threshold circuit 47 can then make discrete decisions 523 for in-focus and 522, 524 for out-of-focus as output 48. The time of decision 523 relates directly to the location 18 on detector 16 which in turn, with the known position of lens 13 in its working range 126 and calibration data for the system, can provide the three-dimensional co-ordinates of the in-focus regions.

As shown in FIG. 6, the calibrated volume 61 will have a unique in-focus distance, X, measured from lens 63 at each specific location in its range of motion 626. Additionally each off-axis location such as A, B, C and D map to unique locations on the imaging surface 67 at a, b, c and d respectively.

Alternate processing, percentage modulation not shown, would operate by computing for each region 100(Vp-Vm)/(Vp+Vm) where Vp is the peak voltage of waveform 54 and Vm is the minimum voltage. This function is only a maximum in regions where sharp focus exists.

If the illumination pulse of arbitrary length can be sharply terminated and the detector shutter 128 opened several nanoseconds later, the intense reflections internal to device 21 and at distances closer than the object 11 can be eliminated from reaching the detector 15 and reducing the contrast between bright and dark portions of image 18 and between image 18 and surrounding out-of-focus images.

The preceding description presents in detail the methods of obtaining three-dimensional measurements of points on surfaces lacking surface detail using lens focusing. Problems of surface curvature or slope relative to the sensor were not discussed. FIGS. 7 to 12 illustrate the problem encountered with high surface curvature or slope relative to the sensor. FIG. 7 shows a pattern suggested above and FIG. 8 shows the expected image for the arrangement shown in 11. FIG. 11 shows a flat surface 213 in a side view with the focal plane 214 of the projected pattern intersecting the surface. The pattern lines that strike surface 213 in the region near the intersection with focal plane 214 will be in relatively sharp focus and are imaged as lines 211. The remainder of the pattern will be out of focus such as lines 212. As already described above, the in-focus lines can be automatically detected and related to distance from the sensor to provide the means of obtaining 3-D measurements. However, if the number of pattern lines 211 in focus are too few as in FIG. 9, then the measurement will not be obtained reliably. Light intensity may be brightest in the region of best focus, but reflectivity variations can confuse this criteria. The number of lines in focus 211 can become too small if surface 213 is at a steep angle or local surface curvature is very great such as ridge 215 of surface 213, as seen in side view of FIG. 12.

The present invention resolves this problem by providing patterns optimized for obtaining data from such surfaces. For example, if the region of high curvature (ridge 215) extends across surface 213 in a direction parallel to projected pattern 210, giving image 216, then by rotating pattern 210 ninety degrees, image 217 will be obtained. The in-focus parts 218 of the projected lines will contain sufficient modulation depth to provide the identifying characteristic needed for automatic measurement.

Likewise, for any known surface type, optimum patterns can be projected to assure reliable mensuration. For unknown surface types, various patterns and/or orientations can be used to obtain data on the surface independent of orientation of the surface features.

The invention has been described and illustrated with reference to an exemplary embodiment. It is not to be considered limited thereto, inasmuch as all modifications and variations which might offer themselves are intended to be encompassed within the scope of the appended claims.