Robotic Vision System

The present invention relates to robotic vision systems. More specifically, the present invention relates to a robotic vision system that can identify the position of objects in real time with the use of a fiduciary code.

Most current industrial robots are relatively simple devices. The robot is "trained" by having a human operator guide it through its duty cycle. The motion is recorded and the robot just replicates this motion. This is an open loop process. The robot is incapable of correcting any positional or placement errors of objects upon which it operates. The solution to this problem is to provide the robot with a sensory system to take data which it can analyze and the use the results to modify its motion, thereby providing a closed loop system. In general, the basic goal of all current industrial robot research efforts is to develop robotic systems which will enable robots to replace humans in performing monotonous and/or dangerous jobs.

Of all five senses, vision accounts for roughly 70% of the information used. On-line industrial robots do not presently have vision capabilities. They perform manufacturing operations by rote memory where the required operations are learned by moving the robot through a learning cycle. The basic requirements of a computer vision system is that it be

(i) inexpensive,

(ii) provide a simple input image,

(iii) do processing in real time,

(iv) have a low error rate, and

(v) be flexible, i.e. be able to accommodate changes in products.

See John F. Jarius, "Visual Inspection Automation," Computer, Vol. 13, No. 5, May 1980, pp. 32-39. John Birk and P. Kelley, "Research Needed to Advance the State of Knowledge in Robotics," National Science Foundation Workshop on Robotics, April 1980, Newport, R.I., pp. 48-52. Current research efforts at providing robots with vision involve (a) using a camera to obtain an image, (b) digitizing this analog information, (c) using a computer to analyze the information via a gray scale and contrast change algorithm to obtain edges and vertices, and (d) using a matching process to identify and obtain the orientation of object. The goal is to accomplish image segmentation--namely to recognize and locate objects For example, in an industrial assembly operation the system would have to (a) identify and locate all the parts, and (b) put all the parts together correctly. The system would thus have to know the position of a robot arm in real time. The real time requirement means one which requires fast algorithms. One current method is to use a backlighted scene in which the parts are black against a white background. The disadvantages of this method are that (a) it only gives a two-dimensional image, and (b) it can only be used if the shape of the objects are significantly different. More advanced methods under investigation involve using a spatial gray level dependence method. This method has demonstrated its ability to analyze relatively complex scenes. However the algorithms used take an unreasonable amount of time on current microcomputers. In addition, they can readily be "fooled" into making false identifications. Algorithms to avoid this problem require even longer processing time requirements. Bin picking systems relying only on remote sensing have not yet been demonstrated. See R. B. Kelley, H. A. S. Martins, J. R. Birk and J. D. Dessimoz, "Three Vision Algorithms for Aquiring Workpieces from Bins," Proceedings of the IEEE, Vol. 71, No. 7, July 1983, pp. 803-820.

There is thus a clear need for computer vision systems which are capable of dealing with scenes which are not entirely predicted and/or structured. S. Tsuji et al., "Panel on Future Direction of Industrial Applications," Proc. Fourth Int'l Joint Conf. Pattern Recognition, 1978 pp. 1143-1149. The goal of most visual scene analysis programs is to be able to identify all the objects in a scene regardless of orientation, with parts of some objects occluded by other parts, and under varying lighting conditions. Computer programs that can barely cope with simple scenes under these conditions are huge, difficult to code or modify, time consuming and require very large computers. John F. Jarius, "Visual Inspection Automation, "Computer, Vol. 13, No. 5, May 1980, pp. 32-39.

Most current systems use a Vidico or CCD visual sensor, various algorithms to determine edges and corners and then by comparing to an internal model, determine the object and its orientation.

Current techniques are too slow for real time processing and have difficulty determining 3-D structure against the changing backgrounds which occur as the object moves on an assembly line. Harry G. Barrow and Jay M. Tenenbaum, "Computational Vision,"Proceedings of the IEEE, Vol. 69, pp. 572-595, May 1981; J. K. Aggarwal, Larry S. Davis, and Worthy N. Martin, "Correspondence Processes in Dynamic Scene Analysis," Proceedings of the IEEE, Vol. 69, No. 5, May 1981, pp. 562-572; Gerald J. Agin, "Computer Visions Systems for Industrial Inspection and Assembly," Computer, Vol. 13, No. 5, pp. 11-20; M. Ejiri, T. Uno, Hy Yoda, T. Goto and K. Takeyasu, "A Prototype Intelligent Robert that Assembles Objects from Plan Drawing," IEEE Trans. Computers, Vol. C-21, No. 2, February 1972, pp. 161-172. While the speed requirement is somewhat nebulous, it is probably convenient to place an upper limit of one second for a effective system to process one frame of image data. Anything slower than this time will have a hard time finding a place in any manufacturing application. TV cameras operate at rates on the order of 60 frames per second. Image processing that keeps up with this rate can be though of as "real time".

Accordingly one object of the present invention is to provide a novel and unique robotic vision system which is inexpensive.

Another object of the present invention is to provide a novel and unique robotic vision system that has a low error rate.

Another object of the present invention is to provide a novel and unique robotic vision system that is able to accommodate changes and variations in the products it is working upon.

These and other objects of the present invention are achieved with an apparatus for robotic vision which enables an object to be moved by a robot, whether the object is situated correctly or not, comprising, a fiduciary code which identifies the position and orientation of the object to be moved by the robot, the fiduciary code being luminescent; luminescent means for causing the fiduciary code to luminesce; sensing means for detecting the position of the fiduciary code; computing means for determining the position and orientation of any portion of the object in relation to the robot in real time by subtracting from the sensed luminescent image the sensed non-luminescent image (or vice-versa), thereby providing an image of the fiduciary code only, which can then be used as input for the computing device, thereby providing a closed loop which will enable the fiduciary code on the object and the robot to be put into a predetermined desired position and orientation.

A more complete appreciation of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

FIG. 1 is a schematic drawing of a robotic vision system.

FIG. 2 is a schematic drawing of a robotic vision system with a robot arm.

FIG. 3 is a schematic drawing of another robotic vision system.

FIG. 4 is a schematic drawing of the robotic vision system shown in FIG. 3, with a robot arm.

Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, and more particularly to FIG. 1 thereof, there is disclosed a robotic vision system 10. The robotic vision system is comprised of a work station 12 along which an object 14 is transported. The object 14 has a luminescent fiduciary code 16 located thereon that luminesces when an ultraviolet (UV) or infrared (IR) light source 18 irradiates the fiduciary code 16 of dots. Ultraviolet light and infrared light are two forms of nonvisible luminescent light. An electrically operated shutter 20 located between the UV light and the work station 12 blocks the UV or IR light during normal work station lighting exposure intervals. Alternatively one can use a UV or IR light that can be turned on and off at will. Video cameras 22 receive the light from the scene, including that from the fiduciary code 16 and pass that information via a connection (not shown) to an interface board and flash convertor 24. The convertor 24 takes the signal from the video cameras 22 and converts it into a signal that a computer 26 can manipulate and store. A main computer 26 substracts off the image without luminescence seen by the cameras 22 from the overall image with luminescence, leaving just the fiduciary code 16 (or vice-versa).

More specifically, luminescent fiduciary coded markings or dots 16 to locate the position of components are used rather than visual edge and corner identification as is used in the prior art. An industrial processing station using the robot vision system for 3-D imaging would employ 2 videocons or CCD cameras 22. These would be used to view the work station 12 and the incoming material 14. The incoming material 14 to the work station 12 would be marked with luminescent marks 16 that would emit a frequency of light which would stand out from the work station 12 when illuminated with ultraviolet or infrared light. For certain operations wherein full 3-D imaging is not required, a single videocon may be utilized

There are several ways one could use this procedure. In one method the image would first be recorded in background light. The ultraviolet or infrared light source 18 would be turned on, then off (or vice-versa). A second image would then be recorded while the light source 18 is on. By subtracting the first image from the second image one would be left with the image of just the dots. By equipping the CCD cameras or Videocons 22 with narrow band light filters (not shown) the luminescent marks would obviously stand out from the background. An alternate procedure, depending on the work environment, would consist of keeping the ultraviolet or infrared light 18 on constantly, using narrow band light filters for the luminescent dyes chosen and having the CCD cameras or Videocons 22 look for the fluorescent marks 14. In either case the position of the marks would be stored in the memory of the host computer 26 at specific memory array locations corresponding to the pixel location from the camera equipment 22. Two cameras 22 would be used to develop three dimensional imaging ability. The stored pixels from the video equipment 22 can then be used as input data to algorithms which would find the exact location and orientation of the material 14 at the work station 12.

This method provides "real time" analysis of motion since it is only necessary to determine the motion of these easily observed luminescent fiduciary marks 16. For example, by observing the position of three marks not on a line, there would be an overly determined set of equations for determining the position of the object 14. The problem of determining the position of objects from the motion of points has been well studied See J. W. Roach and J. K. Aggarwal, "Determining the Movement of Objects from a Sequence of Images," IEEE Trans. Pattern Anal. and Machine Intell., Vol. PAMI-2, No. 6, November 1980, pp. 554-562; Re L. G. Roberts, "Machine Perception of Three-Dimensional Solids," in Computer Methods in Image Analysis, J. K. Aggarwal. R. 0. Duda, and A. Rosenfeld, Eds New Turk: IEEE press, 1977, pp. 285-323. It should also be noted that there is no problem with changing shadows due to changing lighting and viewing conditions.

FIG. 1 shows a work station 12 along with the video equipment 22 and associated computer 26. The system operates in real time because of the reduced computational work load that results from the method of finding the location of the work material 14 directly. The method, applied to the three dimensional case eliminates the need of two simultaneously processed algorithms to find edges and corners, and the correspondence problem. The gray scale to the video equipment 22 is used to control the contrast of the background, and is handled by the computer. To find the pixels in memory it is then only necessary to search the memory array looking for high contrast memory locations.

There appears to be two fairly straight forward ways of achieving image location and orientation by using codes which luminesce due to ultra-violet or infrared illumination. The first is to simply use color filters and set the gray scales of the video or CCD camera 22 to a level at which fiduciary marks will exhibit an extremely high level or contrast. A high speed search of computer memory can pick up these fiduciary marks. While this method seems simple it does place some restriction on the luminescent dyes and on the background lighting

A second method involves pulsing the ultra-violet or infrared source and reading the scene twice; (a) during, or (b) shortly thereafter and at several luminescent decay times later between each pulse. The two readings as stored in memory would be different, and the main scene times should then cancel out, leaving the luminescing codes at very high contrast The luminescence of the codes decays with time, and therefore the technique of comparing the memory for the two separate readings taken at two different times from the excitation pulse provides the required high contrast information. For example, the object and robot could be irradiated for no more than 1 second.

Because of the video equipment 22 setup there would be a one-to-one correspondence between these memory locations and material 14 location and orientation at the work station 12. The location of specific surface features upon which an industrial operation is to be performed would be stored in memory relative to the marking codes. Thus, the robot will be able to find the location and the orientation of the object, and using this information would be able to perform the desired industrial operation.

The camera 22 position and orientation form the reference upon which measurements of the work material are made and therefore need to be carefully controlled. If the work station 12 itself was also arranged with fiduciary marks in the same manner as the work material 14, then all measurements could be made with respect to the work station 14, and the requirements on camera 22 position and orientation could be reduced.

FIG. 2 shows a second feature that is also present. Fiduciary codes 34 could be used to mark a robot 36. These codes could be chosen to exhibit a different spectrum, from that of the material at the work station, when illuminated by ultra-violet or infrared light source 18. This pixel information would then be used for final feedback information to position the robot itself for the preprogramed industrial operation. This method, by providing a closed loop, should serve to increase the allowed accuracy of the robotic system. In both FIGS. 1 and 2 the information coming in from the video equipment 22 can be displayed as the location of the robot and the material to be processed with respect to some reference point. As is well known, See R. A. Jaris, "A Perspective on Range Finding Techniques for Computer Vision," IEEE Trans Pattern Anal. Machine Intell No. 1, PAMI-5, No. 2, March 1983, pp. 122-139 the farther apart the two camera positions are, the potentially more accurate are the depth calculations, but the more prevalent is the missing parts problem and the smaller is the field of view overlap. Since fiduciary marks are being looked for, since there could be several fiduciary marks per item, and since the position of all the fiduciary marks and their relationships to each other are known by the computer, the proposed technique is less sensitive to this type of problem than the alternate procedures. As an example, the optimum separation will probably depend on the distance of the cameras from the object, and the task to be performed.

The advantages of the present technique are numerous First, it could be readily applied to assembly operations. Such operations as mounting wheels or assembling doors on vehicles seem to be tailor-made for this method. Different parts could be identified by different colored fluorescent fiduciary marks. Color identification could easily be incorporated into the visual system. Different fiduciary marks, dot codings, sizes, shapes and/or colors could then be used to determine orientation.

For example, three marks, not in a line, perceived by two different cameras, coupled with the knowledge of the location of these marks on an object (stored in computer memory) will enable one to obtain the location and orientation of a specific part. Similar fiduciary marks placed on the robotic arm could be used to give its position and orientation. With this information relatively straightforward algorithms could be developed for specific application, incorporating real-time feedback of arm motion object position.

Secondly, the method could be used in visual inspection applications. The fiduciary marks would be used to determine the orientation of the part to be checked. The part could then be obtained by the robot arm, oriented to a specific orientation, could proceed.

Thirdly, the method could be used to provide a passive system for mobile robots, or smart wheelchairs to identify passageways, doorways, shelves, etc. Namely by painting specific fiduciary marks on these stationary items, and using the mobile unit to illuminate them in a similar manner as described, they could be readily identified. For example, a mobile robot would "know" exactly where it was, and from a map stored in computer memory, compute how it has to move to reach a programmed location. This procedure would enable general purpose robots to locate specific doors, passageways, etc. A mobile robot of this type would be very useful in nuclear or chemical plants where one would want to test chambers for potentially dangerous contamination. In order for the method just described to locate a test object the system must be able to visually determine the Cartesian coordinates of this test object in a system measured from an arbitrary reference, and at the same time, find the orientation (Euler angles ) of this same object. The system will use this information to position a robot hand at a presepecified location on the test object.

In an actual manufacturing operation this information on location and orientation of the subject would be used to grasp and possibly reorient a specified part.

One preferred embodiment of many possible embodiments consists of using Quantex CX-100 video cameras 22 that feed a Quantex analog board 24 which converts the video cameras 22 signal from analog to digital. The digitalized signal is then passed to a Quantex QX 9200 computer 26 that locates the position of the fiduciary codes 16 and 34 on the object 14 and the robot 36, respectively. The computer 26 then sends a signal to a Fared Motoman L10 Robot 36 causing the fiduciary codes on the object 14 and robot 36 to be put into a pre-determined desired position and orientation in real time. Normal lighting 38 may be used as needed in the work station 10.

Another robotic vision system 20 shown in FIG. 3 differs from system 10 shown in FIG. 1 is that a single video camera 22 is employed in system 20 instead of the two video cameras 22 employed in system 10 receives the light from the scene, including that from the fiduciary code 16 and passes that information via a connection (not shown) to an interface board and flash convertor 24. The convertor 24 takes the signal from a computer and converts it into a signal that a computer 26 can manipulate and store. A main computer 26 subtracts off the image without luminescence seen by the camera 22 from the overall image with luminescence, leaving just the fiduciary code 16 (or vice-versa).

Robotic vision system 20 (shown in FIG. 3) combined with a robot arm as shown in FIG. 4 includes fiduciary codes 34 marking a robot 36 and differs from system 10 (shown in FIG. 2) in that a single video camera 22 is employed in system 20 instead of two video cameras employed in system 10.

Obviously many modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described.