Method and apparatus for positional error correction in a robotic pool systems using a cue-aligned local camera

[0001] This patent application relates to, and claims the priority benefit from, U.S. Provisional Patent Application Ser. No. 60/612,247 filed on Sep. 23, 2004 entitled METHOD AND APPARATUS FOR POSITIONAL ERROR CORRECTION IN A ROBOTIC POOL SYSTEM USING A CUE-ALIGNED LOCAL CAMERA, and which is incorporated herein in its entirety.

[0002] This invention relates generally to the field of robotically controlled games, and more particularly the present invention relates to a method and apparatus for robotically controlled pool games.

[0003] The cue sports include pool, billiards, and snooker, and have recently enjoyed a surge in popularity worldwide. There have been a number of efforts at developing automation aids for these games. One example is the Instant Pool Trainer [Lar01], in which a camera is mounted on the ceiling aimed down at the table. Acquired images are transmitted to a computer and automatically analysed using image processing techniques. The system makes suggestions to the human trainee about the next shot to place, the desired angle of the cue, etc.

[0004] Other systems have attempted to fully automate the play by adding a robotic component [Nak01, Qi99, Shu94, Chu02, Ali04, Lon04, Che04]. In addition to the ceiling-mounted camera, these systems involve some form of computer-controlled robotic actuation device that can position a cue to the correct location and place a shot. The most common example of such robotic devices are gantry systems [Ali04, Lon04, Che04, Shu94], the first of which was proposed by Shu et al. [Shu94]. Other proposed robotic devices include a mobile robot that moves around the perimeter of the table and extends a cue-like end-effector to place a shot [Qi99], and a mobile robot that moves over the surface of the table [Lar02].

[0005] The cue sports demand a high degree of positional accuracy when placing a shot, and one of the main challenges of a robotic system is to position the cue to the desired location with sufficient accuracy. The exact positional accuracy that is required to play well has not been reported in the literature, and is presumed to be unknown, although it is likely to be on the order of 0.1 mm or finer. Whereas mechanical devices can be positioned very precisely, both sensor errors and robot calibration contribute to limitations to positioning accuracy of such systems.

[0006] In the above cases where overhead cameras are the primary sensor to resolve position, a limitation to accuracy is sensor resolution. Standard CCD cameras that are suitable for machine vision applications will often have 640×480 pixels. If the entire length of a standard pool table extends the complete 640 pixels, then this resolves to ˜4 mm/pixel, which is at least an order of magnitude too coarse. Using higher pixel-count sensors, or multiple cameras, are possible remedies. In the case of multiple sensors, each of which images a smaller region of the table at a higher magnification, combining the partial images acquired by each individual sensor into a global coordinate frame requires accurate calibration of the extrinsic camera parameters. Radial distortions in the optical systems also limit the accuracy of the cameras. A further limitation is that, from the overhead vantage, the table appears as a 2-D plane, and vertical displacements of the cue (i.e., normal to the camera place) cannot be perceived. Controlling these vertical displacement to allow the system to strike the cue ball high or low forms an important part of the play.

[0007] The main limitation to positional accuracy is calibration of the robotic device [Lon04]. The proper calibration of robotic devices to ensure positional accuracy is a well-known and challenging problem. The majority of robotic devices are equipped with joint encoders that very precisely measure the location of each revolute or translational joint. Despite their precision, converting these joint values into an accurate position of the robotic end-effector is not straightforward, as there are a number of factors that cannot be directly measured which affect the overall accuracy. The majority of industrial robotic devices do not require absolute positioning accuracy, so long as they are precise and repeatable, so this limitation on accuracy does not present a barrier to use in many cases. An exception where absolute positioning accuracy is required are Coordinate Measurement Machines (CMMs). These devices are finely machined and calibrated so that they can be used in metrological inspection applications. The delicate mechanisms used in CMM construction would unfortunately not be able to withstand a significant load or impact, such as striking a ball.

[0008] In human play, it is an accepted practice to accurately align the cue prior to a shot by locating the eye closely to the axis of the cue [Kan99]. From this vantage, the locations of both the cue ball (which is to be impacted by the cue) and the object ball (which is to be impacted by the cue ball) can be seen. Small positional variations of the cue axis as well as of the tip of the cue can be perceived accurately, as they are parallel to the human's retinal plane. Conversely, motions that are perpendicular to the retinal plane, such as those parallel to the cue axis, are less easily resolved, and are fortunately less important to accurate shot placement.

[0009] Therefore, there is a need to provide a robotically controlled pool game which overcomes the aforementioned shortcomings.

[0010] The present invention provides a method and apparatus for accurately positioning a robotic pool-playing device. The system comprises a computer controlled robotic positioning device, such as a gantry robot, that can position a cue over the pool table and place a shot. A global camera is mounted on the ceiling looking down at the table, and the acquired images are transmitted to the computer for analysis to determine the identity and locations of the balls within the table coordinate reference frame. The computer also automatically determines which ball to strike.

[0011] The invention may use a local camera, mounted on or near the robotic end-effector in a fixed relationship with the cue, to improve the positioning error of the robotic device prior to placing a shot. By comparing the ball locations perceived from the vantage of the local camera with the known ball locations determined from the global camera image, the invention can calculate the acquired robotic positioning error, which can then be corrected for prior to placing the shot.

[0012] Thus, in one aspect of the invention there is provided a method of playing pool, comprising the steps of:

[0013] a) acquiring an image of a pool table using a first camera placed above the pool table and positioned that its image plane is substantially parallel to both a playing surface of the pool table and a longitudinal axis of a pool cue and transmitting this image to a computer processing means for analysis, the result of which is a determination of the position and identity of each pool ball on the pool table;

[0014] b) said computer processing means planning a shot of a cue ball by the pool cue by calculating a desired position of the pool cue with respect to the cue ball in a pool table frame of reference including a tip position of said pool cue and orientation of the longitudinal axis of the pool cue;

[0015] c) said computer processing means instructing said robot connected to said pool cue to position the pool cue to a preferred location from which to place a shot; and

[0016] d) said computer processing means instructing the robot to place a shot.

[0017] In another aspect of the present invention there is provided a method of playing pool, comprising the steps of;

[0018] a) acquiring an image of a pool table using a camera mounted on the robotic end-effector so that its image plane is substantially perpendicular to a longitudinal axis of the pool cue, the position of said camera being fixed with respect to the longitudinal axis of the pool cue, transmitting the image to a computer processing means for analysis, the result of which is a determination of the position and identity of each ball on the pool table within this image within a frame of reference of the camera, and correcting for any errors based upon the relative position of the cue axis and the cue ball;

[0019] b) said computer processing means planning a shot of a cue ball by the pool cue by calculating a desired position of the cue with respect to the cue ball; and

[0020] c) said computer processing means instructing the robot connected to said pool cue to place a shot.

[0021] The present invention also provides an apparatus for playing pool, comprising:

[0022] a) a robot having a robotic arm with an end effector, the robot mounted on a gantry for movement above a pool table, a pool cue attached to the end-effector, the pool cue having a longitudinal axis;

[0023] b) a first image capturing means mounted above the pool table for acquiring an image of the pool table and positioned such that its image plane is substantially parallel to both a playing surface of the pool table and said longitudinal axis of said pool cue;

[0024] c) a second image capturing means mounted on the robotic end-effector so that its image plane is substantially perpendicular to the longitudinal axis of the pool cue for capturing an image of a pool table using a camera, the position of said camera being fixed with respect to the longitudinal axis of the pool cue;

[0025] d) a computer processing means connected to the first image capturing means for analysis of the images transmitted thereto from the image capturing means, the computer processing means including means for determining a position and identity of each pool ball on the pool table within this image within a frame of reference of the image capturing means, said computer processing means including means for planning a shot of a cue ball by the pool cue by calculating a desired position of the cue with respect to the cue ball; and

• • [0026] e) a robotic controller connected to said computer processing means for instructing the robot to place pool shots with the cue.

[0027] In this aspect of the invention the apparatus may include a second image capturing means mounted on the robotic end-effector so that its image plane is substantially perpendicular to the longitudinal axis of the pool cue for capturing an image of a pool table using a camera, the position of said camera being fixed with respect to the longitudinal axis of the pool cue.

[0028] The present invention also provides an apparatus for playing pool, comprising;

[0029] a) a robot having a robotic arm with an end effector, the robot mounted on a gantry for movement above a pool table, a pool cue attached to the end-effector, the cue having a longitudinal axis;

[0030] b) image capturing means mounted on the robotic end-effector so that its image plane is substantially perpendicular to a longitudinal axis of the pool cue for capturing an image of a pool table using a camera, the position of said camera being fixed with respect to the longitudinal axis of the pool cue,

[0031] c) a computer processing means for analysis of the images transmitted thereto from the image capturing means, the result of which is a determination of the position and identity of each ball on the pool table within this image within a frame of reference of the camera, and correcting for any errors based upon the relative position of the cue axis and the cue ball, said computer processing means planning a shot of a cue ball by the pool cue by calculating a desired position of the cue with respect to the cue ball; and

[0032] d) a robotic controller connected to said computer processing means for instructing the robot to place pool shots with the cue.

[0039] The apparatus and method of the present invention utilizes at least one, or two machine vision systems. Illustrations of the system are shown generally at 10 in FIGS. 1 and 2.

[0040] In one embodiment of the invention a single, global overhead camera is used. In this embodiment the system 10 uses global overhead camera 12 for acquiring an image of pool table 14 with camera 12 being placed above the pool table 14 and positioned such that its image plane is substantially parallel to both the pool table 14 and a longitudinal axis 23 of pool cue 22 and transmitting this image to a computer 16 for analysis, the result of which is a determination of the position and identity of each ball 18 on the pool table 14.

[0041] Another computer program is used to plan a shot, which indicates a desired position of the cue 22 with respect to the cue ball in the table frame. The gantry robot 26 mounted above the pool table 14 is controlled by a robotic controller 32 that communicates with a computer 16 and the cue 22 is attached to the end-effector 28 whereby the cue is robotically controlled.

[0042] The shot planning computer program analyses the positions of the balls 18 on the table, and uses geometric and physics computations to determine the likelihood of sinking each target ball 18. Depending upon the arrangement of balls 18 on the pool table 14, some balls may be impossible to sink, whereas other balls may have a number of possible shots. The program not only considers the current arrangement of balls 18 on table 14, but also predicts the resulting arrangement of the balls 18 subsequent to placing each shot. The determination of which shot to place will therefore also include the most favorable positioning of remaining balls 18 in future shots.

[0043] Once the most favorable shot has been determined, the desired cue 22 position to place that shot is transmitted from the computer 16 to the robotic controller 32 to reposition the cue 22. The desired cue position includes not only the tip position 25 of cue 22, but also the orientation of the cue axis 23. The robotic system is then invoked to reposition the cue to the desired cue position, and place a shot.

[0044] One of the main challenges is that the pool games require a high degree of positioning accuracy over a large area. It is possible to design and calibrate a robotic system that is highly accurate (such as a CMM machine), but for a variety of reasons (ruggedness, cost) it is more desirable for us to use a standard robotic gantry that is inherently inaccurate. As an example, it is not uncommon for gantry robots to have accuracies only on the order of 0.01 m, which is clearly far too coarse to play a reasonable game of pool.

[0045] Thus, in repositioning the cue 22, the robotic system may accumulate error, as illustrated in FIG. 3. The actual position of cue 22 may therefore not be perfectly accurate, but rather includes some degree of positional error, which will degrade the performance of the system. In order to compensate for the accumulated positional error, another embodiment the system comprises two cameras, the global camera 12 mounted on the ceiling aimed down at the table 14 and another camera 30 mounted on the end-effector (or cue 22) so that its image plane is substantially perpendicular to the cue axis 23. It is difficult to position the camera 30 so that it is exactly perpendicular to the cue axis 23. It is also unnecessary, as nearly perpendicular is sufficient.

[0046] Generally in this embodiment the apparatus for playing billiards can accurately position the inherently inaccurate gantry mechanism by making use of information from both vision systems 12 and 30. First, the camera 12 determines the ball identities and locations accurately, and the planning system identifies a shot. The gantry robot 26 then moves the cue 22 into position to place the shot, and in doing so, accumulates error, so that the position of the cue 22 is inaccurate. To improve the cue accuracy, the camera 30 next acquires an image, and extracts the locations of landmarks (i.e., balls) from its own vantage. By comparing the location of the landmarks in the frame of reference of camera 30 with those in the frame of reference of camera 12, and given prior knowledge of the cue location in the frame of reference of camera 30, the error in the gantry robot 26 position can be estimated, and its position refined. In this way, the use of two cameras can be used to accurately position an inherently inaccurate positioning device.

[0047] Another possibility is to use landmarks (or targets) other than the pool ball positions to estimate and correct for the robot positioning error. In a preprocessing step, a number of targets, such as identifiable planar patterns, can be positioned at specific accurate locations with respect to the pool table. Their positions can therefore be identified within the frame of reference of the overhead camera 12. Once the cue 22 has been positioned, then the target locations can be identified within the frame of reference of the end-effector mounted camera 30, and the positional error of the cue 22 can then be estimated.

[0048] We assume that all ball 18 and “robot position(s)” are expressed in some global coordinate frame, which we shall call the table frame 38. It will be appreciated by those skilled in the art that due to the characteristics of the robotic mechanism, this “robotic position(s)” can be equivalently expressed in other terms without any ambiguity, i.e., as robot joint values, or as the position of certain other points on the robotic end effector. We therefore often talk about robotic positions in a slightly more general sense.

[0049] The position of this camera 30 is fixed with respect to the longitudinal axis 23 of the cue, and the location of the cue axis 23 within the coordinate reference frame 40 of the local camera 30 is determined in a pre-processing step, which uses a third camera 42 (shown in FIG. 1). This third camera 42 is mounted on a tripod 36 such that its image plane is also substantially perpendicular to the cue axis 23. The cue 22 itself may be repositioned with one degree-of-freedom by sliding along the cue axis 23, and this is done repeatedly and the center of the cue tip 25 within the third camera 42 image is determined at each extreme cue position. By comparing the centers of the cue tip 25 at each extreme position, the angle between the cue axis 23 and the third camera 42 is computed. The robotic joints are then repositioned so as to improve the perpendicularity of the cue axis 23 with the image plane of the third camera 42, and the process iterates.

[0050] Once the cue 22 is thus positioned so that its axis 23 is accurately perpendicular to the image plane of the third camera 42, the relative locations of the third camera 42 and the local camera 30 (which are substantially parallel) are then determined by estimating the rigid transformation that relates them, and the cue 22 location is then directly propagated to the local camera 30 coordinate reference frame.

[0051] The rigid transformation that relates the relative locations of the third camera 42 and the local camera 30 can be determined by first calibrating the intrinsic parameters of each of these two cameras, and then acquiring images of a common planar target of a known dimension from both cameras. The locations of features on the planar target can be used to determine the rigid transformation between the cameras, as disclosed in [Zha98].

[0052] After the cue 22 is repositioned, and before placing a shot, an image is acquired from the vantage of the local camera 30. This image is transmitted to the computer 16 and the locations of the balls 18 within this image are determined within the frame of reference of local camera 30. If at least three balls 18 are visible from both the global camera 12 and local camera 30, then the accumulated positional error of the robotic system can be estimated and corrected for.

[0053] For each ball 18, a ray can be inscribed between the optical center of one of the cameras 12 or 30 and the center of the ball 18. With three balls 18 and two cameras 12 and 30, there are a total of six such rays, i.e., three rays for each camera. By establishing the correct correspondence between pairs of rays from each respective camera, the relative positions of both cameras, with respect to each other can be determined. Equivalently, if the positions of the balls 18 and the global camera 12 are known within the table 14 coordinate reference frame, then the position of the local camera 30 can be determined with respect to the balls 18.

[0054] If there are more than three balls 18 in the scene, then the solution is over-determined, such that more than one solution exists. In this case, the position of the local camera 30 can be determined from the set of possible solutions using least squares estimation methods, such as singular value decomposition. If there are less than three balls 18, then the solution is underdetermined, such that not all position parameters (i.e. dimensions) can be solved. Even in the underdetermined case, it is still possible to estimate and correct some dimensions of the positional error using this technique. For example, with two balls 18, the positional error can be corrected up to a twist around the cue axis 23. With only one ball 18, the translational error can be corrected, although the direction of the shot may still contain errors. In these last two cases, it is also possible to reposition the end-effector so that the local camera 30 which is rigidly mounted to the end-effector is also repositioned so that the local camera 30 can view a sufficient number of balls 18. As small motions of the robotic device only accumulate small amounts of error, the cue 22 can then be repositioned accurately to place the desired shot.

[0055] An example of the process of error correction is illustrated in FIGS. 4(a) and 4(b). FIG. 4(a) shows the true and desired ball locations from the vantage of the local vision system image. It can be seen that positional error causes the true ball positions to differ from the desired ball positions. FIG. 4 (b) shows the true ball locations after the positional error has been corrected. After correction, the true ball positions align more accurately with the desired ball positions, leading to a more successful shot.

[0056] Another possible way to make use of the local vision system to improve accuracy is to first position the robot so that the cue 22 is at an ideal location with respect to the cue ball 18′ and object balls. In this position, a cue ball 18′ when struck by the cue 22 will then proceed to strike an object ball 18″ so that the trajectories traversed by the cue ball 18′ (prior to striking the object ball) and the object ball 18″ (after it has been struck by the cue ball 18′) are collinear. This represents a straight shot, and any other shot can be achieved by first orienting the cue 18 in this position, and then perturbing it slightly to a desired angle and offset. The locations in the local camera 30 reference frame of the cue ball 18′ and object balls 18″ for an ideal straight shot can be recorded. In subsequent shots, the robot joints can be positioned so that the cue ball 18′ and object ball 18″ fall in these previously recorded positions within the local camera 30 reference frame, thereby ensuring that the cue 22 is in the ideal position for a straight shot.

[0057] Utilizing an end-effector mounted camera is known in the robotic literature as an eye-in-hand system. There have been a number of such systems proposed to improve robotic accuracy for grasping and other operations, including some work that includes combinations of both end-effector mounted and global cameras [Fla00]. Despite the great interest and prior work in robotic pool, the use of a global camera 30 parallel to the table 14 plane in combination with a local camera 30 perpendicular to the cue axis 23 has not been previously proposed, and is advantageous and novel to this invention.

[0058] Another embodiment of the present invention utilizes only the local camera 30 to determine the quality of the cue 22 position solely with respect to the vantage of the local camera 30 image. In this application, the cue 22 is controlled by either a human or a robot, and the image would be transmitted, possibly wirelessly, to a computer. The cue 22 position is determined with respect to the imaged balls 18 as either good quality or poor quality, and this information is communicated back to the positioning device for refinement. While the vantage of the local camera 30 alone is more limited than that of the combination of local camera 30 and global camera 12, the flexibility of providing a self-contained cue 22 allows it to be used for human training as well as robotic play.

[0059] As used herein, the terms “comprises”, “comprising”, “including” and “includes” are to be construed as being inclusive and open ended, and not exclusive. Specifically, when used in this specification including claims, the terms “comprises”, “comprising”, “including” and “includes” and variations thereof mean the specified features, steps or components are included. These terms are not to be interpreted to exclude the presence of other features, steps or components.

[0060] The foregoing description of the preferred embodiments of the invention has been presented to illustrate the principles of the invention and not to limit the invention to the particular embodiment illustrated. It is intended that the scope of the invention be defined by all of the embodiments encompassed within the following claims and their equivalents.

• [0061] [Lar01] Larsen, L. B., Jensen, P. M., Kammersgaard, K., Kromann, L., “The Automated Pool Trainer—A multi modal system for learning the game of Pool”, Intelligent Multi Media, Computing and Communications: Technologies and Applications of the Future. John Wiley and Sons, pp. 90-96, June 2001.
• [0062] [Kan99] Kanov, Gerry, and Stauch, Shari, “Precision Pool”, Human Kinetics, ISBN 0-88011-897-0, 1999.
• [0063] [Nak01] Nakama, H., Tokashiki, H., “The Development of the Autonomous Billiard Robot”, Bull. Fac. Eng. Univ. Ryukus (Japan), no. 61, pp. 105-108, March 2001.
• [0064] [Qi99] Bingchen Qi, and Okawa, Y., “Building an Intelligent Mobile Plafform for Billiards”, Proc. 30^th Intl. Sym. Rob., Tokyo, pp. 605-612, 27-29 Oct. 1999.
• [0065] [Shu94] Shu Sang, W. C., “Automating Skills Using a Robot Snooker Player”, PhD Thesis, Bristol University, U.K., April 1994.
• [0066] [Chu02] S. C. Chua, E. K. Wong, Alan W. C. Tan, and V. C. Koo. “Decision algorithm for pool using fuzzy system”, iCAiET 2002: Intl. Conf. Al in Eng. & Tech., pp 370-375, June 2002.
• [0067] [Ali04] Mohammad Ebne Alian, Saeed Bagheri Shouraki, M. T. Manzuri Shalmani, Pooya Kaimian, Payam Sabmeydani, “Roboshark: a gantry pool player robot”, ISR 2004: 35^th International Symposium of Robotics, Paris-Nord Villepinte, France, 23-26 Mar., 2004.
• [0068] [Lon04] Fei Long, Johan Herland, Marie-Christine Tessier, Darryl Naulls, Andrew Roth, Gerhard Roth, Michael Greenspan, “Robotic Pool: An Experiment in Automatic Potting”, accepted in IROS'04, Sendai, Japan, Sep. 28-Oct. 2, 2004.
• [0069] [Lar02] James Larson, “Scratchy: An Autonomous Pool-Playing Robot”, http://www.mil.ufl.edu/imdl/papers/IMDL Report Fall 02/larson_james/scratchy.pdf
• [0070] [Den04] H. Denman, N. Rea, A. Kokaram, “Content-based analysis for video from snooker broadcasts”, Computer Vision and Image Understanding, no. 92, pp. 176-195, 2003.
• [0071] [Fla00] Gregory Flandin, Francois Chaumette, Eric Marchand, “Eye-in-hand/Eye-to-hand Cooperation for Visual Servoing”, ICRA2000: IEEE International Conference on Robotics and Automation, San Francisco, April 2000
• [0072] [Che04] B. Cheng, J. Li, and J. Yang, “Design of the Neural-Fuzzy Compensator for a Billiard Robot.” IEEE Intl. Conf. Networking, Sensing & Control, pp. 909-913, 2004.
• [0073] [Zha98] Z. Zhang, “A Flexible New Technique for Camera Calibration”, Microsoft Research Technical Resport MSR-TR-98-7, Dec. 2, 1998.