Camera and radar fusion for determining position of projectile in motion

[0001] This application is a continuation-in-part of U.S. patent application Ser. No. 19/076,840, filed Mar. 11, 2025, for “DETERMINING POSITION/SPEED OF PROJECTILE USING CAMERA AND RADAR,” which is a continuation-in-part of U.S. patent application Ser. No. 18/802,634, filed Aug. 13, 2024, for “OBJECT TRACKING METHOD AND SYSTEM USING CALIBRATED CAMERA,” which is a divisional of U.S. patent application Ser. No. 18/595,592, filed Mar. 5, 2024, for “SYSTEM AND METHOD FOR CAMERA CALIBRATION AND METHOD OF USING CALIBRATED CAMERA,” issued as U.S. Pat. No. 12,118,750, each of which is incorporated by reference herein in its entirety.

[0002] The present disclosure relates to camera calibration and determining the position and speed of a projectile in three-dimensional (3D) world coordinates using the calibrated camera radar.

[0003] Fundamentally, a camera provides an image mapping of a three-dimensional space (3D) onto a two-dimensional (2D) space or image plane. Current camera calibration techniques supply model parameter values that are needed to compute line of sight rays in space that corresponds to a point in the image plane.

[0004] A calibration or “projection” matrix, which is estimated during a camera calibration, is typically decomposed into eleven geometric parameters that define the standard pinhole camera model. Typically, camera model parameters include extrinsic and intrinsic parameters. The extrinsic camera parameters include 3D location and orientation of a camera in the world and intrinsic camera parameters include, among others, a focal length and relationships between pixel coordinates and camera coordinates.

[0005] In many applications, camera calibration is necessary to recover 3D quantitative measures about an observed scene from 2D images. For example, from a calibrated camera, it can be determined how far an object is from the camera, or the height of the object, etc. Typical calibration techniques use a 3D, 2D or 1D calibration object whose geometry in 3D space is known with very good precision.

[0006] From a set of world points and their image coordinates, one object of the camera calibration is to find a projection “matrix” and subsequently find intrinsic and extrinsic camera parameters from that matrix in a decomposition step. However, the decomposition into extrinsic and intrinsic camera parameters is one of the major issues in calibration due to reprojection error. Further, in the decomposition step, to extract the extrinsic and intrinsic camera parameters, several assumptions and constraints are made, which might be not true, for e.g., no lens distortion or no tilt. Accordingly, it is desirable to have a camera calibration method that does not require decomposition into intrinsic and extrinsic camera parameters.

[0007] In addition, it is also desirable to have a system for taking measurements that avoids having to make any assumptions regarding intrinsic or extrinsic camera parameters. It is further desirable to determine the 3D position of a projection in world coordinates with only a single 2D image and radar measurements.

[0008] The disclosed embodiments are directed to determining the position and speed of a projectile in 3D world coordinates using a single image and continuous wave radar.

[0009] In some embodiments, a method of tracking a projectile in motion comprises: determining, with at least one processor, a zero radial speed time of the projectile based on radar measurements of the projectile; determining, with the at least one processor, a radial speed of the projectile at a particular time based on the radar measurements; determining, by the at least one processor, a radial speed slope at the particular time based on the radial speed measurements; determining, by the at least one processor, a two-dimensional (2D) image point of the projectile at the particular time, the 2D image point represents the location of the projectile in an image captured by a camera; determining a depth scale coefficient at the particular time based on the zero radial speed time, the radial speed, the radial speed slope, the 2D image point, and a homographic transformation; and determining a position of the projectile in 3D space at the particular time based on a three-dimensional (3D) position of the camera, the depth scale coefficient, the 2D image point and the homographic transformation.

[0010] In some embodiments the camera has been calibrated without decomposing the camera parameters into explicit extrinsic and intrinsic parameters.

[0011] In some embodiments, the homographic transformation represents camera parameters describing a projection from a 3D world coordinates to 2D image planes.

[0012] In some embodiments, the position of the projectile in 3D space at the particular time is determined from the sum of the 3D position of the camera and a product of the depth scale coefficient, the 2D image point and an inverse of the homographic transformation.

[0013] In some embodiments, the homographic transformation includes a rotational part of the homographic transformation.

[0014] In some embodiments, the radar is a continuous wave (CW) radar.

[0015] In some embodiments, the camera is time-synchronized with the radar, and wherein the radar is configured to have a higher sampling rate than the camera

[0016] In some embodiments, the 3D position of the camera is determined based on the homographic transformation.

[0017] In some embodiments, the radar and camera are collocated.

[0018] In some embodiments, the depth scale coefficient at the particular time is determined from a radial distance from the radar or camera to the projectile, the 2D image point and the homographic transformation, and wherein the radial distance is determined from the radial speed, the radial speed slope, and the zero radial speed time.

[0019] In some embodiments, a system comprises: a radar; a camera configured to capture an image of a projectile; at least one processor configured to: determine a zero radial speed time of the projectile based on radar measurements of the projectile; determine a radial speed of the projectile at a particular time based on the radar measurements; determine a radial speed slope at the particular time based on the radial speed measurements captured by the radar; determine a two-dimensional (2D) image point of the projectile at the particular time, the 2D image point representing the location of the projectile in the image; determine a depth scale coefficient at the particular time based on the zero radial speed time, the radial speed slope, the 2D image point, and a homographic transformation; and determine a position of the projectile in 3D space at the particular time based on a 3D position of the camera, the depth scale coefficient, the 2D image point and the homographic transformation.

[0020] In some embodiments, the camera has been calibrated without decomposing the camera parameters into explicit extrinsic and intrinsic parameters.

[0021] In some embodiments, the homographic transformation represents camera parameters describing a projection from a 3D world coordinates to 2D image planes.

[0022] In some embodiments, the position of the projectile in 3D space at the particular time is determined from the sum of the 3D position of the camera and a product of the depth scale coefficient, the 2D image point and an inverse of the homographic transformation.

[0023] In some embodiments, the homographic transformation includes a rotational part of the homographic transformation.

[0024] In some embodiments, the radar is a continuous wave (CW) radar.

[0025] In some embodiments, the camera is time-synchronized with the radar, and the radar has a higher sampling rate than the camera.

[0026] In some embodiments, the 3D position of the camera is determined based on the homographic transformation.

[0027] In some embodiments, the radar and camera are collocated.

[0028] In some embodiments, the depth scale coefficient at the particular time is determined from a radial distance from the radar or camera to the projectile, the 2D image point and the homographic transformation, and wherein the radial distance is calculated based on the radial speed, the radial speed slope, and the zero radial speed time.

[0029] In some embodiments, the radial speed is determined from a function of a vector from the radar or camera to the projectile, linear velocity vector of the projectile and a radial distance from the radar or camera to the projectile.

[0030] In some embodiments, the 3D position of the projectile is determined from the sum of the 3D position of the camera and a product of the depth scale coefficient, the 2D image point and an inverse of the homographic transformation.

[0031] Further features as well as the structure and operation of various embodiments are described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers indicate identical or functionally similar elements.

[0050] Notably, technical terms used in the present specification are only for describing specific embodiments and are not intended to impose any limitation on the scope of the present disclosure. In addition, the term used in the present specification, although expressed in the singular, is construed to have a plural meaning, unless otherwise meant in context. The phrase “is configured with,” “include,” or the like, which is used in the present specification, should not be construed as being used to necessarily include all constituent elements or all steps that are described in the specification, and should be construed in such a manner that, among all the constituent elements or among all the steps, one or several constituent elements or one or several steps, respectively, may not be included, or that one or several other constituent elements, or one or several other steps may be further included.

[0051] In addition, in a case where it is determined that a detailed description of the well-known technology in the relevant art to which the present disclosure pertains makes indefinite the nature and gist of the technology disclosed in the present disclosure, the detail description thereof is omitted from the present specification.

[0052] As referred to herein, camera calibration is the process of obtaining camera parameters in the form of extrinsic and/or intrinsic parameters. Methods typically employed in the camera calibration include but are not limited to 3D Direct Linear Transformation (DLT), QR decomposition and perspective n point (PnP). A 3D DLT calibration obtains a homography or transformation matrix. The QR decomposition splits homography matrix into the extrinsic matrix and intrinsic matrix (of specific intrinsic model). The PnP such as P3P where n=3, obtains extrinsic camera parameters only, assuming intrinsic parameters are already known. Camera calibration method may use prior calibration results by the same or a different method. With prior information, camera parameters may be evaluated iteratively for e.g. using gradient descent (GD), from defaults to the ground truth.

[0053] As further referred to herein, DLT is one method for mono camera calibration, with its first step to recover the homography matrix H according to equation (1) as follows:

[0054] $\begin{array}{cc}H=\left\{h_{xx},h_{xy},h_{xz};h_{yx},h_{yy},h_{yz};h_{zx},h_{zy},h_{\mathrm{zz}};h_x,h_y,h_z\right\},& \left(1\right)\end{array}$
from a calibration pattern, a set of feature points {right arrow over (p)}={p_x, p_y, p_z} with known, predefined positions in 3D world, and from image points {right arrow over (q)}={q_x, q_y, 1} as a solution of the following equations: {right arrow over (p)} H∝{right arrow over (q)} or p H×{right arrow over (q)}=0 or according to equation (2) as:

[0055] $\begin{array}{cc}{q}_x=\frac{p_x{h}_{\mathrm{xx}}+p_y{h}_{\mathrm{yx}}+p_z{h}_{\mathrm{zx}}+h_x}{p_x{h}_{\mathrm{xz}}+p_y{h}_{\mathrm{yz}}+p_z{h}_{\mathrm{zz}}+h_x};q_y=\frac{p_x{h}_{\mathrm{xy}}+p_y{h}_{\mathrm{yy}}+p_z{h}_{\mathrm{zy}}+h_y}{p_x{h}_{\mathrm{xz}}+p_y{h}_{\mathrm{yz}}+p_z{h}_{\mathrm{zz}}+h_x}.& \left(2\right)\end{array}$

[0056] To solve the system of equations above, this relation is rearranged into the form: Ah=0 where A is built by eq. 1 from a set of 3D points {right arrow over (p_n )} and a set of corresponding 2D image points {right arrow over (q_n)}, and h={h_xx; h_yx; h_zx; h_x; h_yx; h_yy; h_yz; h_y; h_zx; h_zy; h_zz; h_z} is the vector of unknown H matrix elements. Then, a suitable optimization method can be applied, for e.g., least mean square (LMS), single value decomposition (SVD) or gradient descent (GD) and the system of equations is solved up to the scale, i.e. with 12−1=11 DoF.

[0057] As further referenced herein, PnP uses a set of methods for camera pose estimation, e.g., 6 DoF extrinsic parameters, assuming intrinsic parameters are given or known such as may be the case when the camera has been prior “calibrated”. For this reason, PnP is not considered a complete calibration method since the intrinsic parameters are assumed. A minimum number of 3D points for camera pose estimation is three (e.g., P3P, three pairs of correspondences). As decomposition is not applied in the method for calibrating camera described in the current disclosure, PnP including P3P is not used in a factory calibration step. In some embodiments, the PnP or P3P is therefore used in a field calibration step only. That is affine corrections are applied to the default homography matrix, formally for the “uncalibrated” or “partially calibrated” camera, with intrinsic parameters unknown explicitly and hidden within the homography matrix.

[0058] As referred to herein, an “uncalibrated” or a “partially calibrated” camera is a camera for which pose estimation is either unknown or incomplete. In some embodiments, the pose estimation includes camera orientation R^T and camera position −sR^T. In some embodiments, a homography matrix without the decomposition may provide an incomplete pose estimation, for e.g. position only, but not the orientation. Camera orientation is typically defined by optical axis. However, the optical axis is an intrinsic camera parameter (such as principal points: c_x, c_y).

[0059] Camera position is the translation part of a transform T placing the camera from its own coordinate system into the world coordinate system. Extrinsic camera matrix C is the inverse of transform matrix T according to equation (3) as follows:

[0060] $\begin{array}{cc}T=C^{-1}={\left\{R;s\right\}}^{-1}=\left\{R^T;-s{R}^T\right\}.& \left(3\right)\end{array}$

[0061] Based on the above equation, camera position p_c obtained from extrinsic camera matrix C is: p_c=−sR^T. In some embodiments, the camera position p_c may also be obtained from translation part from the inverse of the homography matrix H (without decomposition). Hence, the camera position p_c=−sR^T is considered to be independent from the intrinsic matrix K.

[0062] As described herein, decomposition of intrinsic and extrinsic camera parameters is the process of splitting the homography matrix H having 11 DoF into extrinsic C (6 DoF) and intrinsic K (5 DoF) parts: H={R; s}K.

[0063] The decomposition of camera parameters into extrinsic and intrinsic parts has an ambiguity as the decomposition gives rise to multiple possible solutions, for e.g., four—for QR decomposition for a given intrinsic model: H=C_nK_n.

[0064] As used herein, “QR decomposition” decomposes “rotational” part (RK) of the homography matrix H into an orthonormal rotation matrix and a lower (or upper) triangular.

[0065] Applying different intrinsic models K_m may lead to even more distinctive solutions. One intrinsic model may be converted into any other models by applying rotation matrix R according to the following equation (4):

[0066] $\begin{array}{cc}H=C_1{K}_1=\left(C_1,R_{12}\right)\left(R_{12}^T{K}_1\right)=C_2{K}_2,& \left(4\right)\end{array}$
where K₁ is a first intrinsic model and K₂ is a second intrinsic model, respectively with R₁₂ is the rotation matrix for converting K₁ to K₂. By keeping the same original homography matrix H intact, distinct intrinsic models lead to distinct camera orientation R. In some embodiments, the camera position may remain the same p_c=−sR^T, irrespective of the intrinsic model K applied.

[0067] As described herein, the homography matrix H is a 4×3 matrix, defined up to an arbitrary scale. Thus, the homography matrix H has 11 DoF. Homography matrix H is the product of extrinsic matrix C and intrinsic matrix K as defined according to the following equation (5):

[0068] $\begin{array}{cc}H=\mathrm{CK}\mathrm{where}C=\left\{R;s\right\}\mathrm{so}\mathrm{that}H=\left\{RK;\mathrm{sK}\right\}.& \left(5\right)\end{array}$

[0069] The homography matrix H that maps 3D points {right arrow over (p)}={p_x, p_y, p_z} to 2D image points {right arrow over (q)}={q_x, q_y, 1} as: {right arrow over (p_n)}H∝{right arrow over (q_n )} (up to scale, proportional to) including projection into an image plane. In some embodiments, the homography matrix H is the result of the 3D DLT method. While DLT is one method, other suitable methods may be used including methods as proposed by: Richard Hartley & Andrew Zisserman, “Multiple View Geometry in Computer Vision”, page 88.

[0070] As described herein, the extrinsic camera matrix C is a 4×3 matrix. Matrix C comprises a rotation matrix R and a translation vector s: C={R; s}. The extrinsic camera matrix C has six independent parameters (6 DoF) i.e. 3 DoF for the rotation and 3 DoF for the translation. The extrinsic camera matrix C maps 3D points from the world 3D coordinate system to the camera 3D coordinate system, without applying a projection onto the image plane.

[0071] The intrinsic camera matrix K is a 3×3 matrix and has five independent parameters (5 DoF). It comprises, among others, inner parameters of the camera such as focal length, principal axis (shift), and tilts. The intrinsic camera matrix K may be represented according to equation (6) or (7) as:

[0072] $\begin{array}{cc}K=\left\{f_x,0,0;s,f_y,0;c_x,c_y,1\right\}\left(“\mathrm{astigmatic}-\mathrm{skew}”\mathrm{model}\right)& \left(6\right)\end{array}$ $\mathrm{or}$ $\begin{array}{cc}K=\left\{f,0,t_x;0,f,t_y;c_x,c_y,1\right\}\left(“\mathrm{tilt}-\mathrm{shift}”\mathrm{model}\right).& \left(7\right)\end{array}$

[0073] A 3D rotation matrix “R”, a 3×3 orthonormal matrix, as used herein, is the component of a general 3D transform T or the extrinsic camera matrix C=T⁻¹. Rotation matrix “R” holds 3 independent parameters (3 DoF). Other equivalent representations of 3D rotation include axis-angle and unit quaternion, all having 3 DoF. A translation vector “s”, a 3D vector, as used herein, is the component of the general 3D transform T or the extrinsic camera matrix C=T⁻¹. Translation vector “s” holds 3 independent parameters (3 DoF). As referred to herein, Degrees of Freedom (DoF) refers to the number of independent parameters, possibly hidden. For example: rotation matrix, quaternion and the angle for axis-angle 3D rotation representation, each has 3 DoF.

[0074] As referred to herein and described in the present disclosure, camera calibration is the process of obtaining camera parameters in the form of extrinsic and/or intrinsic parameters. In some embodiments, the camera calibration comprises a multiple steps process as will be described below. In some embodiments, the camera calibration comprises 2-steps calibration with a first step being referred to as a factory calibration and second step being referred to as a field calibration. In accordance with aspects of the present disclosure, the factory calibration is typically applied at least once for the camera to obtain a transformation matrix H representing a plurality of camera parameters. The plurality of camera parameters comprises intrinsic camera parameters and extrinsic camera parameters. In some embodiments, the transformation matrix H is for mapping 3D real world model features to corresponding 2D image features.

[0075] In some embodiments, the transformation matrix is a homography matrix H. In some embodiments, the transformation matrix H represents intrinsic and extrinsic camera parameters. In some embodiments, the transformation matrix represents implicit intrinsic and implicit extrinsic camera parameters. In some embodiments, the transformation matrix may be obtained by DLT method as will be described below or other suitable algorithms. In some embodiments, the field calibration step may be applied once or multiple times. In some embodiments, the field calibration may be referred to as a calibration correction for the extrinsic camera parameters.

[0076] In some embodiments, the second step is for correcting the extrinsic camera parameters implicitly, within the homography matrix H, without the decomposition of the intrinsic and extrinsic camera parameters. The field calibration may employ PnP algorithm including P3P, using the homography matrix H obtained from the first step or factory calibration as an initial estimation. Therefore, in the field calibration, only affine transform corrections are applied without altering intrinsic camera parameters defined implicitly within the homography matrix H.

[0077] FIG. 1 is an exemplary diagram of an object measurement apparatus 10 employing a plurality of camera devices 100₁, 100₂, . . . , 100_n calibrated in accordance with methods of the present disclosure. Each of the plurality of camera devices are capable of wired or wireless communication with a computing device 200 for performing object tracking and parameter measuring operations.

[0078] Each camera device, such as the exemplary camera device 100 shown in FIG. 1, according to the embodiment of the present disclosure, may include at least components such as: a lens 101 and sensor 105, a digital signal processor or controller 110, which can include one or more hardware microprocessors, a communication unit 120 that is connected to and controlled by the controller 110, an image capture unit 130 for capturing images and/or video, a calibration unit 140 for controlling camera calibration operations, a memory 150, and a tracking unit 160.

[0079] Exemplary camera device 100 as depicted in FIG. 1, in connection with embodiments of the present disclosure, may include any device, system, component, or collection of components configured to capture images and/or video. In an embodiment, the camera device 100 includes an image capture unit 130 having optical elements such as, for example, lenses 101, filters, etc., and at least one image sensor 105 upon which an image may be recorded. Such an image sensor 105 may include any device that converts an image represented by incident light into an electronic signal. The image sensor 105 may include a plurality of pixel elements, which may be arranged in a pixel array (e.g., a grid of pixel elements) for acquiring an Image. For example, the image sensor 105 may comprise a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) image sensor. The pixel array may include a two-dimensional array with any aspect ratio. The image sensor 105 may be optically aligned with various optical elements that focus light onto the pixel array, for example, lens 101. Any number of pixels may be included such as, for example, hundreds or thousands of megapixels, etc.

[0080] In an embodiment, the image capture unit 130 acquires an image under the control of the controller 110, and the acquired image is stored in the memory storage unit 150 under the control of the controller 110. Further data and information for operation of the camera 100 can be stored in the memory 150. Then, the image acquired in the image capture unit 130 is stored in the memory 150. The acquired images have respective pieces of unique identification information, and the pieces of identification information can correspond to points of time, respectively, at which the images are acquired. In addition, other types of tracking information that are detected from at least one image according to the target designation information can be stored in the memory 150. A storage area of the memory 150, in which the images are stored, can be hereinafter alternatively referred to as an image storage unit 152, and a storage area of the memory 150, in which “tracking information” associated with target object tracking are stored, is hereinafter referred to as a tracking information storage unit 155. The camera memory 150 additionally stores local camera calibration information such as the value of extrinsic and intrinsic parameters used for camera calibration. In some embodiments, the camera memory 150 may also store the value of extrinsic and intrinsic camera parameters obtained from the camera calibration.

[0081] An image processing unit 170 is further provided to perform types of digital signal processing to raw camera image data, e.g., filtering, noise reduction, image sharpening, etc.

[0082] As further shown in FIG. 1, exemplary camera 100 can include a communication unit 120 to enable the camera to receive remote control signals for performing camera operations in connection with object tracking and measuring movement parameters and includes at least one module for performing wired and/or wireless communication with a remote controller processor or remote computing apparatus 200 (remote control unit). The communication unit 120 receives a control signal from a remote control apparatus through at least such one module and can transfer an acquired image to the remote controller processor or computing apparatus 200 under the control of the controller 110.

[0083] Whether remotely or locally controlled using controller 110, the camera 100 may operate at certain frame rates or be able to capture a certain number of images in a given time. The camera 100 may operate at a frame rate of greater than or equal to about sixty frames per second (fps), for example between about one hundred and about three hundred frames per second (fps). In some embodiments, a smaller subset of the available pixels in the pixel array may be utilized to allow for the camera to operate at a higher frame rate.

[0084] The tracking unit 160 operates to detect a target that corresponds to a received target object being tracked. In some embodiments, the target comprises a ball used in golf, cricket, baseball, softball, tennis, etc.

[0085] In an embodiment, any number of a variety of triggers may be used to cause the exemplary camera 100 to capture one or more images of the moving object. By way of non-limiting examples, the camera 100 may be triggered upon receipt of a timed activation signal, or when the moving object is detected, known or estimated to be in the field of view of the camera, or when the moving object first begins or modifies its flight (for example when a baseball is pitched, when a baseball is batted, when a golf ball is struck, when a tennis ball is served, when a cricket ball is bowled, etc.), or when a moving object is detected at a leading row of pixels in the pixel array, etc.

[0086] In an embodiment, the remote controller processor or computing apparatus 200 can control overall operation of the remote cameras 100, 100₁, 100₂, . . . , 100_n that can be physically spaced apart to perform object tracking and measurements according to the embodiment of the present disclosure. The computing apparatus 200 can control the image capture unit 130 according to the control signals 175 that is received through the communication unit 120 and to configure the controller 110 to acquire an image. Subsequently, the controller 110 can store the acquired image at the device and/or can transmit the stored image and the related image identification and tracking information to the computing apparatus 200 for further processing. The further processing can include the application of one or more algorithms to receive raw camera image data in a series of image frames. For example, in accordance with embodiments herein, an example camera measurement of a moving object may include, but is not limited to identifying, in each of the images, one or more of: a radius of the object, a center of the object, a velocity, an elevation angle, and an azimuth angle, e.g., calculated based on the radius of the object, the center of the object, and any pre-measured camera alignment values.

[0087] In accordance with embodiments herein, FIG. 2 depicts implementation of the method described in the present disclosure using calibration unit 140 to perform a factory camera calibration method 250 i.e. first step of camera calibration. In some embodiments, the factory calibration 250 is performed using given known 3D features and detected 2D images features from the camera thereby generating the transformation matrix H. Such transformation matrix may be termed as “default” transformation matrix. As described herein, the transformation or projection matrix H represents intrinsic and extrinsic camera parameters having 11 DoF. As can be seen from FIG. 2, the factory calibration is undertaken without performing a decomposition to extract intrinsic and extrinsic camera parameters explicitly. In some embodiments, following the factory calibration, the camera may undergo a further calibration i.e. field calibration prior to using the calibrated camera. Hence, in some embodiments, the camera that has been calibrated following the factory calibration may be still considered as an “uncalibrated” camera. In some embodiments, the camera that has been calibrated following the factory calibration refers to a “partially calibrated” camera.

[0088] In accordance with further embodiments herein, FIG. 3 depicts an application of the present disclosure to perform a field calibration method 300. In some embodiments, the field calibration refers to a second step of the camera calibration described in the present disclosure. In some embodiments, as described above, the field calibration refers to a method 300 for calibrating an ‘uncalibrated’ camera or a “partially calibrated” camera.

[0089] In some embodiments, the field calibration is performed based on given known 3D features and detected 2D images features from the camera, and given default camera parameters from transformation matrix H, where the transformation matrix H is obtained from the first step of calibration i.e., factory calibration (refer to FIG. 2). In some embodiments, the calibration unit 140 performs a method to generate an updated correction matrix H′ by performing affine correction to obtain updated extrinsic camera parameters. Thus, it is the case that the calibration method comprising the factory calibration and field calibration described herein advantageously is performed without decomposition of the intrinsic and extrinsic camera parameters. Accordingly, the accuracy of the calibration method described herein is advantageously better than the decomposition method as no assumption was made.

[0090] In some embodiments, there is provided a method for tracking an object in motion. The tracking method includes capturing, from each of one or more calibrated cameras, one or more image frames of the object in motion. In some embodiments, each of the one or more calibrated cameras may have been calibrated according to a calibration method described herein. In some embodiments, the calibration generates and uses a respective transformation matrix for mapping 3D real world model features to corresponding 2D image features. The tracking method further comprises determining, using a hardware processor, motion characteristics of the object in motion based on the captured one or more image frames from each of the one or more calibrated cameras. In some embodiments, the determining of motion characteristics is based on implicit intrinsic camera parameters and implicit extrinsic camera parameters of the respective transformation matrix from each respective one or more calibrated cameras. In some embodiments the tracking is a 3D tracking.

[0091] In some embodiments, each of the one or more calibrated cameras is calibrated without decomposing the plurality of the camera parameters into explicit extrinsic and explicit intrinsic camera parameters. In some embodiments, each of the first calibration (factory calibration) and the second calibration (field calibration) is performed without the decomposition of the camera parameters into explicit extrinsic and explicit intrinsic camera parameters. In some embodiments, both the first calibration step i.e. factory calibration and the second step calibration i.e. field calibration are performed without the decomposition of the camera parameters into extrinsic and intrinsic camera parameters.

[0092] In some embodiments, following the first step or factory calibration, each of the one or more calibrated cameras is further calibrated by modifying one or more implicit extrinsic camera parameters to obtain the explicit extrinsic camera parameters. In some embodiments, the modifying step comprises adjusting implicit extrinsic camera parameters through an affine correction.

[0093] FIG. 4 is an exemplary diagram illustrating the field calibration method 400 for the exemplary camera 100 of FIG. 1 according to an embodiment. In some embodiments, the field calibration method for e.g. recovery of camera position in 3D world by reference patterns (natural or artificial) is accomplished without using extrinsic and intrinsic camera parameters explicitly. That is, the field calibration method is performed without decomposition of camera parameters into intrinsic and extrinsic parts. The field calibration method may be performed by processing logic that may include hardware (processor, calibration unit 140, circuitry, dedicated logic, etc.) and software (such as is run on the hardware processor 110 or computer system 200), or a combination of both, which processing logic may be included in the calibration unit 140.

[0094] Referring to FIG. 4, in an initial step 403, there is obtained the camera devices default or “given” camera parameters. In some embodiments, the default camera parameters may be inherent in an initial calibration, projection or “transformation” matrix (“H0”) that represents the plurality of default intrinsic and extrinsic camera parameters. In some embodiments, the camera is operated to obtain real world 3D feature points of an image obtained through the camera field of view (FoV) and are represented as feature points “p”. Default 2D feature points of the camera image plane are represented as q₀, and desired 2D feature points are referred to as q_K.

[0095] In some embodiments, the 3D models or reference patterns may include but are not limited to stencils. In some embodiments, the 3D models or reference patterns may be any suitable reference including cage corners, stumps, marks, lines, boundaries. In some embodiments, the 3D models or reference patterns may be static, i.e., its position does not change. In some embodiments, the 3D models or reference patterns may be in motion. In such an embodiment, a first camera system (for e.g. a stereo system, where there are at least two cameras) is calibrated using the method described herein and is used to track a moving object to obtain the 3D position and/or trajectory parameters. This information may be used to perform a calibration of a second camera system, where the second camera system may be the same or different stereo system as the first camera system. In such an embodiment, the second camera system is also used to track the same moving object to obtain the camera parameters of the second camera system.

[0096] There is also provided a step of controlling the camera to update the camera parameters with a 3D space, which is represented as an updated transformation matrix H_k+1 based on the natural or artificial reference patterns described at step 406. This step includes updating of the extrinsic camera parameters having 6 DoF. At step 406, the camera with known or given intrinsic parameters (from the factory calibration) is manually calibrated by adjusting the camera parameters associated with a small axis-angle and/or a translation vector. In some embodiments, the adjustment impacts implicit extrinsic camera parameters having 6 DoF. Next, a manual aligning of obtaining 2D virtual image features (q2) may be performed with 2D image features of a reference image (q1) by applying one or more corrections matrix (H_Δ) to the plurality of the implicit camera parameters of the transformation matrix H_k, thereby obtaining the updated transformation matrix H_k+1 and in turn calibrating the camera. As described above, the camera calibration is performed without the decomposition of the camera parameters into explicit intrinsic and extrinsic parts. In this embodiment, the 2D virtual image features (q2) and 2D image features of the reference image (q1) are represented in pixel coordinates.

[0097] Continuing at step 410 of FIG. 4, based on the manual aligning as performed during the calibration at step 406, the processor may build at the camera, a correction matrix, “H_Δ”. Subsequently at step 414, the built correction matrix H_Δ is applied at step 410 to the initial transformation matrix H_k associated with the default camera parameters, to obtain an updated matrix H_k+1 associated with updated camera parameters according to the following equation (8):

[0098] $\begin{array}{cc}{H}_{k+1}=H_{△}{H}_k.& \left(8\right)\end{array}$

[0099] Further, according to step 417 of FIG. 4, once the updated matrix H_k+1 is obtained at step 414, the updated matrix H_k+1 is subsequently applied to 3D models, where new 2D points may be obtained according to the following: q_k+1∝pH_k+1, where new 2D points (camera position) are obtained from the updated homography matrix H_k+1. As 3D points p are static, 2D points q_k are changing according to H_k and converging to desired q_K by tuning H_k+1 with H_Δ.

[0100] Continuing to step 420 of FIG. 4, a determination is made as to whether the 2D image features (q) and 2D image features points of the virtual reference image (q′) are equal or below some target threshold number of pixels. In other words, it is determined whether the difference between a 2D position (q) evaluated using the current transformation matrix H_k and a desired 2D position (q′) obtained from updated correction matrix H_k+1. i.e., the quantity |q−q′| is less than or more than a threshold value “threshold”. In an embodiment, the threshold value is represented by a distance in pixels, e.g., one (1) pixel and is alternatively referred to as a “reprojection error”.

[0101] If, at step 420, it is determined that the quantity |q−q′| is greater than a predefined threshold value, for e.g., greater than one pixel, then the method 400 repeats by returning to step 406 and all steps 406, 410, 414, 417, and 420 are repeated. In some embodiments, the steps of method 400 are iterative and repeated until such time when the quantity |q−q′| is equal to or less than the threshold value, for e.g., ≤1 pixel, at which time the method 400 terminates as these 2D points are considered “in place”. Following this, the camera is deemed to have been fully calibrated and is ready to be used for measurement purposes. The 2D feature points {right arrow over (q_n)}={q_nx, q_ny, 1} are computed using the obtained correction transformation matrix H_k+1 and the set of feature points {right arrow over (p_n)}={p_nx, p_ny, p_nz} in 3D (Cartesian) coordinate space according to eq. 2. The computed {right arrow over (q_n)}={q_nx, q_ny, 1} approximately yield the 2D position.

[0102] In an embodiment, an automatic camera calibration correction i.e. second stage calibration may be performed. The calibration computation method is described as follows.

[0103] Solution for correction matrix H_Δ={Δr_k; Δs_k} is determined from a system of linear equations:

[0104] $\begin{array}{cc}\left(p\Delta r_k+\Delta s_k\right)r_k\times q_K=-\left(p{r}_k+s_k\right)\times q_K,& \left(9\right)\end{array}$
where equation (9) is derived from projection constraint described in equation (10),

[0105] $\begin{array}{cc}{q}_K\propto p{H}_{k+1}=p{H}_{△}{H}_k=\left(p{r}_{\Delta ,k}+\Delta s_k\right)r_k+s_k.& \left(10\right)\end{array}$

[0106] An angle approximation: r_Δ,k≃(I+Δr_k) is applied to generate a system of linear equations (approximate), where “I” is an identity matrix i.e. unit 3×3 matrix. A valid (orthonormal) rotation matrix r_Δ,k can then be constructed from obtained Δr_k by applying Cayley formula and then apply H_Δ={Δr_k; Δs_k} to obtain updated 2D points q_k+1∝pH_k+1 (using eq. (2). This procedure is repeated till the value of |q_k+1−q_K| is within the threshold. r_Δ,k˜I+Δr_k, where Δr_k is a skew symmetric matrix. Then, setting k=0, where k is an iteration counter, then the H_k is initially provided by the following equation (11):

[0107] $\begin{array}{cc}{H}_k=H,H_k\equiv \left\{r_k,s_k\right\}.& \left(11\right)\end{array}$

[0108] In some embodiments, the method subsequently performs the following steps:

• • [0109] 1. solving (pΔr_k+s_Δ,k)·(q×r_k)=(q×(pr_k+s_k) for Δr_k and s_Δ,k where r_k represents a change in rotation and s_k represents a change in camera translation;
  • [0110] 2. assigning H′=H_k if Δr_k and s_Δ,k are small and then stop;
  • [0111] 3. otherwise, building a valid r_Δ,k from Δr_k (e.g., using half angles, Cayley formula in the event that the value does not converge. This is to avoid the use of polynomial equations); and
  • [0112] 4. updating H_k+1={r_k+1=r_Δ,kr_k,s_k+1=s_Δ,kr_k+s_k}, k=k+1 and repeat.

[0113] In an embodiment, the method solves a system of linear equations where the relation is between 3D points in a scene and their projection onto a camera's 2D the image plane such that the method solves a set of variables using the DLT method from the set of similarity relations: q∝p H′ where q (2D points) and p (3D points) are known vectors, where “∝” denotes proportional to, and H′ is a matrix (or linear transformation) which contains the unknowns to be solved. According to this embodiment, a 2D camera position consisting of q_x and q_y from the transformation matrix H′ is obtained.

[0114] FIG. 5A depicts an exemplary environment 550 in which method according to one embodiment of the present disclosure is applied for conducting an on-site calibration (field calibration) of exemplary camera device 100 in an example second stage of calibration process.

[0115] The view shown in FIG. 7A represents the image at a camera viewer. As shown in FIG. 7A, the camera calibration unit configures the camera to generate a virtual reference image 555 which, for camera calibration at step 406 of FIG. 4, is controllable for translational and rotational movement to align with a corresponding physical real world object 565 in the camera view for an affine correction. In an embodiment, the camera includes a physical directional button(s) or a manipulable “joystick” 575, which is manipulated by a user to move the virtual reference image 555 such that the virtual stumps 555 match the physical stumps 565, i.e., are aligned within a threshold distance (q−q′). The movement/alignment data obtained from the aligning of the calibration reference image 555 to the physical real world object 565 in the camera field of view is an offset correction that is used to generate new/updated extrinsic parameters of the camera.

[0116] In an embodiment, the method described in FIG. 4 recovers camera position in 3D world by reference patterns 565, i.e., a stump or a wicket, without using extrinsic and intrinsic camera parameters explicitly, i.e., without decomposition of camera parameters into intrinsic and extrinsic parts. In an embodiment, natural or artificial reference patterns can be used to perform the affine correction. Artificial reference patterns 565 can include other types of physical calibration objects, e.g., a marker, corner of a cage and so forth.

[0117] FIG. 5B depicts a further method for field calibration/correction. In an example sports cage 580 shown in FIG. 5B, the camera viewer presents virtual reference image as marks 556, e.g., marking a geometric shape such as a square, which can be a reference used for calibration. Other example of virtual references can include markings for corners of a cage (e.g., a batting cage), lines indicated as field line markings 557, field markings, and so forth. The camera includes physical directional button(s) 592 or a manipulable “joystick” (not shown) controllable for adjusting the translational (e.g., offset) and rotational (e.g., angle) parameters 590 associated with the movement of the reference markings 556, 557 when performing a calibration. The movement of the virtual reference calibration markings 556, 557 in the camera view align with a corresponding physical real world object(s) in 3D space represented, e.g., by orthogonal x, y, z directions as in a Cartesian coordinate system, for an affine correction thereby impacting 6 DoF and updated extrinsic camera parameters are thus obtained.

[0118] Additionally, shown in the camera view of FIG. 5B, homographic matrix values 595 for default (H_n), correction (H_Δ) and updated matrices (H_n+1) for each calibration. From the updated homographic matrix values 595, the camera position in 3D world is recovered without using intrinsic and extrinsic camera parameters explicitly, i.e., without decomposition of camera parameters into intrinsic and extrinsic parts.

[0119] Other examples of reference markings include checkerboards, AprilTag, etc. and the calibration apparatus described in pending U.S. patent application Ser. No. 18/527,245 filed on Dec. 2, 2023, as well as a moving object including a ball. Natural patterns are known objects with a-priori placement with respect to camera. That is, in the first stage calibration, the relative position of the patterns to camera is known (a-priori). The parameters of the camera are recovered by the DLT method as described above, without decomposition of the camera parameters into intrinsic and extrinsic parts.

[0120] Thus, referring to the method 400 of FIG. 4, there is provided a two-stage calibration process, wherein a first stage is the recovery of implicit camera parameters using the DLT to recover 11 DoF parameters (e.g., implicit 6 DoF extrinsic and 5 DoF intrinsic). Then, at step 410, the second stage of the calibration requires an on-site affine correction, without altering implicit intrinsic parameters. This on-site correction assumes the following: the implicit intrinsic parameters of the camera do not change, i.e., the focal length, distortion profile, etc. remain the same; and only translation and/or rotation of camera with respect to world coordinate has changed. Thus, the extrinsic 6 DoF parameters are changed implicitly, without decomposing of combined set of calibration parameters (11 DoF) into intrinsic (5 DoF) and extrinsic parts (6 DoF).

[0121] In an embodiment, to extend a depth of focus along a field, e.g., of a stadium, the system further provides support for tilted lenses by performing a lens tilt calibration method such as that described in Scheimpflug principle by an implicit, unspecified model, of intrinsic camera parameters.

[0122] The exemplary camera system of FIG. 1 is further configured as an object measurement system applicable to recover 3D world object positions by using stereo (two or more) cameras without decomposition of camera parameters into intrinsic and extrinsic parts. That is, according to embodiments herein, a measurement system and method for tracking objects is provided. In an embodiment, object tracking measurements include a hard-synchronization, and soft-synchronization parameter processing to recover projectile trajectories by a regularized high order trajectory model by partially overlapping or non-overlapping by field of view (FOV) stereo cameras for e.g. two or more without decomposition of camera parameters into intrinsic and extrinsic parts. It is to be understood that each of the two or more cameras in the object measurement system may have been calibrated according to the method described herein.

[0123] For hard-sync object tracking and measurement, the camera system of FIG. 1 is configured such that the cameras, e.g., one or more multiple cameras 1001, 1002, . . . , 100_n use the same external VSync signal, with cameras hard wired by a general purpose I/O (GPIO) line, and where frames are captured synchronously with the same timestamp.

[0124] In some embodiments, FIG. 6 depicts an application to perform a measurement for e.g., tracking parameters of a moving object in a ‘hard-sync’ method 600. In some embodiments, the hard-sync method uses synchronous cameras that have been calibrated according to the method described herein. In some embodiments, the cameras have been calibrated without decomposition of the camera parameters into intrinsic and extrinsic camera parameters during the calibration. In some embodiments, the hard-sync method further utilizes a 2D-to-3D conversion algorithm. Given parameters from N cameras (1, 2, . . . , N), providing respective projection matrices H_n where n=1, 2, . . . N, there are obtained corresponding tracked object features in 2D with each camera providing a 2D feature (t, q1), (t, q2), . . . , (t, Qin). Then, applying a multi 2D-to-3D DLT, 3D positions of the moving object may be obtained with estimated errors.

[0125] In an embodiment, since pointwise triangulation methods are not used and are replaced with implicit “overlapping” of the trajectory in a parametric space, the requirement for “hard sync” can thus be relaxed.

[0126] As an example, given two cameras (camera “a” and camera “b”) that have been calibrated according to the 2-stages calibration method described in the current disclosure, there are computed 2D feature points q_ax and q_ay for camera “a” and 2D feature points q_bx and q_by for camera “b” using the obtained correction transformation matrix H′. Subsequently, these 2D image points may then be converted to corresponding 3D real world points. The measurement to obtain 3D points from 2D images is thus expedited as the measurement using the calibrated camera disclosed herein does not require a split of extrinsic and intrinsic parameters and the processing is not concerned with focal lengths, principal axes, lens tilts, stereo base distance.

[0127] In accordance with a further embodiment herein, FIG. 7 depicts an application of the present disclosure to perform a measurement for e.g., tracking parameters of a moving object in a ‘soft-sync’ method 700. In some embodiments, the soft-sync method uses asynchronous cameras that have been calibrated according to the method described herein. In some embodiments, the cameras have been calibrated without decomposition of the camera parameters into intrinsic and extrinsic camera parameters. In some embodiments, the soft-sync method utilizes non-overlapping FOV cameras. In some embodiments, the soft-sync method further utilizes a 2D-to-3D conversion algorithm. Given parameters from N cameras (1, 2, . . . , N), providing respective projection matrices H_n where n=1, 2, . . . N, there are obtained corresponding tracked object features in 2D with each camera providing a 2D feature, e.g., {t[1,n],×X[1,n],×Y[1,n]}) from camera 1, {t[2,n],×X[2,n],×Y[2,n]} from camera 2, . . . {t[m,n],×X[m,n],×Y[m,n]}) from camera m. Then, applying a 12p (twelve point) trajectory model, there are computed trajectory parameters with estimated errors from which 3D positions of the moving object are obtained with the corresponding estimated errors.

[0128] For soft-sync object tracking and measurement, the camera system of FIG. 1 is configured such that the cameras, e.g., one or more multiple cameras 100₁, 100₂, . . . , 100_n capture image frames asynchronously. The processing, however, uses the same common time reference by clock. In such an embodiment, each camera is synchronized periodically by precision time protocol (PTP) and/or global position system (GPS). Each camera captures frames in a free run mode with its own interval VSync, thus frames' timestamps are different, but the reference time is the same within a tolerance defined or required by the measurement precision. In some embodiments, for a ball speed of 50 m/s and a distance measurement of 2.5 cm, the time synchronization tolerance is about 0.5 milliseconds (ms). As an exemplary embodiment, for the time tolerance of 0.5 ms and clock oscillators with an accuracy of 10 points per million (ppm), the cameras are to be synchronized every 50 seconds.

[0129] FIG. 8A conceptually depicts an implicit stereo representation 800 for a given trajectory model that assumes a soft sync. In this embodiment, there are recovered multiple cameras' asynchronous (“soft sync”) ball trajectory parameters by trajectory stitching in parametric space. As shown in FIG. 8A, parameters are recovered from cameras 100_A and 100_B such that the trajectory is scaled until it stitches by shape and by time. The scaling is shown to obtain a trajectory and measurement of an object (e.g., 3D positions of a ball by time) along path 805. However, “soft-sync” processing obtains trajectory parameters with camera 100_A frames along path 805A only and with camera 100_B frames along path 805B. In some embodiments, paths 805A and 805B may overlap. In some embodiments, paths 805A and 805B may not overlap by cameras' view.

[0130] FIG. 8B depicts inappropriate application of the triangulation (“hard-sync” method) over “soft-sync” data resulting in object's positions errors. As shown in FIG. 8B, frames are captured asynchronously by a free running camera (using an internal VSync signal), using the same time reference (PTP, GPS, power grid), but without exact matching of timestamps. The method chooses the closest frames by pairs to provide a ground truth object location(s), e.g., ball positions 810. The method estimates ball locations, e.g., a first location 815A based on results of an applied triangulation over the two close frames, e.g., taken by cameras 100_A, 100_B or a second location 815B based on the triangulation. In an example implementation, there is chosen the closest camera image frames, wherein for 250 fps (two milliseconds) and ball speed of 50 m/s (or 200 fps and ball speed of 40 m/s), in which it results in object's position estimation error (“a shift”).

[0131] This soft-sync method includes a feature such as a 9p or 12p trajectory model, possibly non-overlapping views, e.g., using two, three or more cameras. Such an object tracking method can additionally be referred to as: non-overlapping stereo, multiple camera stereo, (implicit) trajectory stitching. This soft-sync method may be applied in cricket, baseball, football, etc.

[0132] In an embodiment, a “Triangulation” is applied to provide a 3D reconstruction for asynchronous (soft-sync) cameras with the common time reference and the synchronous frames captured by common VSync signal.

[0133] The trajectory is physically the same for all cameras. In this embodiment, it is further assumed that: all cameras share a common time reference, to provide proper timestamps even if their frames are asynchronous.

[0134] In an embodiment, a requirement for cameras' overlapping views can be relaxed by applying an “overlapping” in a parametric space. Two cameras might look in opposite directions and observe different part of the same trajectory. In some embodiments, sufficient base distance between cameras may be necessary.

[0135] A requirement for cameras to have a close focal length can therefore be relaxed. In some embodiments, the cameras may have significantly different focal length with small or no FOV overlapping.

[0136] For no-sync, object tracking and measurement, the camera system of FIG. 1 is configured such that the cameras, e.g., one or more multiple cameras 100₁, 100₂, . . . , 100_n run asynchronously, using their own time reference by their own clock. In some embodiments, for no-sync, clock synchronization may be recovered (i.e. conversion from “no-sync” to “soft-sync”) by post-processing using common events, by for e.g. video, audio, vibration or radar. Video events may be but are not limited to ball bouncing and IR flash. Audio and/or vibration events may be but not limited to ball hit and bouncing sound. Radar events may be but are not limited to ball release and/or ball bouncing. In some embodiments, clock synchronization may be recovered by flying ball. In such an embodiment, a specific configuration of cameras positions may be required for e.g. “in-plane” configuration where cameras are positioned side-by-side.

[0137] In an exemplary embodiment, for a camera system having two cameras i.e. camera A and camera B, the camera system is configured to determine the position of the object in motion. Camera A is configured to capture first and third images at first and third time points, respectively. Camera B is configured to capture second image at second time point. The third time point is after the first time point, and the first time point is after the second time point. In another exemplary embodiment, the second time point is after the third time point, and the third time point is after the first time point. Yet, in another exemplary embodiment, the third time point is after the second time point, and the second time point is after the first time point. In some embodiments, cameras A and B are positioned at different locations.

[0138] FIG. 9 illustrates distances between points in a three-dimensional (3D) world coordinates, according to an embodiment. In the example shown, there are three points p₁, p₂ and p₃ of a trajectory of a projectile 900 (e.g., golf ball) in 3D world coordinates. It is understood that projectile may also include any other balls including but not limited to tennis ball, baseball, softball, cricket ball and so forth. A first distance p₁₂ is between points pi and p₂. A second distance p₂₃ is between points p₂ and p₃. The total distance L is equal to the sum of distances p₁₂ and p₂₃. In some embodiments, the points p₁, p₂ and p₃ are assumed to be linear locally (as shown) rather than a curve. In some embodiments, it is under an assumption that the points are captured by a camera 1100 within a short period of time, e.g., between 50-100 ms for a fast-moving object/ball, such as a golf ball. It is to be understood that a substantially vertical line connecting the three points shown in FIG. 9 is for illustration purposes only and the actual trajectory of the projectile may be horizontal, angled (upwards or downwards), or sidewards. In some embodiments, the projectile trajectory is a 3D trajectory e.g. 3D horizontal, 3D vertical, 3D angled or 3D sidewards trajectory.

[0139] Assuming a set of 3 image frames of a projectile are captured sequentially by one or more cameras while the projectile is in flight, or a single frame is captured with multiple exposures, coded exposures or a blinking flash (see, e.g., as described in reference to FIG. 10). In some embodiments, at least one camera and at least one radar are used (e.g., two cameras and one radar, two cameras and two radars and so forth). In some embodiments, one camera and one radar are used.

[0140] The distances p₁₂ and p₂₃ between points p₁, p₂ and p₃ in the 3D world coordinates depend on the time intervals between the frame and the speed of the projectile 900 along its trajectory. The time intervals can be determined from the camera frame rate and/or flash intervals for the multiple exposure case. As the speed of the projectile 900 along its trajectory is unknown by the camera, distances p₁₂ and p₂₃ are also unknown. The ratio of the distances, however, corresponds to a ratio of their respective time intervals when the speed of the projectile 900 is constant over a short time interval. Accordingly, having these assumptions allows one to determine the positions p₁, p₂ and p₃ in the 3D world coordinates. From the determined positions p₁, p₂ and p₃, the speed of the projectile 900 can be determined. In some embodiments, the speed of the projectile 900 is the speed of the projectile of the substantially linear trajectory of the projectile.

[0141] FIG. 10 illustrates image points in a two-dimensional (2D) image plane 1000 corresponding to the points in the 3D world coordinates, according to an embodiment. In some embodiments, the 2D image plane may be located on x- and y-axes. In this example, 2D image frame 1000 includes image points q₁, q₂, and q₃ corresponding to the points p₁, p₂ and p₃ in 3D world coordinates, respectively. A first distance q₁₂ is between image points q₁ and q₂. A second distance q₂₃ is between image points q₂ and q₃.

[0142] FIG. 11 illustrates calculation of depths of points p₁, p₂ and p₃ in the 3D world coordinates based on the image points q₁, q₂, and q₃, the distances between image points q₁₂ and q₂₃ in the 2D image plane 1000, and a homographic transformation matrix H, according to an embodiment. The parameter p also refers to the position of the projectile 900 in the 3D world coordinates and po refers to 3D point of the projectile at time to. As can be seen from FIG. 11, an imager 1101 for e.g. a mono camera 1101 is used to capture a set of images of the projectile 900.

[0143] To determine depths z₀₁, z₀₂ and z₀₃ of points p₁, p₂ and p₃ in 3D world coordinates, a projection of the following form of equation can be used:

[0144] $\begin{array}{cc}{\stackrel{\rightharpoonup }{p}}_{n}H=z_{0n}{\overrightarrow{q}}_n& \left(12\right)\end{array}$
where H is the homographic transformation matrix previously described above (its units may be in pixel per meter), _n are the unknown points in 3D world coordinates (its units may be in meter), _n are the known image points in the 2D image plane 1000 (its units may be in pixel) and z_0n are the unknown depths, where n is equal to the number points, which in this example n=3. It is understood that n can be any other integers, for example 4, 5, 6 and so forth. In some embodiments, depth z_0n including depths z₀₁, z₀₂ and z₀₃ of points p₁, p₂ and p₃ in 3D world coordinates may refer to normalized depth where actual 3D depth (measured in metric units) has been normalized by a constant or arbitrary scale of the homographic transformation matrix H.

[0145] A cross-ratio λ can then be determined from the 2D image plane 1000 and is defined as follows:

[0146] $\begin{array}{cc}\lambda =\frac{q_{03}{q}_{12}}{q_{01}{q}_{23}}\mathrm{or}\lambda =\frac{q_{12}}{q_{23}},& \left(13\right)\end{array}$
assuming the 3D point ₀ is on the image plane 1000 having a depth of zero, the 2D point ₀ is at infinity, as can be seen from FIG. 11.

[0147] A cross-ratio λ is determined in 3D world coordinates is as follows:

[0148] $\begin{array}{cc}\lambda =\frac{p_{03}{p}_{12}}{p_{01}{p}_{23}}=\beta \frac{p_{03}}{p_{01}}=\beta \frac{z_{03}}{z_{01}},& \left(14\right)\end{array}$
where a ratio β is defined as

[0149] $\beta =\frac{p_{12}}{p_{23}}=\frac{t_{12}}{t_{23}},$
and t₁₂ and t₂₃ are the known time intervals between points p₁ and p₂ and between points p₂ and p₃, respectively. Note that λ is measured from the image and β is known by the time intervals between frames (or flashes).

[0150] The depths z₀₁, z₀₂ and z₀₃ are bound to each other by the ratios λ and β as follows:

[0151] $\begin{array}{cc}{z}_{03}=\frac{\lambda }{\beta }\frac{\left(1+\beta \right)}{\left(1+\lambda \right)}{z}_{02},& \left(15\right)\end{array}$ $\begin{array}{cc}{z}_{01}=\frac{\left(1+\beta \right)}{\left(1+\lambda \right)}{z}_{02},& \left(16\right)\end{array}$
where

[0152] $\frac{z_{03}}{z_{01}}=\frac{\lambda }{\beta }.$

[0153] The linear velocity vector v₂ at point p₂ can be represented as:

[0154] $\begin{array}{cc}{\stackrel{\rightharpoonup }{v}}_2=\frac{\left({\stackrel{\rightharpoonup }{p}}_3-{\stackrel{\rightharpoonup }{p}}_1\right)}{\left(t_3-t_1\right)}=\frac{\left(z_{03}{\stackrel{\rightharpoonup }{q}}_3-z_{01}{\stackrel{\rightharpoonup }{q}}_1\right)H^{-1}}{\left(t_3-t_1\right)}=z_{02}\frac{\frac{1+\beta }{1+\lambda }\left(\frac{\lambda }{\beta }{\stackrel{\rightharpoonup }{q}}_3-{\stackrel{\rightharpoonup }{q}}_1\right)H^{-1}}{\left(t_3-t_1\right)},& \left(17\right)\end{array}$
where z₀₂ is unknown.

[0155] The radial velocity u₂ at point ₂ computed by the radar including the CW radar (scalar, by Doppler radar) can be represented as the projection of linear velocity ₂ onto the direction from the radar at point _c (with _c being radar position or camera position) to the projectile at point ₂ given by:

[0156] $\begin{array}{cc}{u}_2=\frac{\stackrel{\rightharpoonup }{v_2}\left({\stackrel{\rightharpoonup }{p}}_2-{\stackrel{\rightharpoonup }{p}}_c\right)}{❘{\stackrel{\rightharpoonup }{p}}_2-{\stackrel{\rightharpoonup }{p}}_c❘}\mathrm{or}{u}_2=\frac{\beta +1}{\lambda +1}\frac{\left({\stackrel{\rightharpoonup }{q}}_3{\stackrel{\rightharpoonup }{q}}_1\right){H^{-1}\left({\stackrel{\rightharpoonup }{p}}_2-{\stackrel{\rightharpoonup }{p}}_c\right)}^T}{\left(t_2-t_1\right)❘{\stackrel{\rightharpoonup }{p}}_2-{\stackrel{\rightharpoonup }{p}}_1❘}\mathrm{or},& \left(18\right)\end{array}$ $z_{02}=u_2\left(t_3-t_1\right)\frac{\lambda +1}{\beta +1}\frac{\sqrt{{{\stackrel{\rightharpoonup }{q}}_2\left(H_r^T{H}_r\right)}^{-1}{q}_2^T}}{\left(\frac{\lambda }{\beta }{q}_3-q_1\right){\left(H_r^T{H}_r\right)}^{-1}{q}_2^T},$
where H_r is a rotational part of the homographic transformation matrix H. With λ, β and time intervals known or computed as described above, equation (18) can then be used to determine depth z₀₂ at point p₂. In some embodiments, other depths z₀₁ and z₀₃ at points p₁ and p₃ can be obtained using equations (15) and (16).

[0157] In some embodiments, using equation (12), points in 3D world coordinates can be determined by _n=z_0nnH⁻¹ (12a). This equation can be further represented as _n−_c=z_0nnH_r⁻¹ (12b). Radar or camera position above may be obtained from _c=−H_sH_r⁻¹ (12c), where H_s is translation part of the homographic transformation matrix H. Note that the depths z₀₁, z₀₂ and z₀₃ can be solved for in any particular order. For example, z₀₂ can be solved, followed by z₀₁ and then z₀₃. Alternatively, z₀₂ can be solved, followed by z₀₃ and then z₀₁.

[0158] During the operation as can be seen from the derivation of the formulas above, it is not necessary to recover the explicit intrinsic and/or extrinsic camera parameters. In other words, the determination of the depth described above and also the speed of the projectile as will be described below, can be undertaken without the decomposition of the camera parameters into extrinsic and intrinsic parameters.

[0159] As described herein, having determined the depths z₀₁, z₀₂ and z₀₃. in equation (12) can be used to solve for the points p₁, p₂ and p₃ in 3D world coordinates.

[0160] In some embodiments, the speed of the projectile can then be determined by the points in 3D world coordinates and time interval:

[0161] $\begin{array}{cc}{\stackrel{\rightharpoonup }{v}}_2=\frac{\left({\stackrel{\rightharpoonup }{p}}_3-{\stackrel{\rightharpoonup }{p}}_1\right)}{\left(t_3-t_1\right)}.& \left(19\right)\end{array}$

[0162] FIG. 12 is a flow diagram of example process 1200 of determining the position and speed of a projectile in 3D world coordinates, according to an embodiment.

[0163] Process 1200 includes the steps of: capturing, with at least one camera, a sequential set of images of a projectile in motion (1201); determining, with at least one radar, a radial velocity of the projectile (1202); determining, with the at least one processor, a set of two-dimensional (2D) image points representing respective locations of the projectile in the sequential set of images (1203); determining, with the at least one processor, respective depths of the 2D image points in their respective 2D image planes based on the 2D image points, time intervals between the captured 2D images, the radial velocity and a rotational part of a homographic transformation, wherein the homographic transformation is configured to project three-dimensional (3D) world coordinates to the 2D image planes (1204); and determining, with the at least one processor, a set of 3D world points of the projectile in the 3D world coordinates based on the 2D image points, the respective depths thereof and the homographic projection (1205). As described hereinabove, process 1200 may further include determining, with the at least one processor, the speed of the projectile in the 3D world coordinates based on the set of 3D world points and time intervals (this step is not shown in FIG. 12).

[0164] In some embodiments, the at least one camera and the at least one radar may be in the proximity of one to another, e.g., the at least one camera and the at least one radar may be collocated in a housing.

[0165] FIGS. 13-15 illustrate an alternative process of determining the position and/or speed of a projectile in 3D world coordinates using a single image point, according to an embodiment.

[0166] Referring to FIG. 13, a projectile 1300 is shown passing above a tracking device 1301. In some embodiments, alternatively, the projectile 1300 may be passing on the side or below the tracking device 1301. The tracking device 1301 is configured to track the position of projectile 1300 in 3D world coordinates. In this embodiment, tracking device 1301 has one camera 1302 and one continuous wave (CW) radar 1303. In some embodiments, the one camera 1302 and the CW radar 1303 are time-synchronized. In some embodiments, tracking device 1301 may have one or more cameras 1302 and one or more CW radars 1303. In some embodiments, one or more cameras 1302 and one or more CW radars 1303 may be positioned in same housing of the tracking device 1301. In this embodiment, the one or more cameras 1302 and one or more CW radars 1303 are collocated. In some embodiments, the CW radar 1303 has a higher sampling rate than the frame rate of the camera 1302. In some embodiments, the field of views (FOVs) of the camera 1302 and radar 1303 substantially or partially overlap. In some embodiments, the CW radar 1303 can be a single-tone CW (STCW) radar.

[0167] As described in further detail below, radial speed, u, derived from radar measurements is used by the tracking device 1301 to resolve the zero-crossing point time t_z. The zero-crossing time is the time when the linear velocity vector of the projectile 1300 and the vector from the radar 1303 to the projectile 1301 is perpendicular ({right arrow over (v)}·{right arrow over (r)}=cos(90°)=0) . In this disclosure, the terms ‘zero-crossing time’ and ‘zero radial speed time’ are used interchangeably and they have the same meaning. In some embodiments, the derivative of the radial speed (represented by a slope), u′, is determined at a particular time t other than t_z, (e.g., t>t_z). The derivative of the radial speed is also referred to as ‘radial speed slope’. In some embodiments, the radial speed, u, can be measured from the spectrum/phase/frequency measurements of the radar 1303, as known by those with skill in the art.

[0168] A single 2D image captured by the camera 1302 is used to resolve a 2D image point, {right arrow over (q)}, of the projectile 1300 at time t. The 2D image point represents the location of the projectile in that image, captured by the camera 1302. As described hereinabove, the camera 1302 or each of the cameras 1302 (when two or more cameras 1302 are used), is calibrated prior to use. The calibration of the camera 1302 is undertaken without decomposition of the camera parameters into explicit extrinsic parameters and explicit intrinsic parameters as described herein. In some embodiments, from {right arrow over (q)}, an unknown depth scale k is resolved at time t. The 3D position, {right arrow over (p)}, of the projectile 1300 is then resolved at time t from the 3D position of the camera {right arrow over (p)}c, {right arrow over (q)}, k and an inverse of the rotational part of a homographic transformation matrix, H_r⁻¹, given by:

[0169] $\begin{array}{cc}\overrightarrow{p}=k\overrightarrow{q}{H}_r^{-1}+{\overrightarrow{p}}_c& \left(20\right)\end{array}$

[0170] In some embodiments, the radial distance ρ (scalar) is given by:

[0171] $\begin{array}{cc}\rho =\sqrt{\left(\overrightarrow{r}·\overrightarrow{r}\right)}=\sqrt{{\overrightarrow{r}}^2}=❘\overrightarrow{r}❘,& \left(21\right)\end{array}$
where {right arrow over (r)}={right arrow over (p)}−{right arrow over (p)}_c, is the vector from the radar 1303 (and camera 1302, if the radar 1303 and camera 1302 are collocated) to the projectile 1300.

[0172] The radial speed (scalar) is given by:

[0173] $\begin{array}{cc}u=\frac{\partial \rho }{\partial t}=\frac{\left(\overrightarrow{v}·\overrightarrow{r}\right)}{\rho },& \left(22\right)\end{array}$
where {right arrow over (v)} is a linear velocity vector.

[0174] Referring to FIGS. 14A, 14B, 15A and 15B, the radial speed of the projectile (m/s) is plotted for example trajectory 1401 of projectile 1300. The radial speed u and its slope u′ at time t (after the zero-crossing time, t_z) for trajectory 1401 is shown in FIG. 14B. FIG. 15B shows how the slope changes at time t from trajectory 1401 (dashed line) to trajectory 1501 when the projectile in motion 1300 is closer to the radar 1303. For example, the derivative of radial speed u′ (radial speed slope) for trajectory 1501 is steeper than the slope of trajectory 1401. This example illustrates how the radial speed u and its slope u′ are a function of the radial distance, ρ, from radar 1303.

[0175] In some embodiments, positions of the camera 1302 and radar 1303 are collocated, and a camera projection matrix H is given. The homographic projection matrix H may be obtained through the calibration process described herein. The homographic projection matrix is defined as H={H_r; {right arrow over (H)}_s}, the extrinsic camera matrix C is defined as C={R;{right arrow over (s)}} and the intrinsic camera matrix is defined as K. Both extrinsic and intrinsic camera parameters, C and K are implicit and therefore are not used directly, but only as a combination: H=CK. The matrix H_r=RK is the rotational part of the homographic projection matrix, H, and H_s=SK is the translational part of the homographic projection/transformation matrix, H. The term homographic transformation or homographic projection is used for the 3D to 2D projection matrix (4×3), for example for mapping from the 3D world coordinates to 2D image plane (or coordinate). This transformation matrix H is a unilateral projection from 3D space (or coordinate) to 2D space (or coordinate), but not from 2D space to 3D space.

[0176] The camera (and the collocated radar) position, {right arrow over (p)}_c, can be found as:

[0177] ${\overrightarrow{p}}_c=-{\overrightarrow{H}}_s{H}_R^{-1}.$
The vector from the camera (and radar) to the object in motion or projectile is given by:

[0178] $\begin{array}{cc}\overrightarrow{r}=\left(\overrightarrow{p}-{\overrightarrow{p}}_c\right)& \left(23\right)\end{array}$
and the radial distance ρ is given by:

[0179] $\begin{array}{cc}\rho =❘\overrightarrow{r}❘=\sqrt{\left(\overrightarrow{r}·\overrightarrow{r}\right)}=\sqrt{\left(\overrightarrow{p}-{\overrightarrow{p}}_c\right)·\left(\overrightarrow{p}-{\overrightarrow{p}}_c\right)}& \left(24\right)\end{array}$

[0180] Note that the vector dot product is given by:

[0181] $\begin{array}{cc}{\rho }^2=\left(\overrightarrow{p}-{\overrightarrow{p}}_c\right)·\left(\overrightarrow{p}-{\overrightarrow{p}}_c\right)& \left(25\right)\end{array}$

[0182] The projection of the 3D point coordinate, {right arrow over (p)}, to the 2D image point {right arrow over (q)} is defined by the homographic projection matrix H as:

[0183] $\begin{array}{cc}\left(\overrightarrow{p}-{\overrightarrow{p}}_c\right)\propto \overrightarrow{q}{H}_r^{-1}\mathrm{or}\overrightarrow{r}\propto \overrightarrow{q}{H}_r^{-1},& \left(26\right)\end{array}$
up to unknown “depth scale” scale k:

[0184] $\begin{array}{cc}\left(\overrightarrow{p}-{\overrightarrow{p}}_c\right)=k\overrightarrow{q}{H}_r^{-1}.& \left(27\right)\end{array}$

[0185] The unknown depth scale k can be found from the unknown range ρ as:

[0186] $\begin{array}{cc}k=\frac{\rho }{\sqrt{{q\left(H_r^T{H}_r\right)}^{-1}{q}^T}},& \left(28\right)\end{array}$ $\mathrm{where}$ $\begin{array}{cc}\rho =\frac{u^2\left(t-t_z\right)}{u-u^{\prime }\left(t-t_z\right)},& \left(29\right)\end{array}$
and the absolute speed is given by:

[0187] $\begin{array}{cc}❘v❘=\sqrt{\frac{u^3}{u-u^{\prime }\left(t-t_z\right)}},& \left(30\right)\end{array}$
where t is the time of a single radar measurement, and t_z(t_z≠t) is the radial speed zero crossing time.

[0188] With the depth scale k determined, the 3D position vector, {right arrow over (p)}, of the projectile 1300 in 3D world coordinates can be computed from Equation (20). In some embodiments, the 3D position of the projectile is the 3D position of the projectile on a locally linear trajectory. In some embodiments, the 3D position of the projectile is the 3D position of the projectile on a locally linear trajectory and the linear portion of the trajectory is located nearby zero-crossing (See the term ‘zero-crossing time’ defined hereinabove).

[0189] FIG. 16 is a flow diagram of example process 1600 of determining the position and speed of a projectile in 3D world coordinates, according to an embodiment.

[0190] Process 1600 includes determining a zero radial speed time of the projectile based on radar measurements of the projectile (1601); determining a radial speed of the projectile at a particular time based on the radar measurements (1602); determining a radial speed slope at the particular time based on the radial speed measurements (1603); determining 2D image point of the projectile at the particular time, the 2D image point representing the location of the projectile in an image captured by a camera (1604); determining a depth scale coefficient at the particular time based on the zero radial speed time, the radial speed, the radial speed slope, the 2D image point, and a homographic transformation (1605); and determining a position of the projectile in 3D space at the particular time based on a three-dimensional (3D) position of the camera, the depth scale coefficient, the 2D image point and the homographic transformation (1606). Each of these steps were described in detail in reference to FIG. 16.

[0191] In some embodiments, the at least one camera and the at least one radar may be in the proximity of each another, e.g., the at least one camera and the at least one radar may be collocated in a housing such that the FOV of the camera and the FOV of the radar partially or substantially overlaps.

[0192] In some embodiments, to contextualize the position of the projectile 1300 relative to the tracking device 1301, the projectile 1300 is passing above the tracking device 1301 when the tracking device 1301 is placed on the ground. In a sport setting, the tracking device 1301 may be used to monitor the trajectory of, e.g., a baseball, where the tracking device 1301 is placed on the ground between a hitter and a pitcher. In this example scenario, the tracking device may be used to monitor the trajectory of the ball pitched by the pitcher and/or after the ball is hit by the hitter. Alternatively, or additionally, the tracking device 1301 may be placed on the ground to monitor the trajectory of a golf ball, where the tracking device 1301 is oriented in-line or perpendicular to the direction of the swing performed by a player. In some embodiments, the tracking device 1301 may be placed on an elevated surface to track the trajectory of the projectile 1300.

[0193] In some embodiments, there is also provided a device or an apparatus where the device includes a radar, a camera and at least one processor. In some embodiments, the at least one processor is configured to determine a zero radial speed time of the projectile based on radar measurements of the projectile. The at least one processor is further configured to determine a radial speed of the projectile at a particular time based on the radar measurements. The at least one processor is also configured to determine a radial speed slope at the particular time based on the radial speed measurements captured by the radar. The at least one processor is additionally configured to determine a two-dimensional (2D) image point of the projectile at the particular time, the 2D image point representing the location of the projectile in the image. Additionally, the at least one processor is configured to determine a depth scale coefficient at the particular time based on the zero radial speed time, the radial speed slope, the 2D image point, and a homographic transformation and to determine a position of the projectile in 3D space at the particular time based on a 3D position of the camera, the depth scale coefficient, the 2D image point and the homographic transformation.

[0194] The specific embodiments of the present disclosure are described above, but various modifications to the specific embodiments are possibly implemented without departing from the scope of the present disclosure. However, it is apparent to a person of ordinary skill in the art to which the present disclosure pertains that various alterations and modifications are possible without departing from the nature and gist of the present disclosure. Therefore, the embodiments of the present disclosure are for describing the technical idea of the present disclosure, rather than limiting it, and do not impose any limitation on the scope of the technical idea of the present disclosure. Accordingly, the scope of the present disclosure should be defined by the following claims. All equivalent technical ideas should be interpreted to be included within the scope of the present disclosure.