Real-time target confirmation for radiation therapy

[0001] Reference is made to commonly-assigned copending U.S. applications Ser. No. 11/039,422, filed Jan. 20, 2005, entitled RADIATION THERAPY METHOD WITH TARGET DETECTION; and Ser. No. 11/221,133, filed Sep. 7, 2005, entitled ADAPTIVE RADIATION THERAPY METHOD WITH TARGET DETECTION by Schildkraut et al., the disclosure of which are incorporated herein.

[0002] The invention relates generally to radiation therapy systems, and in particular, to real-time target confirmation for radiation therapy.

[0003] Organ motion and setup error are major causes of target position uncertainty in external radiotherapy for extracranial targets. Uncertainty of target position and shape can result in decreased radiation dose to the target and an increased dose to the surrounding normal tissues. To compensate for the uncertainty of the target position and shape in irradiation process the planning target volume (PTV) must have a larger margin compared to static targets with the same clinical target volume (CTV). This approach increases the probability that the target will receive a lethal dose of radiation. Unfortunately, it also increases collateral damage to surrounding healthy tissues. Some approaches resort to using a learned target motion trajectory model from a 4D CT in the planning phase to guide the radiation beam in the treatment phase, which obviously has the drawback of mismatch between the model trajectory and the actual target motion trajectory.

[0004] U.S. Pat. No. 6,804,548 (Takahashi et al.) is directed at a system and method for monitoring irradiation target movement and recognizing irradiation target position in an irradiation process without implanted fiducial markers. The method disclosed in '548 employs a high resolution 3D imaging device such as CT or MRI for taking two sets of 3D images of the irradiation target region (one set, denoted by H₀, in radiation planning phase, and another set, denoted by H₁, in the treatment phase immediately before the start of irradiation). The method disclosed in '548 uses mutual information measure to compare the irradiation target region extracted from H₀ and irradiation target region extracted from H₁. As a result of the comparison, the method computes the matching between H₀ and H₁, and obtains a transformation function incorporating the changes in the irradiation conditions from the irradiation plan. The method disclosed in '548 also employs a real-time imaging device such as an echograph for taking images of the irradiation target region immediately before the start of irradiation and also during the irradiation procedure. Note that for small targets echograph may not be an ideal imaging modality. The real-time image taken immediately before the start of irradiation is denoted by R₀. The real-time images taken during the irradiation procedure are denoted by R_n. The high resolution image set H₁ and real-time image R₀ are taken at nearly the same time but with different modalities. The high resolution image set H₁ is then superimposed with the real-time image R₀ so that the new irradiation plan is reflected in the real-time image R₀. During the irradiation treatment, the method '548 compares the real-time images R_n and R₀. According to the result obtained by the comparison, the method identifies the portion in the real-time image R_n, which corresponds to the irradiation target in the real-time image R₀, extracts the target region in the real-time image R_n and computes the position and direction in the 3D coordinate system in the work space for measurement. The method makes a decision as to whether the target tumor is present or not in the region expected in the irradiation plan. To compute the exact position of the target, the method disclosed in '548 installs 3D position and direction sensors such as infrared rays, ultrasound, or magnet on the treatment table, real-time imaging device and high resolution imaging device to find relative position and direction between them.

[0005] Marker-less target position monitoring during radiation treatment is a needed feature for radiotherapy to increase accuracy and mitigate damages to normal tissues. However, it is known that extracranial targets may change their shape due to tissue deformation during a bodily process, e.g. a respiration cycle. Furthermore, targets may shrink after a number of fractions of treatment. Therefore, shape tracking is also very much desirable especially for conformal radiation by using automatic beam shaping device such as a multi-leaf collimator (MLC).

[0006] Methods of using position and direction sensors to find relative position and orientation of imaging devices to a reference 3D system only solves the problem of extrinsic parameters (e.g. position and orientation of the device with respect to a 3D radiotherapy system) estimation for these devices. To compute 3D position for the target region, the intrinsic parameters (e.g. distance from an X-ray source to a detector and pixel pitch of the detector) of an imaging device must be provided. Intrinsic parameters of an imaging device may be obtained from the device specification sheets and on-site installation specs. However, intrinsic parameters are largely device dependent. For instance, the distance from an X-ray source to X-ray detector can change from device to device within a certain statistical range for the type of devices.

[0007] U.S. Patent Application Publication No. 2005/0180544 A1 (Sauer et al.) discloses a system and method for patient positioning for radiotherapy in the presence of respiratory motion. The method disclosed in '544 teaches using one or two X-ray imagers to acquire two sequences of the region that contains the target with an invasively implanted marker. If one X-ray imager is used, images are taken alternatively at two different angles (0° and 90°). The frequency of the image acquisition within a sequence is fixed and trigged by a respiratory monitoring device (noted that the fixed frequency mode may not be ideal because the organ motion caused by respiration is non-linear in nature in terms of 3D positions). After obtaining two sequences of X-ray images of the target region, method '544 teaches using the triangulation scheme to compute 3D coordinates of the target to form a 3D target position trajectory. The 3D target position trajectory enables radiation treatment with beam tracking or gating, thus allowing for motion compensation for all fields in which radiation doses are being delivered. In method '544, the 3D target position is also compared with a 4D target position data in a sequence of CT images to see if they match. If there is a significant change in the planned treatment, the treatment is stopped.

[0008] People skilled in the art understand that to effectively use the triangulation scheme, the intrinsic parameters of the X-ray imagers must be given, which is the same drawback that method '548 has. Noted also that method of '544 teaches manually or semi-automatically identifying targets in the captured images, which is not desirable in real-time irradiation adaptation.

[0009] The present invention is designed to overcome the problems set forth above.

[0010] Briefly, according to one aspect of the present invention, a method of target recognition for radiation therapy has a first target identification system for obtaining first target characteristics within a patient's body. A second target identification system for obtaining second target characteristics within the body is aided by the first target characteristics. Three-dimensional (3D) coordinates of the target region are calculated by a computation means with respect to a 3D radiotherapy system using the second target characteristics.

[0011] This invention builds on U.S. patent application Ser. No. 11/039,422, which discloses a method of real-time target detection in radiotherapy that solves the problem of detecting a target in a 2-D captured radiographic image in two ways:

• • [0012] 1) The capture configuration for a radiograph at treatment time is based on an analysis of digitally reconstructed radiographs (DRR) that are generated from a CT planning image. This analysis determines capture conditions for which the target can be directly detected.
  • [0013] 2) Powerful image processing techniques are used that enable target detection in the presence of superimposed anatomical structures.

[0014] This invention further builds on U.S. patent application Ser. No. 11/221,133, which discloses a method of identifying the region in a captured radiographic image that corresponds to the region of the target's projection in the image. This is accomplished by first, in the planning phase, determining processing conditions that result in the identification of the region of the target's projection in a DRR. A region is identified in the DRR by a method of image segmentation. The identified region is compared with the target's projection in this image. The segmentation process is optimized until the identified region and the target's projection are substantially the same. In the treatment phase, the optimized segmentation procedure is applied to a captured radiographic image in order to identify a region at or near the isocenter. Features of the region identified in the DRR are compared with those of the region identified in the captured radiographic image. Based on this comparison, the probability that the identified region in the captured radiographic image in the target is determined. This probability and the location of the identified region in the captured radiographic image are used to modify the delivery of therapeutic radiation.

[0015] This invention provides methods of target recognition and tracking without implanted fiducial markers for radiotherapy target confirmation. These methods essentially identify, in real-time, target characteristics including not only the morphological and spatial (3D physical location) information but also other features of the target. The method of target recognition mainly deals with measuring patient set-up (spatial and morphological) errors. The method of target tracking deals with target monitoring during the radiotherapy process.

[0016] The target morphological contour computation of the present invention is accomplished as following. In the planning phase, a volumetric target region is delineated from a volume image (CT) of the patient body. Without performing the reconstruction of a digitally reconstructed radiograph (DRR), the target contour in two-dimensional space can be calculated from the delineated volume surface of the target. This calculated contour is represented by a morphological descriptor that can be used to initialize the shape of a radiation collimator. In the treatment phase, the target region is identified (using the method disclosed in Ser. No. 11/221,133) and the region contour is represented by a morphological descriptor that is compared with the morphological descriptor generated in the planning phase, and can be used to redefine the shape of the radiation collimator. The morphological descriptor generated in the treatment phase can also be used in visual feedback display of target shape.

[0017] The target real-time three-dimensional physical location computation of the present invention facilitates measuring the set-up error between the isocenter and the target position in three directions (3D space).

[0018] The target real-time three-dimensional physical location computation of the present invention does not rely on knowing X-ray imagers' extrinsic parameters (e.g. position and orientation of the device with respect to a 3D radiotherapy system) and the intrinsic parameters (e.g. distance from an X-ray source to a detector and pixel pitch of the detector). This approach also does not require a perfect alignment of image planes of the X-ray imagers.

[0019] To accomplish the 3D computation, an adjustable phantom is devised to facilitate finding the 3D to 2D computational models of the X-ray imagers. These models assist the computation of the target real-time three-dimensional physical location. The adjustable phantom has markers that are detachable so that marker distribution patters are reconfigurable for different applications. The accurate physical locations of the markers are measured.

[0020] To obtain the 3D to 2D computational models of the X-ray imagers, the phantom is placed on the couch and two radiographs of the phantom are acquired with the X-ray imagers used in the treatment phase. The projections of the markers of the phantom are identified in the radiographs. The corresponding projections of the markers in two radiographs are established. The 3D to 2D computational models of the X-ray imagers then can be estimated by using the established corresponding projections of the markers and the measured physical locations of the markers.

[0021] In the treatment phase, with the same X-ray imagers, two radiographs of the patient body are captured. Using the algorithms devised in Ser. No. 11/221,133, target regions in the two radiographs can be identified.

[0022] Corresponding feature points such as region centroids in the two target regions in the two radiographs are then computed. Physical 3D locations of the corresponding feature points are subsequently calculated using the 3D to 2D computational models of the X-ray imagers. The computed physical 3D locations are used to represent the target location in 3D space for confirmation.

[0023] The method of target tracking during a bodily process of the present invention identifies target characteristics in the projective radiographs in the treatment phase is aided by the identified target characteristics in the planning phase at the beginning of the tracking process. The subsequent target characteristics tracking is aided by the previously identified target characteristics in the treatment phase. Radiographic data collection (radiograph acquisition) for target characteristics identification is regulated according to the target motion nonlinear property. The data collection is initiated and terminated by a device that monitors a bodily process (e.g. respiration).

[0038] The following is a detailed description of the preferred embodiments of the invention, reference being made to the drawings in which the same reference numerals identify the same elements of structure in each of the several figures.

[0039] FIG. 1 depicts a configuration of projective imaging devices of the second target identification system together with a radiation therapy equipment for target confirmation according to an exemplary embodiment of the current invention, including exemplary X-ray imaging source 1 (102), X-ray imaging source 2 (104), image panel 1 (106), image panel 2 (108), body 110, couch 111, gantry 112, three-dimensional radiotherapy system (3D coordinate system) 114 and control center 120.

[0040] The gantry 112 includes a radiation beam source such as a linear accelerator that is positioned to irradiate a target in body 110 located on couch 111. The imaging X-ray sources 1 (102) and 2 (104) can be mounted on ceilings. Alternatively, the X-ray imaging sources 1 (102) and 2 (104) can be mounted on fixtures that are attached to the gantry 112. The X-ray image sources 1 and 2 may be oriented differently from what shown in FIG. 1. The X-rays for imaging sources 1 (102) and 2 (104) are in the diagnostic KeV range for maximizing the detectability of the target. The image panels 1 (106) and 2 (108) may be full-field direct detector arrays such as Kodak DR 9000 with an active area of 14×17 inch and a 139-μm detector element pitch. The image panels 1 (106) and 2 (108) are installed opposite to the image sources and in such a way that enough clearance exists in order to avoid a collision with the gantry when the gantry rotates around the radiation isocenter.

[0041] The radiation equipment may include an exemplary verification X-ray flat panel detector (not shown) installed underneath the couch 111 for high-energy (MeV) radiotherapy beam dose verification. By using the verification X-ray flat panel detector, an alternative configuration (not shown) of the projective imaging devices of the second target identification system can be realized. Instead of using separate image panels, X-ray sources 1 and 2 may time-share the verification X-ray flat panel detector to simplify the system.

[0042] FIG. 2 is a supplement to FIG. 1 illustrating perspective projection structure of the projective imaging devices of the second target identification system shown in FIG. 1, including a 3D world (radiotherapy) system 212 (also 114) denoted by (X⁰, Y⁰, Z⁰), a 3D local coordinate system 216, denoted by (X¹,Y¹,Z¹), associated with X-ray s₁ (202, also 102), a 3D local coordinate system 214, denoted by (X²,Y²,Z²), associated with X-ray s₂ (204, also 104). Without loss of generality, the origin, o₁, of (X¹,Y¹,Z¹) is set at the center of an image plane I₁ (220, also 106), the origin, o₂, of (X²,Y²,Z²) is set at the center of an image plane I₂ (218, also 108), and the origin, o₀, of (X⁰,Y⁰,Z⁰) is set arbitrarily between o₁, and o₂. Imaging sources s₁ (202) and s₂ (204) are modeled as pinhole X-ray cameras. The perspective projection structure of the projective imaging system shown in FIG. 2 are to be used in deriving 3D to 2D computational models of projective imaging devices of the second target identification system later.

[0043] The control center 120 may include electronic and mechanical units used for controlling couch position, gantry rotation, X-ray imaging device position and orientation, and other devices such as in vitro respiration monitoring unit (not shown). It may also include signal and image processing, graphic user interface and input/output devices that are useful in various procedures described next.

[0044] Turning now to FIGS. 3A and 3B, the methods of the present invention will be described. FIG. 3A is a flow chart illustrating one embodiment of the method for target recognition for radiotherapy of the present invention. Prior to radiotherapy treatment, a volumetric image of the patient body is captured in step 301. From the volumetric image, a volumetric target can be delineated manually with the help of image processing tools. Volumetric images are usually CT or MRI scans. A volumetric image of the target is useful especially in conformal radiotherapy that is most beneficial to the patient for targets that are close to important organs and structures in the body. With the conformal radiotherapy, less healthy tissue is included in the radiotherapy field because metal blocks are put in the path of the radiation beam to change the shape of the beam so that it conforms more closely to the shape of the target.

[0045] The volumetric image also plays a role in patient position verification. Conventionally, to transfer target location information from cross-sectional computed tomographic (CT) scans or magnetic resonance images to the simulation and verification portal images used in planning radiotherapy, radio-opaque markers attached to the patient are used as reference points. The target and any desired normal structures are then outlined manually on each CT section by a radiologist.

[0046] An embodiment of target recognition method of the present invention is detailed next.

[0047] The volumetric image of the body obtained in step 301 contains three-dimensional target region information that is to be transformed (projected) to two-dimensional space. An exemplary setup 302 for transforming (projecting) information from 3D volumetric data to 2D space is depicted in FIG. 4. A volumetric image 401 is placed in a three-dimensional space 412 as if it is placed on the couch 111. In FIG. 4, two exemplary virtual X-ray pairs, X-ray source 1 (402) and panel 1 (406), and X-ray source 2 (404) and panel 2 (408) are the adjustable projective imaging devices for a first target identification system. These two projective imaging devices are optimally positioned and oriented so that overlap of normal anatomy with the target is minimized and the boundary of the target is distinct in the 2D image panels 406 and 408 (see the embodiment of the method of radiation therapy with target detection in the co-pending U.S. patent application Ser. No. 11/039,422).

[0048] The position and. orientation information of the virtual X-ray imagers obtained in step 304 is input (arrow 305) to step 302 to adjust the position and orientation of two exemplary adjustable projective imaging devices, X-ray pairs 102-106 and 104-108, for the second target identification system.

[0049] This adjustment of the position and orientation of X-ray pairs 102-106 and 104-108 results in two X-ray radiographs (106 and 108) that contain, in theory, the same contents appeared in two virtual images (406 and 408) if the same body that is volumetrically imaged is placed on the couch. This facilitates using information gathered in the first target identification system to recognize the target in the second target identification system.

[0050] An alternative approach of the present invention to setting up the projective imaging devices of the first target identification system and that of the second target identification system is to use (arrow 303) the position and orientation information of X-ray pairs 102-106 and 104-108 in step 304 to adjust the position and orientation of the virtual X-ray pairs 402-406 and 404-408. As revealed in the co-pending U.S. patent application Ser. No. 11/039,422, the measuring position and orientation of X-ray pairs 102-106 and 104-108 can be accomplished, for example, with the use of markers that are detected by digital cameras or by any other means of measuring position and orientation.

[0051] After the calibration (setup) of the projective imaging devices of the first target identification system and that of the second target identification system, the volumetric target region is ready to be transformed (projected) to two-dimensional space to facilitate target recognition in the second target identification system. FIG. 8A illustrates an exemplary target volume 800 whose boundary is delineated by the radiologist. With the known boundary position in the volumetric image and the measurement of the position and the orientation of the projective imaging devices of the first target identification system, the projections of the target volume 800 can be readily computed through projective geometry. In practice, it may project just a partial volume that contains the target region and related volumes along the projection path onto the 2D projective radiograph space (step 308). These two projective radiographs are used in step 312 to identify target characteristics.

[0052] Note that it is not necessary to perform volume projection if only the target contour in two-dimensional space is needed. The target contour can be easily calculated from the delineated volume surface of the target without performing the reconstruction of a digitally reconstructed radiograph (DRR). An exemplary calculation of the target contour is briefly explained next.

[0053] Define a local target 3D coordinate system (X^v,Y^v,Z^v) 820 whose axes are parallel to that of a projective imaging device of the first system. Denote the delineated volume surface 800 by V(x^v,y^v,z^v) in said coordinate system (X^v,Y^v,Z^v) 820. Note that the axes of coordinate system (X^v, Y^v, Z^v) 820 are in parallel to that of one of the local three-dimensional coordinate systems of the projective imaging devices (X¹,Y¹,Z¹) or (X²,Y²,Z²) at a time. The origin, o^v, of the system (X^v,Y^v,Z^v) can be arbitrarily placed inside or outside the volume 800.

[0054] It is obvious that at a certain height z_n^v the expression V_n(x^v,y^v,z_n^v) represents a surface curve of the delineated target volume surface in a plane that is parallel to the projection plane of the projective imaging device of the first system, where z_n^v is a constant. Exemplarily curves are shown in FIG. 8C as V₁(x^v,y^v,z₁^v) (828), V₂(x^v,y^v,z₂^v) (830), and V₃(x^v,y^v,z₃^v) (832).

[0055] The calculation of the tumor contour starts by collapsing, in the z^v direction, all the surface curves V_n(x^v,y^v,z_n^v) into a common two-dimensional plane of z_n^v=n that is parallel to the projection plane of the projective imaging device of the first system. Note that z^v direction is perpendicular to said plane.

[0056] As is shown in FIG. 8D, an exemplary common plane is the (X^v,Y^v) plane (822) of the coordinate system (X^v,Y^v,Z^v) (820), that is, z_n^v=0. After collapsing the original curves, three new curves V₁(x^v,y^v,0) (838), V₂(x^v,y^v,0) (840), and V₃(x^v,y^v,0) (842) appear in the (X^v,Y^v) plane.

[0057] The calculation process then performs vertical and horizontal scans across the plane. An exemplary vertical scan line 826 intersects with the three exemplary curves six times. The outmost intersection points 844 and 846 on this exemplary scan line are saved to a contour point list. An exemplary horizontal scan line 824 intersects with the curves six times. The outmost intersection points 848 and 850 on this exemplary scan line are collected and saved to the contour points list. The calculated contour points are represented by a morphological descriptor that can be used to initialize the shape of a radiation collimator.

[0058] In a normal radiograph or in a DRR, the contour finding is accomplished by separating projection of the target volume and the projection of the neighboring voxels with the method disclosed in the co-pending U.S. patent application Ser. No. 11/221,133.

[0059] For convenience, denote the target characteristics by a symbol C that contains data members of features, position (spatial parameter), descriptor (morphological parameter), functions functions, and metadata metadata.

[0060] Member C.features describes region area, statistics, gradient, texture, surface etc. The method of extraction of C.features is revealed in the co-pending U.S. patent application Ser. No. 11/221,133. Exemplary C.functions are image and computer vision operations such as classification, segmentation, and registration. Exemplary C.metadata are the time when the target processed and device specs.

[0061] An exemplary C.position is a centroid of the target region that describes the region gravity center and can be readily computed. FIG. 8B illustrates the projection 802 of the target volume and its immediate neighboring voxels. Projection 802 may be generated by one of the virtual X-ray imagers, for instance, virtual X-ray imager pair 402-406. Each grid 804 represents a pixel in projection 802. The gray area is the projection of the target volume, the white area is the projection of the neighboring voxels.

[0062] Denote the coordinate system of projection 802 by (c₁,c₂); the origin of the coordinate system is at the center of the projection 802 as shown in FIG. 8B. The C.position 808 of the exemplary target projection is then computed as
C.position.c₁=m₁₀/m₀₀
C.position.c₂=m₀₁/m₀₀
where C.position.c₁ is the coordinate of C.position in the horizontal direction and C.position.c₂ is the coordinate of C.position in the vertical direction, and the moments are computed as:

[0063] $m_{\mathrm{pq}}={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{c}_1^p{c}_2^{q}f\left(c_1,c_2\right)d{c}_{1}d{c}_2$
where ƒ(c₁,c₂)=1 in the gray area and 0 otherwise in this application.

[0064] Member C.descriptor describes the contour of the projection of the target volume that is highlighted by black dots in FIG. 8B. An exemplary representation of C.descriptor is the well-known chaincode (see http://www.bmva.ac.uk/bmvc/1997/papers/062/node2.html) in computer vision and pattern recognition. In FIG. 8B, the chaincode (e.g. 4 (810)) is shown next to the contour. Choosing a start point 806, the representation of the contour will be C.descriptor=00077666666545434334412121.

[0065] Noted that steps 302 and 304 assure that radiographs generated in step 306 will match the radiographs produced in step 308. Therefore, if the same body that is used in step 301 is placed in the projective path of X-ray pairs 102-106 or 104-108, the same projection of the target region should be expected in the radiographs obtained in step 306. However, in practice, this rarely happens due to various reasons. The target characteristics C obtained in the first target identification system in step 312 is therefore served as an aid to identify target characteristics in the second target identification system in step 310 after acquiring projective radiographs with X-ray imager pairs 102-106 and 104-108 in step 306. For instance, functions and their associated parameters used in the first target identification system are employed in the second target identification system with or without modifications. Spatial and morphological parameters identified in the first target identification system are used as initiation parameters in the second target identification system. Features derived in the first target identification system are compared with that derived in the second target identification system.

[0066] Noted that steps 304, 308 and 312 constitute a first target identification system for obtaining target characteristics of a target region within a body; steps 302, 306 and 310 constitute a second target identification system for obtaining target characteristics of a target region within the body. It is also understood that steps 302, 303, 304 and 305 provide a calibration means for the first and second target identification systems.

[0067] An exemplary of a first target identification system is a planning target identification system in a planning phase for the radiotherapy. The first target identification system (or the first system for short) and the planning target identification system (or the planning system for short) are used interchangeably in the following discussions.

[0068] An exemplary of a second target identification system is a treatment real-time target identification system in a treatment phase for the radiotherapy. The second target identification system (or the second system for short) and the treatment real-time target identification system (or treatment system for short) are used interchangeably in the following discussions.

[0069] In step 310, target characteristics C¹ and C² are generated with the two radiographs (from image panel 1 (106, also 220) and image panel 2 (108, also 218)). Their members C¹.position and C².position are used in step 314 to calculate target 3D coordinates with respect to 3D radiotherapy system 114 (also 212).

[0070] A preferred method of calculating 3D coordinates of a physical point from its 2D projections of the present invention is to implicitly use intrinsic and extrinsic parameters of the projective imaging devices. To do so, the relationship between the three-dimensional space and two-dimensional projection space need to be found. That is equivalent to computing a 3D to 2D computational model for a projective imaging device with respect to a three-dimensional coordinate system. The process of finding the 3D to 2D computational model and using the model to calculate 3D coordinates of a physical point from its 2D projections is elaborated next.

[0071] Denote a three-dimensional physical point (206 in FIG. 2) with respect to 3D coordinate systems 212, 214 and 216 by homogeneous representations {tilde over (p)}^k=[x_p^k y_p^k z_p^k 1]^T; k=0,1,2, and its associated projections (222 and 224) by homogeneous vectors {tilde over (q)}^m=[wx_q^m wy_q^m 0 w]^T; m=1,2, where w is a scaling factor.

[0072] Noted that coordinate system 216 is a local 3D coordinate system with its (X¹,Y¹) plane aligned with the 2D coordinate system of image plane I₁(218), and coordinate system 214 is a local 3D coordinate system with its (X²,Y²) plane aligned with the 2D coordinate system of image plane I₂(218). The coordinate system 212 is a world 3D coordinate system.

[0073] It is readily to verify that the projections of the physical point 206 in the coordinate systems associated with the X-ray imaging systems (202-220 and 204-218) can be computed as

[0074] $\begin{array}{cc}{\tilde{q}}^m=F_0^m{\tilde{p}}^m;m=1,2;\mathrm{where}{F}_0^m=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 0& 0\\ 0& 0& -1/d^m& 1\end{array}\right];m=1,2;& \left(1\right)\end{array}$
and d^m is the distance between the X-Ray source s_m and the origin o_m. Noted that the line (208 or 210) s_mo_m is perpendicular to the image plane I_m(220 or 218) and aligns with axis Z^m, m=1,2. Noted also that in Equation (1), the three-dimensional point 206 is represented by {tilde over (p)}^m in local three-dimensional coordinate systems (216 and 214). To express the projections (222 and 224) in terms of the world coordinate representation {tilde over (p)}⁰, a coordinate transformation operation is applied as

[0075] $\begin{array}{cc}{\tilde{p}}^m=\prod _{i=1}^4{F}_i^m{\tilde{p}}^0;m=1,2,& \left(2\right)\end{array}$
where Π is a matrix product operator.

[0076] In the above expression, F₁^m performs a rotational transformation around the Z⁰ axis, F₂^m performs a rotational transformation around the Y⁰ axis, and F₃^m performs a rotational transformation around the X⁰ axis. F₄^m performs a translational transformation with respect to origin o along X⁰, Y⁰ and Z⁰ axes.

[0077] Substituting Equation (2) to Equation (1) yields

[0078] $\begin{array}{cc}{\tilde{q}}^m=\prod _{i=0}^4{F}_i^m{\tilde{p}}^0;m=1,2;& \left(3\right)\end{array}$
In fact, vector {tilde over (q)}^m can be shortened as {tilde over (q)}^m=[wx_q^m wy_q^m w]^T, and all the transformation matrices F_i^m can be grouped together to become a single matrix A_m. It then yields
{tilde over (q)}^m=A_m{tilde over (p)}⁰;  (4)
where A_m=[a_ij^m]; iε[1,2,3], jε[1,2,3,4] that defines the 3D to 2D computational model which is to be estimated with a preferred method described next.

[0079] With N, (N≧6), known non-coplanar 3D points {p₁⁰, . . . p_n⁰, . . . p_N⁰} and their corresponding projections {q₁¹, . . . q_n¹, . . . q_N¹} and {q₁², . . . q_n², . . . q_N²}, where p⁰=[x_p^k y_p^k z_p^k]^T and q_n^m=[x_qn^m y_qn^m]^T, the twelve unknown variables, a_ij^m; iε[1,2,3], jε[1,2,3,4], of matrix A_m can be solved for each of the X-ray cameras (m=1,2).

[0080] In practice, set a₃₄^m=1. Denote the rest eleven variables by a^m=[a₁₁^m a₁₂^m a₁₃^m a₁₄^m a₂₁^m a₂₂^m a₂₃^m a₂₄^m a₃₁^m a₃₂^m a₃₃^m]^T. The solution of a^m will be in the form of

[0081] $\begin{array}{cc}{a}^m={\left({\left(B^m\right)}^T{B}^m\right)}^{-1}{\left(B^m\right)}^T{r}^m\mathrm{where}{r}^m={\left[{\left(q_1^m\right)}^T,\dots ,{\left(q_N^m\right)}^T\right]}^T\mathrm{and}{B}^m=\left[\begin{array}{ccc}{\left({\tilde{p}}_1^0\right)}^T& 0& -{x_{q_1}^m\left(p_1^0\right)}^T\\ 0& {\left({\tilde{p}}_1^0\right)}^T& -{y_{q_1}^m\left(p_1^0\right)}^T\\ ⋮& ⋮& ⋮\\ {\left({\tilde{p}}_N^0\right)}^T& 0& -{x_{q_N}^m\left(p_N^0\right)}^T\\ 0& {\left({\tilde{p}}_N^0\right)}^T& -{y_{q_N}^m\left(p_N^0\right)}^T\end{array}\right];\mathrm{where}0=\left[\begin{array}{cccc}0& 0& 0& 0\end{array}\right]\mathrm{and}m=1,2.& \left(5\right)\end{array}$

[0082] After having estimated the 3D to 2D computational models for the two projective imaging devices, the 3D coordinates of an unknown physical point, for example p⁰(206), can be calculated by using its projections (q¹ (222) and q² (224)) and the computational models A_m, which is formularized below.

[0083] Returning back to Equation (4), by eliminating the scale factor wand rearranging the variables, Equation (4) can be re-written in a matrix operation form of
Mp⁰=N;  (6)
where

[0084] $M=\left[\begin{array}{c}{M}^1\\ M^2\end{array}\right];\mathrm{and}{M}^m=\left[\begin{array}{c}{a}_{1j}^m\\ a_{2j}^m\end{array}\right]-\left[\begin{array}{c}{a}_{3j}^m\\ a_{3j}^m\end{array}\right]q^m;$
the column index j=1,2,3, and

• [0085] m=1,2;
• [0086] p⁰=[x_p⁰ y_p⁰ z_p⁰]^T;
• [0087] N=[N¹ N²]^T; where N^m=a₃₄^m(q^m)^T−[a_i4^m]^T; the row index i=1,2, and m=1,2.

[0088] The 3D coordinates of an unknown physical point p⁰ can be optimally calculated with respect to the world coordinate system (X⁰,Y⁰,Z⁰) as
p⁰=(M^TM)⁻¹M^TN.  (7)

[0089] The above procedures for computational model estimation are illustrated in the flowchart in FIG. 6.

[0090] To obtain N, (N≧6), known non-coplanar 3D points, an adjustable phantom is developed (step 602). An illustrative phantom 510 is depicted in FIG. 5. The phantom 510 has a non-metallic base with threaded holes made on its surface. There are threaded non-metallic rods in the holes and metallic ball bearings of different diameters at the tips of the rods. The rods are of different heights. For different applications, for example, different sizes of X-ray detectors, additional rods can be added or inserted rods can be removed. The positions of the rods with respect to some reference points (e.g. corners of the base or some visible markers on the base (not shown)) and the heights of the rods are precisely measured using a high precision XYZ table, or laser measurement instrument. Some of the rods 514 are randomly selected as the known non-coplanar 3D points used for estimating the computational models A_m. Some other rods 516 are used as ground truth to test the precision of the models.

[0091] Before the radiotherapy treatment phase, the phantom is placed on the couch 111 and the position and orientation of the phantom 510 with respect to 3D radiotherapy system 512 (or 114, 212) are measured (step 604). With the projective imaging devices (X-ray imager pairs (502-506 and 504-508)), two radiographs of the phantom are captured (step 606). The 2D coordinates of the metallic ball bearings measured in the radiographs and the known 3D positions of the metallic ball bearings are used in Equation (7) (step 608) to compute 3D to 2D computational models A_m with respect to the 3D radiotherapy system (X⁰,Y⁰,Z⁰) (512, 114, 212).

[0092] FIG. 7 summarizes the workflow of estimating the 3D position of a target using 2D projections of its distinct points. In step 702 two radiographs with detected target regions are received. Distinct points (e.g. corners, centroids) of the target regions in the two radiographs are then computed in step 704. These distinct points are used in step 706 to calculate 3D coordinates of the corresponding physical points in the 3D coordinate system with the computational models estimated in step 608. Exemplary distinct points are C¹.position and C².position which are described previously.

[0093] Now turning back to FIG. 3A, the remaining steps of the method for target recognition are discussed. Step 316 receives the 3D position information of the target region from step 314 and morphological information, C¹.descriptor and C².descriptor, from step 310. Step 316 compares the received information with a treatment plan that is designed based on the identified target characteristics in the first target identification system. It then performs radiotherapy mode adaptation in response to the results of the comparison. The radiotherapy mode adaptation may include, but are not limited to, administering the dose, refraining from administering the dose, repositioning the patient, and redefining the therapeutic radiation beam shape. Concurrently, the 3D position information of the target region from step 314 and morphological information, C¹.descriptor and C².descriptor from step 310 are fed into step 318 where display devices are used to provide the patient and the oncologist with visual feedback of position and shape changes.

[0094] Recall that the control center 120 may include electronic and mechanical units used for controlling couch position, gantry rotation, X-ray imaging device position and orientation, and other devices such as in vitro respiration monitoring unit (not shown). It may also include signal and image processing, graphic user interface and input/output devices that are used to display information gathered in step 318 for oncologists.

[0095] As shown in FIG. 9, an exemplary device used in feedback visualization for the patient 904 is a head-up display 902 such as MicroOptical's SV-6 PC Viewer (see http://www.microopticalcorp.com/Products/HomePage.html). The head-up display 902 is worn by the patient 904. The patient receives information visually from a screen 906. Screen window 908 depicts an exemplary layout of the screen that has a graphics window 910 and a text window 912. Visible content shown in the graphics window 910 could be the shape and position of the target. Visible content shown in the text window could be instructions from the oncologist, status of the radiation process, time duration, and other medical related information. Visible content shown in the graphics window 910 could also be medical assistive (e.g. mind-soothing) information. Visible content shown in the text window could also be medical assistive information.

[0096] Graphics window 910 content display is manipulated by the oncologist through link 914 from the control center 120. Link 914 could be a physical cable connection or a wireless connection.

[0097] The method of target recognition essentially focus on reducing setup errors for radiotherapy by repositioning the body or modifying treatment plan with the identified target characteristics in the second target identification system. The problem of mitigating bodily process (e.g. respiration) caused irradiation uncertainty in real-time after initial setup remains to be addressed.

[0098] FIG. 3B is a flowchart illustrating one embodiment of the method of treatment target tracking for bodily process compensation of the present invention. Steps such as volumetric image acquisition, projective imaging device setup, producing projective radiographs etc. (301, 302, 303, 304, 305, 306, 308, 310, 312, 314) used in the method of target tracking were already detailed in previous discussions. Steps that are unique to the method of target tracking are explained next starting with the issue of real-time data collection.

[0099] The data collection of real-time projective radiographs using X-ray imager pairs 102-106 and 104-108 is regulated in steps 313 and 315. For a respiration cycle, data collection can be initiated when beginning to inhale and terminated at the end of exhale phase, or vise versa. The data collection process can continue for several cycles. The respiration cycle can be monitored by a breathing device such as an infrared sensing system with infrared markers placed on the body, or by a real-time respiration gating device (RPM) made by Varian. Data collection interval is determined in step 315 where irregular sampling pattern (different sampling rate) adopted to accommodate the nonlinear property of respiratory motion revealed in “Novel Prospective Respiratory Motion Correction Approach for Free-Breathing Coronary MR Angiography Using a Patient-Adapted Affine Motion Model” by D. Manke et al., Magnetic Resonance in Medicine, 2003; 50: 122-131.

[0100] Recall that in step 310, for patient setup error correction before radiation in the second system, the target characteristics identification is aided by the identified characteristics from the first system. While subsequent identification of target characteristics during the treatment can be aided by the previously identified target characteristics in the second system. Therefore there is a step 309 that performs an aid selection. If it is the initial data collection (for patient setup error correction), step 310 is executed where the target characteristics from the first system obtained in step 312 serves as the aid of identifying real-time target characteristics of the target in the second system. Otherwise, the process branches to step 311 where the target real-time target characteristics obtained in a previous data collection instance in step 311 serves as the aid (indicated by arrow 320) for identifying target characteristics of the target region in step 311 after acquiring a new set of real-time projective radiographs with X-ray imager pairs 102-106 and 104-108.

[0101] Denote data C obtained in the first system by C_p^m and that in the second system by C_t^m. The real-time target tracking aid selection can be expressed as
C_t^m(n)C_p^m; n=0
C_t^m(n+1)C_t^m(n); n>0
where is a symbol signifying the operation of aiding the detection of target region, n=0 indicating the initial data collection, m=1,2 for two projective imaging devices.

[0102] The relevant members of the target characteristics data array are used in step 316 for radiotherapy mode adaptation that was detailed in the previous discussions.

[0103] The 3D position information of the target region from step 314 and morphological information, C¹.descriptor and C².descriptor from steps 310 and 311 are fed into step 318 where display devices are used to provide the patient and the oncologist with visual feedback of position and shape changes and other information.

[0104] The relevant members of the target characteristics data array are used in step 316 for radiotherapy mode adaptation that was detailed in the previous discussions.

[0105] The subject matter of the present invention relates to digital image processing and computer vision technologies, which is understood to mean technologies that digitally process a digital image to recognize and thereby assign useful meaning to human understandable objects, attributes or conditions, and then to utilize the results obtained in the further processing of the digital image.

[0106] The invention has been described in detail with particular reference to presently preferred embodiments, but it will be understood that variations and modifications can be effected within the scope of the invention. The presently disclosed embodiments are therefore considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the appended claims, and all changes that come within the meaning and range of equivalents thereof are intended to be embraced therein.

• [0107] 102 X-ray source 1
• [0108] 104 X-ray source 2
• [0109] 106 image panel 1
• [0110] 108 image panel 2
• [0111] 110 body
• [0112] 111 couch
• [0113] 112 gantry
• [0114] 114 3D radiotherapy system
• [0115] 120 control center
• [0116] 202 X-ray (s₁)
• [0117] 204 X-ray (s₂)
• [0118] 206 physical point (p^k)
• [0119] 208 line (Z¹)
• [0120] 210 line (Z²)
• [0121] 212 3D world (radiotherapy) system (X⁰Y⁰Z⁰)
• [0122] 214 3D local coordinate system (X²Y²Z²)
• [0123] 216 3D local coordinate system (X¹Y¹Z¹)
• [0124] 218 center of image plane (I₂)
• [0125] 220 center of image plane (I₁)
• [0126] 222 projection (q¹)
• [0127] 224 projection (q²)
• [0128] 301 capture volumetric image of body
• [0129] 302 exemplary setup
• [0130] 303 arrow
• [0131] 304 obtain information of virtual X-ray imager
• [0132] 305 arrow
• [0133] 306 produce projective radiographs
• [0134] 308 compute optional projective radiographs
• [0135] 310 identify target characteristics
• [0136] 312 identify target characteristics
• [0137] 314 calculate-target coordinates
• [0138] 316 perform radiotherapy mode adaptation
• [0139] 318 visual feedback
• [0140] 320 arrow
• [0141] 401 volumetric image
• [0142] 402 virtual x-ray source 1
• [0143] 404 virtual x-ray source 2
• [0144] 406 virtual image panel 1
• [0145] 408 virtual image panel 2
• [0146] 412 3D radiotherapy system
• [0147] 502 x-ray source 1
• [0148] 504 x-ray source 2
• [0149] 506 image panel 1
• [0150] 508 image panel 2
• [0151] 510 phantom
• [0152] 512 3D radiotherapy system
• [0153] 514 rods
• [0154] 516 rods
• [0155] 602 develop adjustable phantom
• [0156] 604 measure 3D radiotherapy device
• [0157] 606 take radiographs of phantom
• [0158] 608 compute 3D to 2D computational models
• [0159] 702 receive target region detected radiographs
• [0160] 704 compute distinct points of target regions in radiographs
• [0161] 706 compute 3D coordinates of physical point
• [0162] 800 target volume
• [0163] 802 projection
• [0164] 804 grid
• [0165] 806 start point
• [0166] 808 centroid
• [0167] 810 chaincode
• [0168] 820 coordinate system
• [0169] 822 coordinate system
• [0170] 824 horizontal scan line
• [0171] 826 vertical scan line
• [0172] 828 curve
• [0173] 830 curve
• [0174] 832 curve
• [0175] 838 curve
• [0176] 840 curve
• [0177] 842 curve
• [0178] 844 intersection point
• [0179] 846 intersection point
• [0180] 848 intersection point
• [0181] 850 intersection point
• [0182] 902 head-up display
• [0183] 904 patient
• [0184] 906 screen
• [0185] 908 screen window
• [0186] 910 graphics window
• [0187] 912 text window
• [0188] 914 link