US 8,064,669 B2Grant
Fast 3D-2D image registration system with application to continuously guided endoscopy
Issue Date:2011-11-22
•19 Claims
•3 Drawing Sheets
Abstract
A novel framework for fast and continuous registration between two imaging modalities is disclosed. The approach makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to localize the cameras and register the two sources. A disclosed example includes computing or capturing a set of reference images within a known environment, complete with corresponding depth maps and image gradients. The collection of these images and depth maps constitutes the reference source. The second source is a real-time or near-real time source which may include a live video feed. Given one frame from this video feed, and starting from an initial guess of viewpoint, the real-time video frame is warped to the nearest viewing site of the reference source. An image difference is computed between the warped video frame and the reference image. The viewpoint is updated via a Gauss-Newton parameter update and certain of the steps are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source. The invention has far-reaching applications, particularly in the field of assisted endoscopy, including bronchoscopy and colonoscopy. Other applications include aerial and ground-based navigation.
Metadata
Assignee
- The Penn State Research Foundation
Inventors
- William E. Higgins
- Scott A. Merritt
- Lav Rai
Application Information
Application Number:US 13/022,241
Filing Date:2011-02-07
Priority Date:2005-05-23
Art Unit:2624
Classifications
IPC:
G06K9/00G06K9/32A61B5/05
Patent Drawings (3 sheets)
Description
Reference to Related Applications
[0001] This application is a continuation of U.S. patent application Ser. No. 11/437,230, filed May 19, 2006, which claims priority to U.S. Provisional Patent Application Ser. No. 60/683,588, filed May 23, 2005, the entire content of each application is incorporated herein by reference.
Statement of Government Sponsorship
[0002] This invention was made with government support under Grant No. CA074325, awarded by the National Institutes of Health. The Government has certain rights in the invention.
Field of the Invention
[0003] This invention relates generally to image correlation and, in particular, to a fast image registration method applicable to guided endoscopy and other fields.
Background of the Invention
[0004] Registration and alignment of images taken by cameras at different spatial locations and orientations within the same environment is a task which is vital to many applications in computer vision and medical imaging. For example, registration between images taken by a mobile camera and those from a fixed surveillance camera can assist in robot navigation. Other applications include the ability to construct image mosaics and panoramas, high dynamic range images, or super-resolution images, or the fusion of information between the two sources.
[0005] However, because the structure of a scene is inherently lost by the 2D imaging of a 3D scene, only partial registration information can typically be recovered. In many applications, depth maps can be generated or estimated to accompany the images in order to reintroduce the structure to the registration problem.
[0006] Most currently available 2D alignment algorithms use a gradient descent approach which relies on three things: a parameterization of the spatial relationship between two images (e.g., the 2D rotation and translation between two 2D images), the ability to visualize these images under any value of the parameters (e.g., viewing a 2D reference image rotated by 30 degrees), and a cost function with associated image gradient information which allows an estimate of the parameter updates to be calculated. Among the most straightforward and earliest of these algorithms is the Lucas-Kanade algorithm, which casts image alignment as a Gauss-Newton minimization problem [5]. A subsequent refinement to this algorithm includes the inverse compositional alignment algorithm which greatly speeds the computation of the parameter update by recasting the problem, allowing all gradient and Hessian information to be calculated one time instead of every iteration [6]. Several other improvements have centered around the choice of parameters and the corresponding image warps these parameterizations induce. For example, images obtained from two identical cameras observing the same scene from a different location can be approximately related by an affine transformation or an 8-parameter homography [7].
[0007] The main problem with these types of parameterizations is that they do not truly capture the physically relevant parameters of the system, and, in the case of the homography, can lead to overfitting of the image. A more recent choice of parameters attempts to match two images obtained from a camera that can have arbitrary 3D rotations around its focal point [8]. This algorithm succeeds in extracting the physically relevant parameters (rotation angles about the focal point). However, while it is able to handle small translations, it cannot handle general translation and treats it as a source of error.
[0008] Little has been done to tackle the problem of registration of two images generated by cameras related by a general rigid transformation (i.e., 3D rotation and translation). The main reason for this is that the accurate visualization of a reference image as seen from a different camera location ideally requires that the depth map associated with that image be known—something which is not generally true. In certain situations, such as a robot operating in a known man-made environment, or during bronchoscopy where 3D scans are typically performed before the procedure, this information is known. Indeed, even in situations where the depth map is unknown, it can often be estimated from the images themselves.
[0009] An example of this is the aforementioned shape-from-shading problem in bronchoscopy guidance [9]. Current practice requires a physician to guide a bronchoscope from the trachea to some predetermined location in the airway tree with little more than a 3D mental image of the airway structure, which must be constructed based on the physician's interpretation of a set of computed tomography (CT) films. This complex task can often result in the physician getting lost within the airway during navigation [1]. Such navigation errors result in missed diagnoses, or cause undue stress to the patient as the physician may take multiple biopsies at incorrect locations, or the physician may need to spend extra time returning to known locations in order to reorient themselves.
[0010] In order to alleviate this problem and increase the success rate of bronchoscopic biopsy, thereby improving patient care, some method of locating the camera within the airway tree must be employed. Fluoroscopy can provide intraoperative views which can help determine the location of the endoscope. However, as the images created are 2D projections of the 3D airways, they can only give limited information of the endoscope position. Additionally, fluoroscopy is not always available and comes with the added cost of an increased radiation dose to the patient.
[0011] A few techniques also exist that determine the bronchoscope's location by attempting to match the bronchoscope's video to the preoperative CT data. One method uses shape-from-shading, as in [2], to estimate 3D surfaces from the bronchoscope images in order to do 3D-to-3D alignment of the CT airway surface. This method requires many assumptions to be made regarding the lighting model and the airway surface properties and results in large surface errors when these assumptions are violated. A second method of doing this is by iteratively rendering virtual images from the CT data and attempting to match these to the real bronchoscopic video using mutual information [3] or image difference [4].
[0012] While these methods can register the video to the CT with varying degrees of success, all operate very slowly and only involve single-frame registration—none of them are fast enough to provide continuous registration between the real video and the CT volume. They rely on optimization methods which make no use of either the gradient information nor the known depth of the CT-derived images, and thus require very computationally intensive searches of a parameter space.
Summary of the Invention
[0013] This invention resides in a novel framework for fast and continuous registration between two imaging modalities. A method of registering an image according to the invention comprises the steps of providing a set of one or more reference images with depth maps, and registering the image to at least one of the reference images of the set using the depth map for that reference image. The image and the reference set may both be real, virtual, or one real with the other virtual. The set of reference images may endoscopic, derived from a bronchoscope, colonoscope, laparoscope or other instrument. The registration preferably occurs in real-time or near real-time, and one or more of the images in the set of reference images can be updated before, during, or after registration.
[0014] According to a robust implementation, the set of reference images represents viewpoints with depth maps and image gradients, and the image to be registered is derived from a video feed having a plurality of consecutive frames. The method includes the steps of:
[0015] a) warping a frame of the video to the nearest viewpoint of the reference source;
[0016] b) computing an image difference between the warped video frame and the reference image;
[0017] c) updating the viewpoint using a Gauss-Newton parameter update; and
[0018] d) repeating steps a) through c) for each frame until the viewpoint converges or the next video frame becomes available.
[0019] The invention makes it possible to completely determine the rigid transformation between multiple sources at real-time or near real-time frame-rates in order to register the two sources. A disclosed embodiment involving guided bronchoscopy includes the following steps:
[0020] 1. In the off-line phase, a set of reference images is computed or captured within a known environment, complete with corresponding depth maps and image gradients. The collection of these images and depth maps constitutes the reference source.
[0021] 2. The second source is a real-time source from a live video feed. Given one frame from this video feed, and starting from an initial guess of viewpoint, the real-time video frame is warped to the nearest viewing site of the reference source.
[0022] 3. An image difference is computed between the warped video frame and the reference image.
[0023] 4. The viewpoint is updated via a Gauss-Newton parameter update.
[0024] 5. Steps 2-4 are repeated for each frame until the viewpoint converges or the next video frame becomes available. The final viewpoint gives an estimate of the relative rotation and translation between the camera at that particular video frame and the reference source.
[0025] The invention has far-reaching applications, particularly in the field of assisted endoscopy, including bronchoscopy and colonoscopy. Other applications include aerial and ground-based navigation.
Brief Description of the Drawings
[0026] FIG. 1 shows a block diagram of registration algorithm for guided bronchoscopy;
[0027] FIGS. 2A-2F show source images and results for virtual-to-real registration; specifically, FIG. 2A shows a real video frame, FIG. 2B shows a warped real image at initial viewpoint, FIG. 2C shows edges from final reference image overlaid, FIG. 2D shows a reference virtual image corresponding to final registration, FIG. 2E shows a warped real image at final viewpoint, and FIG. 2F shows edges of corresponding virtual image overlaid; and
[0028] FIGS. 3A-3C show source images and sample results for virtual-to-virtual registration; specifically, FIG. 3A shows a real image, FIG. 3B shows a reference image, and FIG. 3C shows a warped real image.
Detailed Description of the Invention
[0029] Broadly, this invention is a 2D image alignment algorithm which is augmented to three dimensions by introducing the depth maps of the images. The method provides an ideal way to extend the existing matching framework to handle general 3D camera motion, allowing one to directly solve for the extrinsic parameters of the camera and localize it within its environment.
[0030] For the purpose of explaining the method in a very concrete fashion, discussion shall focus on a situation similar to that used in guided bronchoscopy. In a typical bronchoscopic procedure, a CT scan is initially performed and can subsequently be processed to extract the airway tree surfaces. The interior of the hollow airway tree constitutes the known environment. During bronchoscopy, the bronchoscope is inserted into the airways and a camera mounted on the tip transmits in real-time a sequence of real bronchoscopic (RB) video images. Assuming that the calibration parameters of the endoscope are known, virtual bronchoscopic (VB) images (endoluminal renderings) can be rendered at arbitrary viewpoints within the airway tree. It is also clear that the depths corresponding to each pixel of the VB image can be immediately calculated and form a virtual depth map (VDM).
[0031] The problem is that we have a fixed real-time RB image from an unknown location within the interior of an airway, but we also have a known VB source with known location and 3D information that enables us to create manifestations of the same hollow airway structure from arbitrary viewpoint. Given the above setup, the goal is to locate the source of the RB image by attempting to find the best match between that fixed RB image and any possible VB endoluminal rendering. A fairly straightforward approach to accomplish this is by employing a Gauss-Newton gradient descent algorithm that attempts to minimize a difference measure between the RB image and the VB image with respect to the viewing parameters (i.e., viewpoint) of the virtual image. The method for doing this is similar to the Lucas-Kanade image alignment algorithm [5].
[0032] The objective function used in [5, 6] is the sum squared difference (SSD) between the pixel intensities of the two images, although weighted SSD is equally viable, and weighted or unweighted normalized cross-correlation (CC) can be used if some additional weak assumptions are made. Using the SSD, the objective function can thus be written as
[0033]
where p is the vector of viewing parameters, Iv (u, v; p+Δp) is the virtual VB image rendered from the viewpoint p+Δp, u and v are the row and column indices, and Ir is the real RB image. Following the procedure of [5], it is shown that that the Gauss-Newton parameter update Δp can be found as
where p is the vector of viewing parameters, Iv (u, v; p+Δp) is the virtual VB image rendered from the viewpoint p+Δp, u and v are the row and column indices, and Ir is the real RB image. Following the procedure of [5], it is shown that that the Gauss-Newton parameter update Δp can be found as
[0034]
where the Hessian H is approximated per Gauss-Newton as
where the Hessian H is approximated per Gauss-Newton as
[0035]
where
where
[0036]
is a vector that gives the change in the intensity of a pixel (u, v) in a VB image Iv rendered at viewpoint p with respect to each of the components of the parameter vector Δp.
is a vector that gives the change in the intensity of a pixel (u, v) in a VB image Iv rendered at viewpoint p with respect to each of the components of the parameter vector Δp.
[0037]
can also be interpreted as a vector of steepest descent images, where each component of the vector is actually an image that describes the variation of the image intensities with respect a component of the parameter vector. Because the steepest descent images
can also be interpreted as a vector of steepest descent images, where each component of the vector is actually an image that describes the variation of the image intensities with respect a component of the parameter vector. Because the steepest descent images
[0038]
change at every viewpoint p, they, and the Hessian must be recomputed every iteration, leading to a very computationally costly algorithm.
change at every viewpoint p, they, and the Hessian must be recomputed every iteration, leading to a very computationally costly algorithm.
[0039] To speed up the iteration, the inverse compositional algorithm was proposed [6]. Under this strategy, instead of moving the virtual viewpoint towards the real viewpoint using the parameter update, we instead move the real viewpoint toward the virtual viewpoint using the inverse of the parameter update. Since the computer obviously has no control over the location of the bronchoscope tip, this may seem to be an unfeasible strategy. However, using a depth-based warping, the RB image can be warped to simulate its appearance from other viewpoints. This strategy results in comparing a warped version of the real image to a stationary virtual image. Under this formulation, the objective function we seek to minimize is:
[0040]
[0041] The warping function W (·) warps the image coordinates of the RB image Ir and hence warps the image itself. It is important also to note that the warp in this case is dependent on the depth map of the real image Zr. Solving for the Gauss-Newton parameter update associated with 4 yields
[0042]
[0043] While this may seem to add unnecessary complexity and error to the problem, it actually serves to greatly speed the iteration and has the additional side benefit of eliminating the need to render arbitrary viewpoints on the fly if you instead have a collection of pre-rendered (or pre-captured) images and corresponding depth maps. The reason for this significant increase in speed is that the VB image and VB image gradients are always evaluated at p=0, the reference viewing site, and as such allows all of the following operations to be pre-computed before iteration begins:
[0044] 1. The known environment is sampled as a set of viewing sites.
[0045] 2. Virtual images Iv are pre-rendered at each viewing site.
[0046] 3. Virtual depth maps Zv are computed at each site.
[0047] 4. Steepest descent images
[0048]
are computed with respect to each of the viewing parameters in vector p.
are computed with respect to each of the viewing parameters in vector p.
[0049] 5. The inverse Hessian H−1 is Gauss-Newton estimated from the steepest descent images
[0050]
via equation (14).
via equation (14).
[0051] The iterative portion of the algorithm may then be carried out in the following steps:
[0052] 1. Warp the real image from pose p to the nearest reference site.
[0053] 2. Compute the error image (Ir(W(u,v,Zr; p))−Iv(u,v; 0 ).
[0054] 3. Compute the parameter update Δp via equation (5).
[0055] 4. Find the new values of p by incrementing the old parameters by the inverse of the update (Δp)−1.
[0056] These steps are illustrated in FIG. 1 . Ignoring the warp function, all the equations presented thus far are general and can apply equally well to 2D transformations, such as affine or homography, or 3D rotations. The focus is now narrowed, however, to the full 3D motion case with our choice of coordinate system and parameters. One may realize from inspection of the warps in (4) that the problem is defined in terms of several local coordinate systems as each reference view is defined to be at p= 0 , yielding a different coordinate system for each viewing site used. It is, however, a trivial matter to relate each of these coordinate systems to a global coordinate frame in order to perform parameter conversions between frames. Therefore, given a camera pose with respect to the global camera frame, we can define our parameter vector as
p=[θrθpθytxtytz]T (6)
with three Euler rotation angles and three translations with respect to the nearest reference view.
p=[θrθpθytxtytz]T (6)
with three Euler rotation angles and three translations with respect to the nearest reference view.
[0057] With this parameterization, the warping W(u,v,Z; p) is governed by the matrix equation
[0058]
where R is the rotation matrix defined by the Euler angles (θr, θp, θy), u and v are the columns and rows of the image, f is the focal length, and Z is the entry on the depth map Z corresponding to the point (u,v). Here (u′,v′) gives the warped image coordinate of interest, and Z′ gives the warped depth corresponding to that point. Note that in the problem statement, we assume only that the virtual depth map Zv is known. However, when using the inverse compositional algorithm, the warp is applied to the real image Ir and the real depth map Zr must first be calculated by warping the virtual depth map Zv to the current estimated pose of the real camera via p. This can also be performed using (7) and then interpolating the resulting warped depth map onto the coordinate system of the real image. In doing so, we are implicitly assuming that our estimate of p is relatively close to its actual value. If this is not the case, the parameter error can lead to large errors in the real depth map Zr, and therefore large errors in the image warping. Under such circumstances, the forward gradient descent method governed by (1-2) may be better suited to the problem.
where R is the rotation matrix defined by the Euler angles (θr, θp, θy), u and v are the columns and rows of the image, f is the focal length, and Z is the entry on the depth map Z corresponding to the point (u,v). Here (u′,v′) gives the warped image coordinate of interest, and Z′ gives the warped depth corresponding to that point. Note that in the problem statement, we assume only that the virtual depth map Zv is known. However, when using the inverse compositional algorithm, the warp is applied to the real image Ir and the real depth map Zr must first be calculated by warping the virtual depth map Zv to the current estimated pose of the real camera via p. This can also be performed using (7) and then interpolating the resulting warped depth map onto the coordinate system of the real image. In doing so, we are implicitly assuming that our estimate of p is relatively close to its actual value. If this is not the case, the parameter error can lead to large errors in the real depth map Zr, and therefore large errors in the image warping. Under such circumstances, the forward gradient descent method governed by (1-2) may be better suited to the problem.
[0059] In order to apply the warping function, at each pixel coordinate (u,v), with intensity I(u,v) and depth Z(u,v), a new coordinate (u′,v′) and depth Z′(u′,v′) are found via (7). The original intensities and depths may then be mapped onto the new image array I(u′,v′). Some special care must be taken when performing the warping. Firstly, the image difference in (4) requires that the coordinate locations be the same for both images. The resultant array must therefore be interpolated onto the same coordinate grid as the original arrays. Because of this interpolation, and because the depth-based warping may result in occlusion, it can be difficult to choose the proper intensity corresponding to an output pixel. This can be mitigated somewhat if the intensities corresponding to larger depths are discarded when they overlap with those of smaller depths.
[0060] Finally, we turn to the calculation of the steepest-descent images
[0061]
There are several ways to generate the steepest descent images. They may be generated numerically by taking the difference of the reference images warped to small positive and negative values of each parameter. They may also be generated analytically by expanding the derivative via the chain rule:
There are several ways to generate the steepest descent images. They may be generated numerically by taking the difference of the reference images warped to small positive and negative values of each parameter. They may also be generated analytically by expanding the derivative via the chain rule:
[0062]
where ∇uI and ∇uI are the image gradients with respect to the rows and columns of the image, and Jp is the Jacobian of the warped coordinates with respect to p and thus can be found by differentiating u′ and v′ from (7) with respect to each of the warp parameters and evaluating it at a particular current value of p. In the case of the inverse compositional algorithm, the image derivatives are always evaluated at p=0 and thus the Jacobian is constant for each reference viewing site:
where ∇uI and ∇uI are the image gradients with respect to the rows and columns of the image, and Jp is the Jacobian of the warped coordinates with respect to p and thus can be found by differentiating u′ and v′ from (7) with respect to each of the warp parameters and evaluating it at a particular current value of p. In the case of the inverse compositional algorithm, the image derivatives are always evaluated at p=
[0063]
[0064] We now have all the necessary information to calculate the iterated parameter update Δp. The final step is to invert this update, and compose it with the current estimate of p. The Euler angles can be found from the rotation matrix resulting from
R′=RRdT (10)
where Rd is the incremental rotation matrix associated with the rotation angles in Δp. The updated translations can be found from
R′=RRdT (10)
where Rd is the incremental rotation matrix associated with the rotation angles in Δp. The updated translations can be found from
[0065]
where Δti are the translation elements of the parameter update Δp.
where Δti are the translation elements of the parameter update Δp.
[0066] In order to improve the performance when applying the above approach, several optimizing techniques are used. Operations performed on full-resolution images can be very computationally intensive. Therefore, a resolution pyramid is used wherein all images, depth maps, and gradients are down-sampled, preferably by a factor of 4, at each level. As we are not particularly concerned with computation time regarding the precomputed virtual views and gradients, and most video capture hardware provides real-time hardware subsampling for the real image, the computational cost of this subsampling is inconsequential and provides much quicker iteration times.
[0067] When implementing the above registration algorithm using pyramid decomposition, the algorithm is begun at the lowest resolution level of the pyramid (experimental results in this paper were performed starting at level 3; i.e., a factor of 64 reduction in resolution) and run until a reasonable stopping criteria was met before proceeding to a higher resolution level. This pyramidal approach not only speeds computation, it also serves to prevent convergence to local optima, because only the largest features are present in the highly subsampled images, while sharper features are introduced in higher resolution levels to aid in fine adjustment.
[0068] A second optimization that is used in practice is the use of the weighted normalized cross-correlation objective function
[0069]
that allows images of different mean intensities and intensity ranges to be compared and also allows weighting of individual pixel values. It should be noted that in order to use this objective function under the inverse compositional algorithm, the weights must be constant and they must be chosen prior to the computation of the steepest descent images (i.e. they must be based off features of the virtual images). Taking advantage of the equivalence of normalized SSD and normalized cross-correlation, the update can be found as:
that allows images of different mean intensities and intensity ranges to be compared and also allows weighting of individual pixel values. It should be noted that in order to use this objective function under the inverse compositional algorithm, the weights must be constant and they must be chosen prior to the computation of the steepest descent images (i.e. they must be based off features of the virtual images). Taking advantage of the equivalence of normalized SSD and normalized cross-correlation, the update can be found as:
[0070]
where the Hessian in this case is
where the Hessian in this case is
[0071]
is the set of mean-subtracted steepest descent images divided by the variance of the virtual image Iv, and Īii are the normalized images.
is the set of mean-subtracted steepest descent images divided by the variance of the virtual image Iv, and Īii are the normalized images.
Examples
[0072] To validate the algorithm, sample results for the virtual-to-real and virtual-to-virtual registration cases are given. In both of the cases outlined below, the virtual environment is a CT chest scan of a human patient designated h005. The airway surfaces were automatically generated using the methods of Kiraly et al. [10]. Airway centerlines were extracted using the methods of Swift et al. and the virtual viewing sites were chosen along these airway centerlines at intervals varying between 0.3 mm and 1 mm, with the viewing direction chosen parallel to the airway centerline [11]. Virtual images and depth maps were generated by an OpenGL renderer assuming a spot light source at the camera focal point, a field of view of 78.2 degrees and a 264×264 image size to match the calibration parameters of the bronchoscope camera.
Virtual-to-Real Registration
[0073] The virtual-to-real registration was performed using pyramid decomposition starting from level 3 and ending at level 1. To account for the difference in intensity characteristics between the imaging sources, the weighted normalized cross-correlation (12) was used as the objective function, with weights wu,v chosen as
wu,v=1−Iv(u,v) (15)
in order to emphasize dark areas, which tend to have more information in bronchoscopic video. The video frame, taken from a bronchoscopic procedure performed on h005 was first processed to remove the geometric barrel distortion from the lens to obtain the real image Ir. In the virtual-to-real registration case, it is difficult to give ground truth locations as the location of the scope tip is in practice unknown. Without external localization, the quality of a good registration is somewhat qualitative in nature.FIG. 2 shows a sample of the registration results, with edges from the virtual image overlaid on the unregistered and registered real views. The results show that the alignment is qualitatively very satisfying.
Virtual-to-Virtual Registration
wu,v=1−Iv(u,v) (15)
in order to emphasize dark areas, which tend to have more information in bronchoscopic video. The video frame, taken from a bronchoscopic procedure performed on h005 was first processed to remove the geometric barrel distortion from the lens to obtain the real image Ir. In the virtual-to-real registration case, it is difficult to give ground truth locations as the location of the scope tip is in practice unknown. Without external localization, the quality of a good registration is somewhat qualitative in nature.
Virtual-to-Virtual Registration
[0074] In the virtual-to-virtual registration case, the “real” image is actually a rendering generated at a specified location in the airway, but with all depth information discarded. The algorithm uses pyramid decomposition starting from level 3 and ending at level 1, and the weighted SSD objective function was used where the weights wu,v were chosen as in (15) as before.
[0075] FIG. 3 shows the “real” image Ir prior to registration, the virtual image Iv at the nearest reference site and the warped real image Ir (W(u,v,Z; p)) after registration is complete.
[0076]
| X | Y | |||||
|---|---|---|---|---|---|---|
| position | position | Z position | θγ | |||
| Viewpoint | (mm) | (mm) | (mm) | θα (deg) | θβ (deg) | (deg) |
| Initial | 147.5 | 149.2 | 71.1 | −20.2 | −1.7 | 0 |
| Reference | 146.7 | 149.4 | 73.3 | −7.3 | 5.1 | −19.9 |
| Site | ||||||
| Registered | 147.6 | 149.0 | 73.9 | −20.9 | 1.2 | −3.2 |
| Ground | 147.1 | 148.9 | 73.8 | −20.24 | −1.8 | −0.4 |
| Truth | ||||||
| Error | 0.6 | 0.1 | 0.5 | −0.7 | 3.0 | 2.8 |
[0077] At least four different alternatives are available for registering the real and virtual sources in the case of bronchoscopy. These scenarios are outlined below:
[0078] 1. Virtual-to-real registration: real-time or pre-recorded video images Ir from a bronchoscope at an unknown location are registered to a set of endoluminal CT renderings Iv and depth maps Zv.
[0079] 2. Virtual-to-virtual registration: an endoluminal rendering Ir with unknown location and with or without an associated depth map Zr is registered to a set of endoluminal CT renderings Iv and depth maps Zv.
[0080] 3. Real-to-real registration: real-time video images Ir from an endoscope at an unknown location is registered to a set of previously recorded video images Iv with known or estimated depth maps Zv.
[0081] 4. Real-to-virtual registration: an endoluminal rendering Ir with unknown position and with or without an associated depth map Zr is registered to a set of previously recorded video images Iv with known or estimated depth maps Zv.
[0082] The application has far-reaching applications, particularly in the field of assisted endoscopy. The registration between a CT volume and real-time bronchoscopic video allows the fusion of information between the CT realm and the bronchoscope. This allows regions of interest (ROIs) defined only in the CT volume to be superimposed on the real video frame to assist the physician in navigating to these ROIs. Likewise, airway centerlines, branch labels and metric information such as distances to walls can be displayed on the video frame.
[0083] A natural extension of this concept is to other fauns of endoscopy such as colonoscopy, where similar guidance information could be displayed on the registered colonoscopic image. Virtual-to-real registration can also be applied to pre-recorded endoscopic video, and opens the door to many post-processing options, such as mapping textural and color information available only in the endoscopic video onto the CT-derived surfaces to enable their visualization from viewpoints not available in the video alone.
[0084] An application of the real-to-real registration scenario that can be envisioned for this approach, is for aerial navigation. Satellite imagery, combined with topographic terrain information provides the known 3D environment, while real-time images from a mobile camera aboard an aircraft can be registered to this environment to give the aircraft's location and orientation without GPS or radar information. Similarly, this method also assists in ground-based robotic navigation within a known environment. Reference images and depth maps can be captured at known locations throughout the robot's working environment using a stereo camera setup, and a camera mounted on the robot can be registered to this set of images and depth maps.
References
- [0085] 1. H. Minami, Y. Ando, F. Nomura, S. Sakai, and K. Shimokata, “Interbronchoscopist variability in the diagnosis of lung cancer by flexible bronchoscopy,” Chest 105(2), pp. 1658-1662, June 1994.
- [0086] 2. I. Bricault, G. Ferretti, and P. Cinquin, “Registration of real and CT-derived virtual bronchoscopic images to assist transbronchial biopsy,” IEEE Transactions On Medical Imaging, Vol. 17, No. 5, pp. 703-714, October 1998.
- [0087] 3. J. Helferty, Image-Guided Endoscopy and its Application To Pulmonary Medicine. PhD thesis, The Pennsylvania State University, 2002.
- [0088] 4. D. Deguchi, K. Mori, J. Hasegawa, J. Toriwaki, and H. Natori et al., “Camera motion tracking of real bronchoscope using epipolar geometry analysis and CT derived bronchoscopic images,” SPIE Medical Imaging 2002: Physiol. Func. from Multidim. Images A Clough and C. Chen (ed.), v. 4683, pp. 30-41, 2002.
- [0089] 5. B. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674-679, 1981.
- [0090] 6. S. Baker and I. Matthews, “Equivalence and efficiency of image alignment algorithms,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 1, pp. 1090-1097, 2001.
- [0091] 7. H.-Y. Shum and R. Szeliski, “Panoramic Image Mosaics,” Technical Report MSR-TR-97-23, Microsoft Research.
- [0092] 8. H.-Y. Shum and R. Szeliski, “Construction of panoramic image mosaics with global and local alignment,” International Journal of Computer Vision 36(2), pp 101-130 (2000)
- [0093] 9. T. Okatani and K. Deguchi, “Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection center,” Computer Vision and Image Understanding 66, pp. 119-131, May 1997.
- [0094] 10. A. P. Kiraly, E. A. Hoffman, G. McLennan, W. E. Higgins, and J. M. Reinhardt, “3D human airway segmentation for virtual bronchoscopy,” SPIE Medical Imaging 2002: Physiology and Funct. from Multidim. Images, A. V. Clough and C. T. Chen, eds. 4683, pp. 16-29, 2002.
- [0095] 11. R. Swift, A. Kiraly, A. Sherbondy, A. L. Austin, E. A. Hoffman, G. McLennan, and W. E. Higgins, “Automatic axes-generation for virtual bronchoscopic assessment of major airway obstructions,” Computerized Medical Imaging and Graphics 26, pp. 103-118, March-April 2002.
Claims
Certificate of Correction present
The claims shown below may not include correction changes. Use the corrected PDF for authoritative text.
The invention claimed is:
1. A system for registering images associated with a continuously guided endoscopic procedure, comprising:
a memory storing a plurality of reference images of a body lumen, each reference image including a viewpoint with a depth map and image gradients;
a connector adapted for connection to an endoscope for receiving therefrom one or more live video frames associated with a body lumen; and
a processor operative to register each video frame received from the endoscope to at least one of the reference images, the processor being programmed to perform the steps of:
a) warping the video frame to the viewpoint of one reference image of the plurality of reference images or warping one reference image of the plurality of reference images at a viewpoint to the video frame;
b) automatically computing an image difference between (i) the warped video frame and the reference image, or (ii) the warped reference image and the video frame;
c) updating a viewpoint of the video frame using an optimization technique; and
d) repeating steps a) through c) for each video frame until the viewpoint of the video frame converges on a final viewpoint or a next video frame becomes available.
2. The system of claim 1, wherein at least one reference image is updated before, during, or after registration.
3. The system of claim 1, wherein the processor is further operative to receive a guess regarding an initial viewpoint of the video frame.
4. The system of claim 1, wherein the final viewpoint gives an estimate of the relative rotation and translation between the video frame of the video feed and the reference image.
5. The system of claim 1, wherein the processor is further operative to use a resolution pyramid wherein all images, depth maps, and gradients are down-sampled.
6. The system of claim 1, further including the step of using a weighted normalized cross-correlation objective function.
7. The system of claim 1, wherein the optimization technique is based on a Gauss-Newton parameter update.
8. The system of claim 1, wherein the computed set of reference images of a body lumen comprises a 3-D model of a bronchial tree.
9. The system of claim 1 further comprising an endoscope to which the connector is connected.
10. The system of claim 1 further comprising a bronchoscope to which the connector is connected.
11. A system for registering images associated with a continuously guided endoscopic procedure, comprising:
a memory storing a plurality of reference images of a body lumen, each reference image including a viewpoint with a depth map and image gradients;
an endoscope generating at least one video frame; and
a processor operative to register each video frame received from the endoscope to at least one of the reference images, the processor being programmed to perform the steps of:
a) warping the video frame to the viewpoint of one reference image of the plurality of reference images or warping one reference image of the plurality of reference images at a viewpoint to the video frame;
b) automatically computing an image difference between (i) the warped video frame and the reference image, or (ii) the warped reference image and the video frame;
c) updating a viewpoint of the video frame using an optimization technique; and
d) repeating steps a) through c) for each video frame until the viewpoint of the video frame converges on a final viewpoint or a next video frame becomes available wherein the final viewpoint gives an estimate of relative rotation and translation between the video frame and the reference image.
12. The system of claim 11, wherein at least one reference image is updated before, during, or after registration.
13. The system of claim 11, wherein the processor is further operative to receive a guess regarding an initial viewpoint of the video frame.
14. The system of claim 11, wherein the processor is further operative to use a resolution pyramid wherein all images, depth maps, and gradients are down-sampled.
15. The system of claim 11, wherein the processor is further operative to use a weighted normalized cross-correlation objective function.
16. The system of claim 11, wherein the optimization technique is based on a Gauss-Newton parameter update.
17. The system of claim 11, wherein the endoscope generates at least one live bronchoscopic video frame.
18. The system of claim 11, wherein the computed set of reference images of the body lumen comprises a 3-D model of a bronchial tree.
19. The system of claim 11, wherein the body lumen is an airway.
Patent Citations (72)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4791934(A) | 1988-12-01 | Brunnett | Applicant |
| US5740802(A) | 1998-04-01 | Nafis et al. | Applicant |
| US5748767(A) | 1998-05-01 | Raab | Applicant |
| US5765561(A) | 1998-06-01 | Chen et al. | Applicant |
| US5776050(A) | 1998-07-01 | Chen et al. | Applicant |
| US5782762(A) | 1998-07-01 | Vining | Applicant |
| US5792147(A) | 1998-08-01 | Evans et al. | Applicant |
| US5830145(A) | 1998-11-01 | Tenhoff | Applicant |
| US5891034(A) | 1999-04-01 | Bucholz | Applicant |
| US5901199(A) | 1999-05-01 | Murphy et al. | Applicant |
| US5920319(A) | 1999-07-01 | Vining et al. | Applicant |
| US5956418(A) | 1999-09-01 | Aiger et al. | Examiner |
| US5963612(A) | 1999-10-01 | Navab | Applicant |
| US5963613(A) | 1999-10-01 | Navab | Applicant |
| US5971767(A) | 1999-10-01 | Kaufman et al. | Applicant |
| US5999840(A) | 1999-12-01 | Grimson et al. | Applicant |
| US6006126(A) | 1999-12-01 | Cosman | Applicant |
| US6016439(A) | 2000-01-01 | Acker | Applicant |
| US6019725(A) | 2000-02-01 | Vesely et al. | Examiner |
| US6049582(A) | 2000-04-01 | Navab | Applicant |
| US6083162(A) | 2000-07-01 | Vining | Applicant |
| US6190395(B1) | 2001-02-01 | Williams | Applicant |
| US6201543(B1) | 2001-03-01 | O'Donnell et al. | Applicant |
| US6236743(B1) | 2001-05-01 | Pratt | Applicant |
| US6241657(B1) | 2001-06-01 | Chen et al. | Applicant |
| US6272366(B1) | 2001-08-01 | Vining | Applicant |
| US6311116(B1) | 2001-10-01 | Lee et al. | Applicant |
| US6334847(B1) | 2002-01-01 | Fenster et al. | Applicant |
| US6343936(B1) | 2002-02-01 | Kaufman et al. | Applicant |
| US6351573(B1) | 2002-02-01 | Schneider | Applicant |
| US6363163(B1) | 2002-03-01 | Xu et al. | Examiner |
| US6366800(B1) | 2002-04-01 | Vining et al. | Applicant |
| US6442417(B1) | 2002-08-01 | Shahidi et al. | Applicant |
| US6470207(B1) | 2002-10-01 | Simon et al. | Examiner |
| US6480732(B1) | 2002-11-01 | Tanaka et al. | Examiner |
| US6491702(B2) | 2002-12-01 | Heilbrun et al. | Applicant |
| US6514082(B2) | 2003-02-01 | Kaufman et al. | Applicant |
| US6535756(B1) | 2003-03-01 | Simon et al. | Examiner |
| US6546279(B1) | 2003-04-01 | Bova et al. | Applicant |
| US6556695(B1) | 2003-04-01 | Packer et al. | Examiner |
| US6593884(B1) | 2003-07-01 | Gilboa et al. | Applicant |
| US6674879(B1) | 2004-01-01 | Weisman et al. | Applicant |
| US6675032(B2) | 2004-01-01 | Chen et al. | Applicant |
| US6694163(B1) | 2004-02-01 | Vining | Applicant |
| US6714668(B1) | 2004-03-01 | Kerrien et al. | Applicant |
| US6725080(B2) | 2004-04-01 | Melkent et al. | Examiner |
| US6771262(B2) | 2004-08-01 | Krishnan | Applicant |
| US6785410(B2) | 2004-08-01 | Vining et al. | Applicant |
| US6816607(B2) | 2004-11-01 | O'Donnell et al. | Examiner |
| US6819785(B1) | 2004-11-01 | Vining et al. | Applicant |
| US6859203(B2) | 2005-02-01 | van Muiswinkel et al. | Applicant |
| US6909913(B2) | 2005-06-01 | Vining | Applicant |
| US6920347(B2) | 2005-07-01 | Simon et al. | Applicant |
| US6928314(B1) | 2005-08-01 | Johnson et al. | Examiner |
| US6947584(B1) | 2005-09-01 | Avila et al. | Applicant |
| US6980682(B1) | 2005-12-01 | Avinash et al. | Applicant |
| US7019745(B2) | 2006-03-01 | Goto et al. | Applicant |
| US7343036(B2) | 2008-03-01 | Kleen et al. | Applicant |
| US2002/0057345(A1) | 2002-05-01 | Tamaki et al. | Examiner |
| US2003/0103212(A1) | 2003-06-01 | Westphal et al. | Examiner |
| US2003/0152897(A1) | 2003-08-01 | Geiger | Applicant |
| US2004/0209234(A1) | 2004-10-01 | Geiger | Applicant |
| US2004/0252870(A1) | 2004-12-01 | Reeves et al. | Applicant |
| US2005/0074151(A1) | 2005-04-01 | Chen et al. | Examiner |
| US2005/0078858(A1) | 2005-04-01 | Yao et al. | Applicant |
| US2005/0096526(A1) | 2005-05-01 | Reinschke | Applicant |
| US2005/0182295(A1) | 2005-08-01 | Soper et al. | Examiner |
| US2005/0272999(A1) | 2005-12-01 | Guendel | Applicant |
| US2006/0084860(A1) | 2006-04-01 | Geiger et al. | Applicant |
| US2008/0262297(A1) | 2008-10-01 | Gilboa et al. | Examiner |
| KR20020041577 | 2002-06-01 | Applicant | |
| WO-2006076789 | 2006-07-01 | Applicant |
Non-Patent Literature (94)
- Deligianni, F., A. Chung, and G. Yang. “Patient-specific bronchoscope simulation with pq—space-based 2D/3D registration.” Computer Aided Surgery, vol. 9, No. 5, p. 215-226 (2004).Applicant
- Lee, P.Y. and J.B. Moore. “Pose Estimation via Gauss-Newton-on-manifold.” 16th International Symposium on Mathematical Theory of Network and System (MTNS), Leuven, 2004.Applicant
- Hamadeh, A., S. Lavallee, and P. Cinquin. “Automated 3-Dimensional Computed Tomographic and Fluoroscopic Image Registration.” Computer Aided Surgery 1998, vol, 3: p. 11-19.Applicant
- Roberson, R.E. and P.W. Likins. “A Linearization Tool for Use with Matrix Formalisms of Rotational Dynamics.” Archive of Applied Mathematics, vol. 37, No. 6: p. 388-392, Nov. 1969.Applicant
- Asano, F., Y. Matsuno, T. Matsushita, H. Kondo, Yoshio Saito, A. Seko, and Y. Ishihara. “Transbronchial Diagnosis of a Pulmonary Peripheral Small Lesion Using an Ultrathin Bronchoscope with Virtual Bronchoscopic Navigation.” Journal of Bronchology (2002), vol. 9, No. 2, p. 108-111.Applicant
- Geiger, B., G.M. Weiner, K. Schulze, J. Bilger, P. Krebs, K. Wolf, T.Albrecht. “Virtual Bronchoscopy Guidance System for Transbronchial Needle Aspiration.” Proceedings of SPIE vol. 5746 (2005).Applicant
- Grimson, W.E.L., G.J. Ettinger, S.J. White, T. Lozano-Perez, W.M. Wells III, and R. Kikinis. “An Automatic Registration Method for Frameless Stereotaxy, Image Guided Surgery, and Enhanced Reality Visualization.” IEEE Transactions on Medical Imaging, Apr. 1996, vol. 15, No. 2, p. 129-140.Applicant
- Hopper, K.D., T.A. Lucas, K. Gleeson, J.L. Stauffer, R. Bascom, D. T. Mauger, R. Mahraj. “Transbronchial Biopsy with Virtual CT Bronchoscopy and Nodal Highlighting.” Radiology Nov. 2001, vol. 221, No. 2, p. 531-536.Applicant
- Maurer, C.R., J.M. Fitzpatrick, M.Y. Wang, R.L. Galloway, Jr., R.J. Maciunas, and G.S. Allen. “Registration of Head Volume Images Using Implantable Fiducial Markers.” IEEE Transactions on Medical Imaging, Aug. 1997, vol. 16, No. 4, p. 447-462.Applicant
- McAdams, H,P., P.C. Goodman, and P. Kussin. “Virtual Bronchoscopy for Directing Transbronchial Needle Aspiration of Hilar and Mediastinal Lymph Nodes: A Pilot Study.” AJR May 1998, vol. 170, p. 1361-1364.Applicant
- Sato, Y., M. Nakamoto, Y. Tamaki, T. Sasama, I. Sakita, Y. Nakajima, M. Monden, and S. Tamura. “Image Guidance of Breast Cancer Surgery Using 3-0 Ultrasound Images and Augmented Reality Visualization.” IEEE Transactions on Medical Imaging, Oct. 1998, vol. 17, No. 5, p. 681-693.Applicant
- Schwarz, Y., A.C. Mehta, A. Ernst, F. Herth, A. Engel, D. Besser, and H. D. Becker. “Electromagnetic Navigation during Flexible Bronchoscopy.” Respiration 2003, vol. 70, p. 516-522.Applicant
- Shinagawa, N., K. Yamazaki, Y. Onodera, K. Miyasaka, E. Kikuchi, H. Dosaka-Akita, and M. Nishimura. “CT-Guided Transbronchial Biopsy Using an Ultrathin Bronchoscope with Virtual Bronchoscopic Navigation.” Chest, Mar. 2004, vol. 25, p. 1138-1143.Applicant
- Shoji, H., K. Mori, J. Sugiyama, Y. Suenaga, J. Toriwaki, H. Takabatake, and H. Natori. “Camera motion tracking of real endoscope by using virtual endoscopy system and texture information.” Proceedings of SPIE vol. 4321, p. 122-133 (2001).Applicant
- Stefansic, J.D., A.J. Herline, Y. Shyr, W.C. Chapman, J.M. Fitzpatrick, B.M. Dawant, and R.L. Galloway, Jr. “Registration of Physical Space to Laparoscopic Image Space for Use in Minimally Invasive Hepatic Surgery.” IEEE Transactions on Medical Imaging, Oct. 2000, vol. 19, No. 10, p. 1012-1023.Applicant
- Turcza, P. and M. Duplaga. “Navigation Systems Based on Registration of Endoscopic and CT-derived Virtual Images for Bronchofiberoscopic Procedures.” Studies in Health Technology and Informatics, vol. 105, p. 253-263 (2004).Applicant
- White, C.S., E.A. Weiner, P. Patel, and E.J. Britt. “Transbronchial Needle Aspiration: Guidance with CT Fluoroscopy.” Chest 2000, vol. 118, No. 6, p. 1630-1638.Applicant
- Higgins, W., W.J.T. Spyra, R.A. Karwoski, and E.L. Ritman. “System for Analyzing High-Resolution Three-Dimensional Coronary Angiograms.” IEEE Transactions on Medical Imaging, Jun. 1996, vol. 15, No. 3, p. 377-385.Applicant
- Brady, M.L., W.E. Higgins, K. Ramaswamy. “Interactive navigation inside 3D radiological images.” IEEE 1995, p. 33-40.Applicant
- Higgins, W.E. and K. Ramaswamy. “Toward dynamic visualization for endoscopy simulation.” IEEE 1994, p. 700-701.Applicant
- Bricault, I., G. Ferretti, and P. Cinquin. “Registration of Real and CT-Derived Virtual Bronchoscopic Images to Assist Transbronchial Biopsy.” IEEE Transactions on Medical Imaging, Oct. 1998, vol. 17, No. 5, p. 703-714.Applicant
- Sherbondy, A.J., A.P. Kiraly, A.L. Austin, J.P. Helferty, S. Wan, J.Z. Turlington, T. Yang, C. Zhang, E.A. Hoffman, and G. McLennan. “Virtual Bronchoscopic approach for combining 3D CT and Endoscopic Video.” Proceedings of SPIE 2000, vol. 3978, No. 104.Applicant
- Helferty, J.P., A.J. Sherbondy, A.P. Kiraly, J.Z. Turlington, E.A. Hoffman, G. McLennan, W.E. Higgins. “Experiments in virtual-endoscopic guidance of bronchoscopy.” Proceedings of SPIE 2001, vol. 4321, No. 111.Applicant
- Helferty, J.P. and W.E. Higgins. “Combined endscopic video tracking and virtual 3D CT registration for surgical guidance.” Proceedings of the 2002 International Conference on Image Processing, vol. 2, pp. 961-964.Applicant
- Higgins, W.E., J.P. Helferty, and D.R. Padfield. “Integrated bronchoscopic video tracking and 3D CT registration for virtual bronchoscopy.” Proceedings of SPIE 2003, vol. 5031, No. 80.Applicant
- Kiraly, A.P., J.P. Helferty, E.A. Hoffman, G. McLennan, W.E. Higgins. “Three-dimensional path planning for virtual bronchoscopy.” IEEE Transactions on Medical Imaging 2004, vol. 23, No. 11, pp. 1365-1379.Applicant
- Helferty, J.P., E.A. Hoffman, G. McLennan, W.E. Higgins. “CT-video registration accuracy for virtual guidance of bronchoscopy.” Proceedings of SPIE 2004, vol. 5369, pp. 150-164.Applicant
- Higgins, W.E., L. Rai, S.A. Merritt, K. Lu, N. T. Linger, and K.C. Yu. “3D image fusion and guidance for computer-assisted bronchoscopy.” Proceedings of SPIE 2005, vol. 6016.Applicant
- Asano, F., Y Matsuno, N. Shinagawa, K. Yamazaki, T. Suzuki, T. lshida, and H. Moriya. “A Virtual Bronchoscopic Navigation System for Pulmonary Peripheral Lesions.” Chest 2006, vol. 130, No. 2, pp. 559-566.Applicant
- Kukuk, M. “Modeling the Internal and External Constraints of a Flexible Endoscope for Calculating its Workspace: Application in Transbronchial Needle Aspiration Guidance.” Proceedings of SPIE 2002, vol. 4681, pp. 539-550.Applicant
- Gibbs, J.D. and W.E. Higgins. “3D Path Planning and Extension for Endoscopic Guidance.” Proceedings of SPIE 2007, vol. 6509.Applicant
- Mori, K., S. Ema, T. Kitasaka, Y. Mekada, I. Ide, H. Murase, Y. Suenaga, H. Takabatake, M. Mori, and H. Natori “Automated Nomenclature of Bronchial Branches Extracted from CT Images and Its Application to Biopsy Path Planning in Virtual Bronchoscopy.” Medical Image Computing and Computer-Assisted Intervention 2005, Lecture Notes in Computer Science 3750, pp. 854-861.Applicant
- Kiraly, et al., 3D human airway segmentation for virtual bronchoscopy, Proceedings of SPIE, 4683: 16-29, 2002.Applicant
- Swift, et al., Automatic axis generation for virtual bronchoscopic assessment of major airway obstructions, Computerized Medical Imaging and Graphics, 26: 103-118, 2002.Applicant
- Ansar, et al., Linear pose estimation from points or lines, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(5):578:589, 2003.Applicant
- Hager, et al., Fast and globally convergent pose estimation from video images, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(6):610-622, 2000.Applicant
- Helferty, J., Image-guided endoscopy and its application to pulmonary medicine, A Thesis in Electrical Engineering, The Pennsylvania State University, 2002.Applicant
- Deligianni, F., a. Chung, and G. Yang. “Patient-specific bronchoscope simulation with pq-space-based 2D/3D registration.” Computer Aided Surgery, vol. 9, No. 5, p. 215-226 (2004).Applicant
- Lee, P.Y. And J.B. Moore. “Pose Estimation via Gauss-Newton-on-manifold.” 16th International Symposium on Mathematical Theory of Network and System (MTNS), Leuven, 2004.Applicant
- Hamadeh, A., S. Lavallee, and P. Cinquin. “Automated 3-Dimensional Computed Tomographic and Fluoroscopic Image Registration.” Computer Aided Surgery 1998, vol, 3: p. 11-19.Applicant
- Roberson, R.E. and P.W. Likins. “A Linearization Tool for Use with Matrix Formalisms of Rotational Dynamics.” Archive of Applied Mathematics, vol. 37, No. 6: p. 368-392, Nov. 1969.Applicant
- Asano, F., Y. Matsuno, T. Matsushita, H. Kondo, Yoshio Saito, A. Seko, and Y. Ishihara. “Transbronchial Diagnosis of a Pulmonary Peripheral Small Lesion Using an Ultrathin Bronchoscope with Virtual Bronchoscopic Navigation.” Journal of Bronchology (2002), vol. 9, No. 2, p. 108-111.Applicant
- Geiger, B., G.M. Weiner, K. Schulze, J. Bilger, P. Krebs, K. Wolf, T.Albrecht. “Virtual Bronchoscopy Guidance System for Transbronchial Needle Aspiration.” Proceedings of SPIE vol. 5746 (2005).Applicant
- Grimson, W.E.L., G.J. Ettinger, S.J. White, T. Lozano-Perez, W.M. Wells III, and R. Kikinis. “An Automatic Registration Method for Frameless Stereotaxy, Image Guided Surgery, and Enhanced Reality Visualization.” IEEE Transactions on Medical Imaging, Apr. 1996, vol. 15, No. 2, p. 129-140.Applicant
- Helferty, J.P., A.J. Sherbondy, A.P. Kiraly, and W.E. Higgins. “Computer-based System for the Virtual Endoscopic Guidance of Bronchoscopy.” (believed to have been offered for sale, publicly used, and/or published prior to the filed of this application).Applicant
- Hopper, K.D., T.A. Lucas, K. Gleeson, J.L. Stauffer, R. Bascom, D. T. Mauger, R. Mahraj. “Transbronchial Biopsy with Virtual CT Bronchoscopy and Nodal Highlighting.” Radiology Nov. 2001, vol. 221, No. 2, p. 531-536.Applicant
- Maurer, C.R., J.M. Fitzpatrick, M.Y. Wang, R.L. Galloway, Jr., R.J. Maciunas, and G.S. Allen. “Registration of Head Volume Images Using Implantable Fiducial Markers.” IEEE Transactions on Medical Imaging, Aug. 1997, vol. 16, No. 4, p. 447-462.Applicant
- McAdams, H,P., P.C. Goodman, and P. Kussin. “Virtual Bronchoscopy for Directing Transbronchial Needle Aspiration of Hilar and Mediastinal Lymph Nodes: A Pilot Study.” AJR May 1998, vol. 170, pp. 1361-1364.Applicant
- Merritt, S.A., L Rai, and W.E. Higgins. “Real-Time CT-Video Registration for Continuous Endoscopic Guidance.” (believed to have been offered for sale, publicly used, and/or published prior to the filed of this application).Applicant
- Mori, K., T. Enjoji, D. Deguchi, T. Kitasaka, Y. Suenaga, J. Toriwaki, H. Takabatake, and H. Natori. “New image similarity measures for bronchoscope tracking based on image registration between virtual and real bronchoscopic images.” (believed to have been offered for sale, publicly used, and/or published prior to the filed of this application).Applicant
- Rai, L., S.A. Merritt, and W.E. Higgins. “Real-time Image-based Guidance Method for LungCancer Assessment.” (believed to have been offered for sale, publicly used, and/or published prior to the filed of this application).Applicant
- Sato, Y., M. Nakamoto, Y. Tamaki, T. Sasama, I. Sakita, Y. Nakajima, M. Monden, and S. Tamura. “Image Guidance of Breast Cancer Surgery Using 3-0 Ultrasound Images and Augmented Reality Visualization.” IEEE Transactions on Medical Imaging, Oct. 1998, vol. 17, No. 5, pp. 681-693.Applicant
- Schwarz, Y., A.C. Mehta, A. Ernst, F. Herth, A. Engel, D. Besser, and H. D. Becker. “Electromagnetic Navigation during Flexible Bronchoscopy.” Respiration 2003, vol. 70, pp. 516-522.Applicant
- Shinagawa, N., K. Yamazaki, Y. Onodera, K. Miyasaka, E. Kilkuchi, H. Dosaka-Akita, and M. Nishimura. “CT-Guided Transbronchial Biopsy Using an Ultrathin Bronchoscope with Virtual Bronchoscopic Navigation.” Chest, Mar. 2004, vol. 25, pp. 1138-1143.Applicant
- Shoji, H., K. Mori, J. Sugiyama, Y. Suenaga, J. Toriwaki, H. Takabatake, and H. Natori. “Camera motion tracking of real endoscope by using virtual endoscopy system and texture information.” Proceedings of Spie vol. 4321, pp. 122-133 (2001).Applicant
- Stefansic, J.D., A.J. Herline, Y. Shyr, W.C. Chapman, J.M. Fitzpatrick, B.M. Dawant, and R.L. Galloway, Jr. “Registration of Physical Space to Laparoscopic Image Space for Use in Minimally Invasive Hepatic Surgery.” IEEE Transactions on Medical Imaging, Oct. 2000, vol. 19, No. 10, pp. 1012-1023.Applicant
- Turcza, P. and M. Duplaga. “Navigation Systems Based on Registration of Endoscopic and Ct-derived Virtual Images for Bronchofiberoscopic Procedures.” Studies in Health Technology and Informatics, vol. 105, pp. 253-263 (2004).Applicant
- Helferty, J.P., E.A. Hoffman, G. McLennan, W.E. Higgins. “CT-video registration accuracy for virtual guidance of bronchoscopy.” Proceedings of SPIE 2004, vol. 5369, pp. 150-164.Applicant
- Higgins, W.E., L. Rai, S.A. Merritt, K. Lu, N. T. Linger, and K.C. Yu. “3D image fusion and for computer-assisted bronchoscopy.” Proceedings of SPIE 2005, vol. 6016.Applicant
- Asano, F., Y Matsuno, N. Shinagawa, K. Yamazaki, T. Suzuki, T. lshida, and H. Moriya. “A Virtual Bronchoscopic Navigation System for Pulmonary Peripheral Lesions.” Chest 2006, vol. 130, No. 2, pp. 559-566.Applicant
- Kukuk, M. “Modeling the Internal and External Constraints of a Flexible Endoscope for Calculating its Workspace: Application in Transbronchial Needle Aspiration Guidance.” Proceedings of SPIE 2002, vol. 4681, pp. 539-550.Applicant
- Gibbs, J.D. And W.E. Higgins. “3D Path Planning and Extension for Endoscopic Guidance.” Proceedings of SPIE 2007, vol. 6509.Applicant
- Mori, K., S. Erna, T. Kitasaka, Y. Mekada, I. Ide, H. Murase, Y. Suenaga, H. Takabatake, M. Mori, and H. Natoli, “Automated Nomenclature of Bronchial Branches Extracted from CT Images and Its Application to Biopsy Path Planning in Virtual Bronchoscopy.” Medical Image Computing and Computer-Assisted Intervention 2005, Lecture Notes in Computer Science 3750, pp. 854-861.Applicant
- Kiraly, et al., 3D human airway segmentation for virtual bronchoscopy, Proceedings of SPIE, 4683: 16-29, 2002.Applicant
- Lucas, et al., An iterative image registration technique with an application to stereo vision, Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI '81), Apr. 1981, pp. 674-679. cited by other.Applicant
- Minami, et al., Interbronchoscopist variability in the diagnosis of lung cancer by flexible bronchoscopy, Chest, 105:1658-1662, 1994. cited by other.Applicant
- Okatani, et al., Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection center, Computer Vision and Image Understanding, 66:119-131, 1997. cited by other.Applicant
- Shum, et al., Systems and experiment paper: construction of panoramic image mosaics with global and local alignment, International Journal of Computer Vision, 36(2): 101-130, 2000.Applicant
- Shum, et al, Panoramic image mosaics, Microsoft Research, Technical Report MSR-TR-97- 23, 1997.Applicant
- Swift, et al., Automatic axis generation for virtual bronchoscopic assessment of major airway obstructions, Computerized Medical Imaging and Graphics, 26: 103-118, 2002.Applicant
- Asano, et al., Transbronchial diagnosis of a pulmonary peripheral small lesion using an ultrthin bronchoschope with virtual bronchoscopic navigation, Journal of Bronchology, 9: 108-11, 2002.Applicant
- Helferty, et al., Technique for registering 3D virtual CT images to endoscopic video, IEEE Image Processing, 893-896, 2001.Applicant
- Baker, et al., Equivalence and efficiency of image alignment algorithms, Proceedings of the 2001 Conference on Computer Vision and Pattern Recognition, 1090-1097.Applicant
- Bricault, et al., Registration of real and CT-derived virtual bronchoscopic images to assist transbronchial biopsy, IEEE Transactions of Medical Imaging, 17(5):703-714, 1998.Applicant
- Deguchi, et al., Camera motion tracking of real bronchoscope using epipolar geometry analysis and Ct derived bronchoscopic images, Proceedings of SPIE, 4683: 30-41, 2002.Applicant
- Haralick, et al., Analysis and solutions of the three point perspective pose estimation problem Proceedings Computer vision and pattern recognition, IEEE Computer Society Conference, 592-598, 1991.Applicant
- Harris, et al., A combined corner and edge Conference, 147-151, 1988.Applicant
- Horaud, et al., A analytic solution for the perspective 4-point problem, Computer Vision, Graphics, and Image Processing, 47:33-44, 1989.Applicant
- Fischler, et al, Random sample consensus: a paradigm for model fitting with application to image analysis and automated cartography, Communications of the ACM, 24(6): 381-395, 1981.Applicant
- Zuliani, et al., A mathematical comparison of point detectors, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.Applicant
- Ansar, et al., Linear pose estimation from points or lines, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(5):578:589, 2003.Applicant
- Mori, et al., A method for tracking the camera motion of real endoscope by epipolar geometry analysis and virtual endoscopy system, Lecture Notes in Computer Science, 2208: 1-8, 2001.Applicant
- Shi, et al., Good features to track, IEEE Conference on Computer Vision and Pattern Recognition, 593-600, 1994.Applicant
- Triggs, B., Detecting keypoints with stable position, orientation, and scale under illumination changes, Lecture Notes on Computer Science, 3024: 100-113, 2004.Applicant
- Quan, et al., Linear N-point camera pose determination, IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8): 774-780, 1999.Applicant
- Haralick, et al., Pose estimation from corresponding point data, IEEE Transaction on Systems, Man, and Cybernetics, 19(6): 1426-1446, 1989.Applicant
- Lowe, D., Fitting parameterized three-dimensional models to images, IEEE Transaction on Pattern Analysis and Machine intelligence, 13(5):441-450, 1991.Applicant
- Hager, et al., Fast and globally convergent pose estimation from video images, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(6):610-622, 2000.Applicant
- Hartley, at al,, Multiple View Geometry in Computer Vision, Cambridge University Press, ISBN:0521623049, pp. 238, 239, 281, 2000.Applicant
- Higgins, at al., Integrated bronchoscopic video tracking and 3D CT registration for virtual bronchoscopy, Medical Imaging 2003: Physiology and Function: Methods, Systems, and Applications.Applicant
- Mori, et al., New image similarity measures for bronchoscope tracking based on image registration between virtual and real bronchoscopic images, Medical Imaging 2004: Physiology, Function, and Structure from Medical Images, 165-176.Applicant
- Shoji, at al., Camera motion tracking of real endoscope by using virtual endoscopy system and texture information, Medical Imaging 2001: Physiology and function from Multidimensional Images.Applicant
- Helferty, et al., Combined endoscopic video tracking and virtual 3D CT registration for surgical guidance, IEEE International Conference on Image Processing, 2002.Applicant
- Helferty, J., Image-guided endoscopy and its application to pulmonary medicine, A Thesis in Electrical Engineering, The Pennsylvania State University, 2002.Applicant