Remote control of medical devices using a virtual device interface

[0001] This application claims priority of prior provisional application Ser. No. 60/401,670, filed Aug. 6, 2002, entitled Method and Apparatus for Improved Magnetic Surgery Employing Virtual Device Interface, the disclosure of which is incorporated by reference in its entirety.

[0002] This invention relates to the remote control of medical devices in a subject's body, and in particular to a user interface for operating a remotely controllable medical device which employs a “virtual device” interface.

[0003] Advances in technology have resulted in systems that allow a physician or other medical professional to remotely control the orientation of the distal end of a medical device. It is now fairly routine to steer the distal end of a medical device inside a subject's body by mechanically manipulating controls on the proximal end of the medical device. Recently magnetic navigation systems have been developed that allow a physician to orient the distal end of a medical device using the field of an external source magnet. Other systems have been developed for the automated remote orientation of the distal end of a medical device, for example by operating magnetostrictive or electrostrictive elements incorporated into the medical device. However the medical device is controlled, it can still be difficult for a physician to visualize the procedure site (which is out of view inside the patient's body), to selected the desired direction in which to orient the distal end of the medical device and communicate the selected direction to the system in order to orient the distal end of the medical device in the selected direction.

[0004] As stated above, magnetic navigation systems have been developed which apply a controlled magnetic field to an operating region in a subject, to orient a magnetically responsive element on a medical device in the operating region. Examples of such systems include Ritter et al., U.S. Pat. No. 6,241,671, issued Jun. 5, 2001, for Open Field System For Magnetic Surgery (incorporated herein by reference). Magnetic navigation systems permit faster and easier navigation, and allow the devices to be made thinner and more flexible than conventional mechanically navigated devices which must contain pull wires and other components for steering the device. Because of the advances made in magnetic surgery systems and magnetically responsive medical devices, the determination of the appropriate field direction, and instructing the magnetic surgery system to apply the determined magnetic field are probably the most difficult tasks remaining in magnetically assisted medical procedures. Significant efforts have been made to help the user to visualize the procedure, and improve the user's ability to control the magnetic surgery system during the procedure. There is often a lag between the direction of the applied field, and the actual direction of the distal end of the medical device. In some current systems, the user specifies a field direction, and mentally must take into account the lag between the applied field and the actual device direction.

[0005] This invention provides a method and apparatus for controlling a flexible medical device in a subject's body which employs a virtual device interface, i.e. an interface using a physical or computational model of the actual device, possibly including a computerized control interface for real-time/interactive navigational control of the device. Such a computerized device control interface would accept user input from an input device (for example, a joystick) and interpret these inputs through a computer to apply appropriate controls to drive the device tip according to a pre-defined mapping. Generally the method of this invention comprises displaying an image of the distal end portion of a “virtual” medical device, and allowing the user to manipulate the displayed image of the distal end portion of the virtual medical device into a desired configuration (shape and/or orientation), and then remotely operate the device to cause the distal end portion of the actual medical device to assume the desired configuration represented by the image of the virtual device.

[0006] In a preferred embodiment of the system and method of this invention, the configuration of the displayed virtual medical device can be controlled by identifying a target point to which the virtual medical device configures itself to. In another preferred embodiment, controls that change the shape of virtual medical device, e.g., a deflection control and a rotation control are used to control configuration. In still another preferred embodiment of the system and method of this invention, the configuration of the displayed virtual medical device can be changed by changing at least one control parameter (e.g., the applied magnetic field for a magnetically controlled device) and updating the image of the virtual medical device to show the configuration of the distal end of the virtual medical device “as if” the at least one control parameter was changed (in the particular case of a magnetically controlled device, as if the new field were applied). In another preferred embodiment, the user can identify a target location and the interface uses the virtual device model to determine the control parameter(s) to cause the actual device to reach the target. In still another alternate preferred embodiment of the system and method of this invention, a surface of points of the possible positions of the distal end of the medical device is displayed. The user can then select a point on the surface, and operate the system to automatically apply the correct magnetic field to cause the distal end of the medical device to align in the direction of the selected point.

[0007] A preferred embodiment of this invention provides an interface system and method for facilitating the specification and application of a magnetic field to the operating region in a subject to control the distal end of a medical device in the operating region. This invention can provide a method and apparatus for controlling a magnetic navigation system, and in particular provides an interface system and method for facilitating the specification and application of a magnetic field to the operating region in a patient to control the distal end of a magnetically enabled medical device in the operating region. However, the virtual device interface of the present invention is not so limited, and can be used with any system for controlling the configuration of a medical device and actuated by any of a variety of means. Furthermore, the interface can be used with any elongate medical device, including guide wires, catheters, and endoscopes, etc.

[0008] The system and method of this invention allows the user to visualize the configuration of the distal end portion of the medical device before actually applying the control variable(s) used to modify the configuration of the device. This is faster and easier to learn and to operate than many current user interfaces; for instance some present magnetic navigation interfaces require the user to specify a magnetic field direction without reference to the current orientation of the distal end portion of the medical device. This also facilitates automating the procedure combining directional control with an advancer for advancing the medical device. Furthermore the present invention lets the user directly manipulate the distal end of the device in an interactive manner through the use of an input device whose manipulation by the user is mapped by a computer to changes in actuation control variables which then produce a corresponding change in device configuration. The change in device configuration may be visually displayed to the user through the use of any of a variety of imaging systems so that the user has interactive control over the device. The input device for this purpose may for example be a joystick, graphical menu buttons, keyboard buttons, or any other choice that is known to those skilled in the art. The mapping of input device manipulations to changes in device actuation controls may be chosen in a variety of ways that provide intuitive spatial control of the device. A selection of possible mapping choices can be offered to the user so that the user can pick the one that she or he deems most appropriate in a given navigational circumstance. These and other features and advantages will be in part apparent, and in part pointed out hereinafter.

[0030] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.

[0031] The virtual device interface of the present invention can be used with any type of remotely controllable medical device, including for example mechanically, electrically, and magnetically actuatable medical devices. One possible use of the invention is in the control of magnetically actuatable devices, such as with the magnetic navigation system shown in FIG. 1. While described below primarily in connection with a preferred embodiment in the form of a magnetic surgery system in conjunction with an X-ray imaging system, this invention is not so limited. For example, a Magnetic Resonance imaging system could be used, with the catheter being steered by torques generated by the use of changeable magnetic moments located on the device that interact with the static magnetic field of the Magnetic Resonance imaging system. Likewise, any other imaging modality or actuation modality could be employed.

[0032] As shown in FIG. 1, a magnetic surgery system is set up in the procedure room 50 where the patient is located, and in a control room 52. The control room 52 is preferably adjacent the procedure room 50, and there may be a window 54 between the control room and the procedure room to permit direct observation of the patient, however the control room could be remote from the patient, and with the aid of the present interface, a physician could conduct a procedure on a patient in the procedure from a control room on a different floor, in a different building, or even in a different city.

[0033] The magnetic surgery system comprises a patient bed 56, and a magnetic navigation system 58 comprising opposed magnet units 60 and 62 on opposite sides of the patient bed operated by a processor 64 and controlled by controls 66 adjacent the patient 56. An imaging system 68, such as an x-ray imaging system on a C-arm, displays images of the operating region on a set of monitors 70 in the procedure room 50. The interface system of the present invention provides a convenient way for a user to operate the magnetic navigation system 58 to control the distal end of a medical device in the operating region inside the patient's body.

[0034] The interface includes a display on, for example, an lcd monitor 72, and a digital tablet 74 in the procedure room 50, a processor 76, a display on, for example, monitor 78, a key board 80, and a mouse/digital tablet 82 in the control room 54. Additional displays on monitors 86 and 88 can be provided in the procedure room 50 which integrate images from the imaging system 68 with the interface. One or more additional monitors 90 can be provided in the control room so that the images are available in the control room as well. The monitor 90 preferably displays a multi-pane display.

[0035] In one preferred embodiment of this invention as shown in FIG. 2, one of the displays, for example display 90 has panes 92 and 94 for receiving biplane images of the operating region. The display 90 preferably also includes a control pane 96. As shown in FIG. 2, the user selects a discrete set of points on the distal end of the medical device to characterize the device. In the preferred embodiment, the operating region is imaged with a bi-planar imaging (such as bi-plane fluoroscopy), providing two images of the operating region and of the distal end of the medical device, in different, and preferably mutually perpendicular, planes. For example, the bi-plane imaging system might provide left anterior oblique (LAO) and right anterior oblique (RAO) images of the operating region, in panes 92 and 94, respectively. Bi-plane imaging could be provided with a single x-ray source and imaging plate that are moved in tandem to provide imaging in multiple planes.

[0036] The user preferably identifies the “support” point or “pivot” point x₁ of the distal end portion of the medical device on each of the panes 92 and 94. By identifying the point in both planes, the user has uniquely identified the point in three dimensional space. The user preferably also identifies the distal end x₄ of the medical device. The user preferably also identifies at least one, and preferably at least two other points x₂ and x₃, between points x₁ and x₄. The user could of course identify more points, but the additional accuracy achieved usually does not outweigh the inconvenience to the user.

[0037] Other methods of reconstructing the shape of the distal end of the medical device are possible. For example, points could be identified independently on each of the views. Based upon the these points the processor could develop splines (a sequence of polynomial curves) for each plane, and then identify corresponding points on the two splines to characterize the shape of the medical device in three dimensions.

[0038] As shown in FIG. 4A, the points can be identified by manipulating a cursor on each of the bi-planar displays, and clicking on a point, such as point 100. As shown in FIG. 4B, after the user clicks on point 100 on the LAO, a line 102 is displayed on the RAO, along which the point 100 identified in the LAO must lie. As shown in FIG. 4C, the user then uniquely identifies the point in three dimensional space by identifying the desired point 104 along the line 102 displayed on the RAO. The user could begin on the RAO, and complete the point identification process in the LAO. Some other method can be used to identify points, or the points can be identified automatically, for example using image processing.

[0039] Of course, instead of using bi-plane images to uniquely identify points on the medical device to characterize the shape of the medical device, a localization system could be employed to characterize the shape of the medical device. For example a magnetic localization system employing reference transmitters in the procedure room can transmit to one or more receivers on the medical device to locate the receiver and thus the medical device in three-dimensional space (or one or more transmitters on the medical device can transmit to reference receivers in the procedure room). Other localization systems, for example using ultrasound, or electric potential, could be used.

[0040] Once the points x₁, x₂, x₃, and x₄ are identified in three-dimensional space, the points can be processed to determine the configuration of the distal end of the medical device. This processing can determine whether the point x₁ is in fact the support point and pivot point of the medical device. For example in a uniform medical device, such as a uniform wall catheter, the distal end portion of the medical device, i.e. the portion distal to the pivot point will assume a generally circular shape. Thus, whether a selected point x₁ is in fact the support point or pivot point can be determined by checking the circularity of the reconstructed curve between the points x₁, x₂, x₃, and x₄. With other catheter configurations in which the properties are not uniform along the length, the validity of x₁ as the support point or pivot point can be determined by checking the shape of the distal end portion against a calculated or empirically determined shape.

[0041] In this preferred embodiment, if the point x₁ selected as the tentative pivot point or support point, is determined not to be a valid pivot point or support point, the fact is signaled to the user, for example with a text message or by changing the color of the point on the display to signal the user to select another tentative pivot point or support point. Through appropriate processing the system can even suggest one or more appropriate pivot or support points for the user to simply accept. Once the pivot or support point is correctly determined, the processor can then determine the free length l of the medical device that is distal to the pivot or support point.

[0042] With the pivot or support point x₁, the free length l, and the properties of the distal end portion, as indicated by the positions of the points x₁, x₂, x₃, and x₄ identified by the user when a known magnetic field was applied, it is possible to calculate the configuration of the distal end portion of the medical device when a different magnetic field is applied. These calculations can take into account the properties of the medical device as represented in a lookup table or in one or more equations, developed by mathematical modeling or experimental measurements. These calculations can also take into account previous sets of points x₁, x₂, x₃, and x₄ identified by the user for the same (or even similar) medical devices.

[0043] Thus, it is possible to display a representation 106 of a virtual medical device representing the configuration (shape and orientation) of the actual device if a different magnetic field were applied. (Of course, for other remotely controllable medical devices, the interface would display a representation of a virtual medical device representing the configuration of the actual device as if the control parameter(s) were changed as selected by the user). This representation can be superimposed over the images of the operating region in panes 92 and 94. Thus, the user can select a desired new magnetic field, and then see the configuration of the distal end of the medical device as if the new field were applied, before the new field is applied, and make appropriate adjustments. A variety of systems and methods have been devised for identifying the desired direction of the applied magnetic field. For example, the user can identify the starting point and the ending point (for example on displays of a bi-plane imaging system), and the field can be applied in the selected direction. Alternatively, the user could manipulate a vector representation of the desired field direction, or select previously used directions, or select directions associated with previously identified points (with other remotely controllable medical devices the user can manipulate appropriate control parameters to change the configuration of the device). However, in each case the resulting configuration of the medical device was merely a conjecture of the user, based upon experience. With the present method a display of the configuration that the distal end of the medical device should assume when the desired field is applied can be made before the field is actually applied.

[0044] Furthermore, as shown in FIG. 3A, it is possible to generate a surface 108 representing all the possible points that the distal end of the medical device can reach with only changes to one or more selected control parameters, such as the magnetic field direction and/or intensity. In this preferred embodiment, this surface 108 has a generally parabolic shape. This surface 108 can also be displayed, and the user can identify a point on this surface, and through processing the system can determine the correct field to apply to cause the distal end of the medical device to reach the selected point. The user can orient the device in the desired direction and reach a point slightly beyond the surface by advancing the medical device slightly. As shown in FIG. 3B, the system can even generate and display surfaces 108′ and 108″ of possible points for different free lengths l′ and l″, so that if the surface representing the current set of possible points does not reach the desired location, the user can select from a surface of a different set of possible points, and either manually adjust the free length l, or if the system is equipped with a mechanized advancer mechanism, allow the system to automatically adjust the free length l.

[0045] The system and method of this invention also allow the user to change the configuration of the displayed representation 106 of the medical device, and apply the field to achieve the desired configuration. Thus the display can be coupled with controls that change the configuration of the displayed virtual image 106 of the distal end portion of the medical device. These controls, can be for example, controls that change the deflection and rotation of the distal end of the device. As shown in FIG. 5, these controls may be implemented by screen controls 110 and 112, which can be operated by pointing, clicking and dragging the control to change the degree of deflection and to change the degree of rotation. These controls 110 and 112 are similar to the controls provided on conventional mechanically navigable devices. In fact, rather than being implemented on the screen, the controls could be provided in a catheter handle, similar to conventional catheter handles, which the user can manipulate to change the deflection and rotation of the distal end of the medical device. Once a desired deflection and rotation are achieved using controls 110 and 112, as indicated by the display of the virtual device in panes 92 and 94, the user can apply the appropriate field, for example by pointing and clicking on a virtual button 114 (or other control), to apply the magnetic field. Alternatively, the system can operate in a continuous mode, in which the configuration of the distal end portion of the medical automatically changes upon a change in the displayed image 106, without the need to operate button 114). The possible configurations of the displayed virtual device are preferably limited to configurations which can be achieved with possible applied magnetic fields, so that user cannot manipulate the virtual device into a configuration that cannot be achieved. This could be built into the controls as well, so that the controls will not allow the user to manipulate the image 106 of the virtual device into a configuration that cannot be physically achieved.

[0046] Just as the processor can determine the configuration for a given applied field, the processor can determine the appropriate applied field for a given configuration. Thus the user can specify a configuration, or more preferably a target point, and the interface can determine the control parameters (e.g., magnetic field direction, magnetic field strength, and free length) to reach the target. The interface displays the hypothetical new configuration and if satisfactory, the user can accept it, and the interface-determined control parameter applied to reach the target.

[0047] The system can store the properties of various types of medical devices, and one of these stored values could be used in the mathematical model that determines the shape of the display 106 of the virtual medical device. The user can select some of these stored values, and as shown in FIG. 5, the pane 96 can also include a window 116 to display indicia identifying the stored values being used. The pane can also include displays 118 of the magnetic field direction currently being applied.

[0048] The interface employs some model of the medical device. The following description illustrates a model in the simple case of a flexible device with uniform elastic properties and with a single permanent magnet located at the device tip. The model and details generalize in a manner that may be determined by those skilled in the art to the case when non-uniform elastic properties and more than one magnet are employed, or when alternative modes of device actuation are used. Mathematically, the configuration of the medical device can be represented as follows:

The distances between the points x₁, x₂, x₃, and x₄, are given as follows:

[0049] $\begin{array}{cc}{d}_1={\overrightarrow{x}}_2-{\overrightarrow{x}}_1& \left[1\right]\\ d_2={\overrightarrow{x}}_3-{\overrightarrow{x}}_2& \left[2\right]\\ d_3={\overrightarrow{x}}_4-{\overrightarrow{x}}_3& \left[3\right]\end{array}$
The unit tangent vectors at points x₁ and x₄ ({right arrow over (n)}₁ and {right arrow over (n)}₄) may be estimated as follows:

[0050] $\begin{array}{cc}{\overrightarrow{t}}_1=\frac{\left({\overrightarrow{x}}_2-{\overrightarrow{x}}_1\right)}{d_1}& \left[4\right]\\ {\overrightarrow{t}}_2=\frac{\left({\overrightarrow{x}}_3-{\overrightarrow{x}}_2\right)}{d_2}& \left[5\right]\\ {\overrightarrow{t}}_3=\frac{\left({\overrightarrow{x}}_4-{\overrightarrow{x}}_3\right)}{d_3}& \left[6\right]\\ {\overrightarrow{n}}_4^{\prime }={\overrightarrow{t}}_3+\frac{\left({\overrightarrow{t}}_3-{\overrightarrow{t}}_2\right)}{d_2}·d_3& \left[7\right]\\ n_4=\frac{{\overrightarrow{n}}_4^{\prime }}{{\overrightarrow{n}}_4^{\prime }}& \left[8\right]\\ {\overrightarrow{n}}_1^{\prime }={\overrightarrow{t}}_2+\frac{\left({\overrightarrow{t}}_3-{\overrightarrow{t}}_2\right)}{d_2}·d_1& \left[9\right]\\ n_1=\frac{{\overrightarrow{n}}_1^{\prime }}{{\overrightarrow{n}}_1^{\prime }}& \left[10\right]\end{array}$

[0051] θ, the deformation angle between x₁ and x₄, can be found from cos θ={right arrow over (n)}₁·{right arrow over (n)}₄.

Deviation from Planarity

[0052] The points x₁, x₂, x₃, and x₄ can be checked to make sure that they fall generally in a plane. {right arrow over (p)} the unit vector between x₁ and x₄ is:
{right arrow over (p)}=({right arrow over (x)}₄−{right arrow over (x)}₁)/|{right arrow over (x)}₄−{right arrow over (x)}₁|  [11]

[0053] Then {right arrow over (q)}₁, a vector orthogonal to {right arrow over (n)}₁ and {right arrow over (n)}₄, is given by:
{right arrow over (q)}₁={right arrow over (n)}₁×{right arrow over (n)}₄  [12]

[0054] and {right arrow over (q)}₂, a vector orthogonal to {right arrow over (n)}₁ and {right arrow over (p)}, is given by:
{right arrow over (q)}₂={right arrow over (n)}₁×{right arrow over (p)}  [13]

[0055] The angle φ between {right arrow over (q)}₁ and {right arrow over (q)}₂ is a measure of the planarity of the points, is given by:
φ=cos⁻¹({right arrow over (q)}₁·{right arrow over (q)}₂)  [14]

[0056] For planarity it is desirable that φ≦10°≈π/20

Circularity Check (Uniform Stiffness)

[0057] For a medical device of uniform stiffness, the distal end portion will generally assume a circular configuration. The selection of the points, and particularly the pivot point x₁ can be validated by ensuing that they lie substantially along a circle. (For not uniform devices, some other check can be performed on the points). Letting {right arrow over (u)} represent the field direction unit vector and defining ψ=cos⁻¹({right arrow over (u)}·{right arrow over (n)}₁), then α, the lag angle between the applied magnetic field and the direction of the is given by and define α≡(ψ−θ). If m is the magnetic moment of the magnet on the distal end of the medical device, and β its stiffness (β=EI where E is the Young's modulus of the material and I is the bending moment of area), then:

[0058] $\begin{array}{cc}l=\frac{\beta }{\mathrm{mB}}·\frac{\theta }{\sin \alpha }& \left[15\right]\end{array}$
where B is the magnitude of the field strength and l is the length of the medical device between {right arrow over (x)}₁ and {right arrow over (x)}₄.

[0059] The chord length is given by:

[0060] $\begin{array}{cc}{d}^{\prime }=\frac{2l}{\theta }\sin \frac{\theta }{2}& \left[16\right]\end{array}$
If the points lie along a circle, the chord length d′ should be close to c≡|{right arrow over (x)}₄−{right arrow over (x)}₁|, for example, within ten percent, or

[0061] $\frac{c-d^{\prime }}{c}\le 0.1.$

[0062] If the above constraints are satisfied, the curve of the medical device can be estimated as follows:

[0063] v, the unit vector in the plane of the medical device orthogonal to {right arrow over (n)}₁:
{right arrow over (v)}′≡{right arrow over (n)}₄−{right arrow over (n)}₁ cos θ  [17]
{right arrow over (v)}={right arrow over (v)}′/|{right arrow over (v)}′|  [18]

[0064] The curve of the medical device is defined by:

[0065] $\begin{array}{cc}{\overrightarrow{x}}^{\prime }={\overrightarrow{x}}_1+\frac{l}{\theta }\left(1-\cos {\theta }^{\prime }\right)\overrightarrow{v}+\frac{l}{\theta }\sin {\theta }^{\prime }{\overrightarrow{n}}_1& \left[19\right]\end{array}$
for 0≦θ′≦θ

[0066] The envelope or surface of the tip as field direction varies can be determined, based upon the following formula for tip position:

[0067] $\begin{array}{cc}{\overrightarrow{x}}_{\mathrm{tip}}={\overrightarrow{x}}_1+\frac{l}{\theta }\left(1-\cos \theta \right)\overrightarrow{v}+\frac{l}{\theta }\sin \theta {\overrightarrow{n}}_1& \left[20\right]\end{array}$
Varying θ from 0 to θ_mas (=lMB/β) gives the envelope or locus of tip positions as the field is varied.
Determining the Field Direction for a Selected Point

[0068] If a point Z is selected on the envelope, the field direction that take the tip to that location may be computed as follows:

[0069] $\begin{array}{cc}{\overrightarrow{u}}_1=\frac{\left(\overrightarrow{z}-{\overrightarrow{x}}_1\right)}{\overrightarrow{z}-{\overrightarrow{x}}_1}& \left[21\right]\\ \stackrel{\_}{\varphi }={\cos }^{-1}\left({\overrightarrow{u}}_1·{\overrightarrow{n}}_1\right)& \left[22\right]\\ \stackrel{\_}{\theta }={\cot }^{-1}\left[\frac{\left(1-{\tan }^2\stackrel{\_}{\varphi }\right)}{2\tan \stackrel{\_}{\varphi }}\right]=2\stackrel{\_}{\varphi }& \left[23\right]\\ \stackrel{\_}{\psi }=\stackrel{\_}{\theta }+{\sin }^{-1}\left(\frac{\beta \stackrel{\_}{\theta }}{\mathrm{MBl}}\right)& \left[24\right]\end{array}$

[0070] Then the corresponding field direction to orient the tip to point z is given by the unit vector:
{right arrow over (u)}_B={right arrow over (n)}₁ cos ψ+{right arrow over (v)} sin ψ  [25]

[0071] An “accessible surface” may also be defined by rotating the envelope obtained in equation 25 about {right arrow over (n)}₁. {right arrow over (w)}, the unit vector orthogonal to {right arrow over (v)} and {right arrow over (n)}₁, is given by {right arrow over (w)}={right arrow over (v)}×{right arrow over (n)}₁, and {right arrow over (v)}_R, the rotation of {right arrow over (w)} about {right arrow over (n)}₁ is given by {right arrow over (v)}_R=cos ξ{right arrow over (v)}+sin ξ{right arrow over (w)} for 0≦ξ≦2π

Then:

[0072] $\begin{array}{cc}{x}_s=x_1+\frac{l}{\theta }\left(1-\cos \theta \right)\left(\cos \xi \overrightarrow{v}+\sin \xi \overrightarrow{w}\right)+\frac{l}{\theta }\sin \theta {\overrightarrow{n}}_1& \left[26\right]\end{array}$
defines a surface of revolution about n₁ that is the surface accessible to the tip by changing the field direction by varying θ and ξ, a surface is generated analogous to generating a sphere with longitude and latitude.
Generating the Field Direction to any Point on the Surface

[0073] As in Equation [25], given a desired point {right arrow over (z)} on this surface, a corresponding field direction is obtained from:
u_B,S={right arrow over (n)}₁ cos ψ+{right arrow over (v)}_R,S sin {right arrow over (ψ)}  [27]
where {right arrow over (v)}_R,S is obtained from:
{right arrow over (v)}_R,S=cos ξ_s{right arrow over (v)}+sin ξ_s{right arrow over (w)}  [28]
where

[0074] $\begin{array}{cc}{\xi }_s={\tan }^{-1}\left(\frac{\overrightarrow{u}·\overrightarrow{w}}{\overrightarrow{u}·\overrightarrow{v}}\right)& \left[29\right]\end{array}$

[0075] When the desired tip location is beyond the accessible surface, then a combination of change of inserted length and field direction is needed to move the distal end of the device to {right arrow over (z)} from its current location.

[0076] As before, define {right arrow over (u)}₁, and find φ and θ. If l is the new medical device length, and

[0077] $\begin{array}{cc}y=\left(\overrightarrow{z}-{\overrightarrow{x}}_1\right)·{\overrightarrow{n}}_1,\mathrm{then}\stackrel{\_}{l}\frac{\sin \stackrel{\_}{\theta }}{\stackrel{\_}{\theta }}=y\mathrm{or}\stackrel{\_}{l}=\frac{y\stackrel{\_}{\theta }}{\sin \stackrel{\_}{\theta }}.\stackrel{\_}{\psi }\equiv \stackrel{\_}{\theta }+{\sin }^{-1}\left(\frac{\beta \stackrel{\_}{\theta }}{\mathrm{MB}\stackrel{\_}{l}}\right)& \left[30\right]\end{array}$
Given Equations 28 and 29:
{right arrow over (u)}_B,S={right arrow over (n)}₁ cos ψ+{right arrow over (v)}_R,S sin ψ  [31]
and
δl=( l−l)  [32]
is the change in inserted length which together with the change in field direction will take the end of the medical device to the desired location.

[0078] Of course, while described herein in the context of controlling a magnetic surgery system, as stated earlier the virtual device interface of this invention could be applied to other systems which can control the configuration of medical device, including devices whose configuration is controlled mechanically, hydraulically, or through magnetostrictive and electrostrictive means.

[0079] A second embodiment of user interface is illustrated in FIG. 10. The interface of the second embodiment is adapted to help a user control a magnetic navigation system that applies a magnetic field to an operating region in a patient to control the direction of a magnetic medical device in the operating region. An advancer/retractor mechanism is preferably also provided for the controlled advancement/retraction of the medical device. As shown in FIG. 10, the user is presented an interface with several virtual or real buttons, which the user can operate using a mouse, track ball, space ball, joy stick, tablet, or other input device. Operating button 200 initiates the navigation under the present invention, and a window (or other message) is displayed, prompting the user to use the input device to identify a number of points on the medical device on the display 202, including an image 203 of the medical device including the anchor point.

[0080] The user identifies points, for example, by positioning the cursor over the medical device on the display 202. In this preferred embodiment, the inventors have determined that four points is an appropriate balance between user effort and accuracy in characterizing the medical device to be navigated, but other numbers of points could also be used. The points identified are checked or qualified (as described above) and a window (or other message) alerts the user if one or more points are not properly identified.

[0081] With information about the currently applied magnetic field and the position and configuration of the medical device (resulting from the identification of points), the processor implementing the interface can characterize the medical device. The user can then use the input device to operate direction control buttons. In this preferred embodiment there are four such buttons, buttons 204 and 206 for increasing and decreasing the deflection of the medical device, and buttons 208 and 210 for rotating the medical device. The interface can be operated in a discrete navigation mode, a continuous navigation mode, or the interface can allow the user to switch between discrete and continuous navigation. In the discrete navigation mode the user clicks on the buttons 204, 206, 208, or 210, and the image 212 of the a virtual device is superposed over the display 202 showing the configuration of the medical device with the specified new magnetic field applied. Once the user is satisfied with the new field, as represented by the image 212 of the virtual device, then the user operates a control (such as a button) to activate the magnetic navigation system and apply the specified field, causing the actual medical device to assume substantially the configuration and position represented by the image 212 of the virtual device. In the continuous navigation mode, the user could operate in a continuous navigation mode, changes in the direction specified by operating buttons 204, 206, 208, and 210 are automatically implemented, moving the medical device. The user may enter the continuous navigation mode, for example, by holding down a control button. The buttons 204, 206, 208, and 210 preferably can be operated to change deflection and rotation in predetermined, discrete amounts, which preferably can be customized by the user, or can be held down to continuously change the deflection and rotation to the limits of the navigation system. Many other mappings besides separate control of rotation and deflection of the device are possible. For example, the device tip could be controlled or actuated suitably to move within a chosen plane.

[0082] In another embodiment a joystick is provided for interactive device control purposes. In this case, the user would select from a set of possible mappings from joystick deflections to changes in actuation control variables that would modify the configuration of the device. In one particular case where a magnetic surgery system is used for device actuation, the magnetic field would be driven from the joystick. Visual feedback from an imaging system that could employ X-ray, Magnetic Resonance Imaging, Ultrasound or other imaging modalities known to practitioners of the art provides the user with the device configuration, so that the device may be driven interactively by the user to reach a desired target. Other embodiments may use spaceballs or a variety of other input devices, including those that are custom-built, for interactive remote control of device actuation.

[0083] The interface also allows the user to click on a point in the display to mark a target point 214 on the two dimensional display. The interface then determines the magnetic field required of the magnetic navigation system and extension or retraction of the medical device (which is preferably controlled by an automated advancing/retracting system) required to reach the target point 214, and displays an image 216 of a virtual device with the calculated magnetic field applied, and with the appropriate adjustment in length made. The user then operates a control (such as a button), so that calculated field and adjustments in length are applied. Alternatively, the user could operate in a continuous navigation mode (such as by holding down another button on a joystick) so that the applied magnetic field and device length automatically change to bring the distal end of the medical device to the point 214 identified by clicking on the display 202.

[0084] Because the user is attempting to identify a point in three dimensional space by identifying a point on a two dimensional display, the distal end of the medical device may not be precisely where the user intended. The user can switch to another display (if biplane imaging of the operating region is available) to refine the position. Alternatively, where biplane imaging or two corresponding images are available, the user can identify the target point on each image, uniquely identifying the point in three dimensional space. After the user identifies the target in one image, the interface assists the selection of a proper point in the second image. For example, the cursor color or shape can indicate when the cursor is over a valid or invalid position based upon the user's selection on the other display. In discrete navigation, the magnetic navigation system applies the required field to achieve the specified target point when the user actuates an apply field button. In one embodiment of continuous navigation, the magnetic navigation system automatically begins to apply the required field to achieve the specified target point as soon as the target point is properly specified. To prevent unintended movement, the user preferably has to hold down a button to remain in the continuous navigation mode, and when the button is released the magnetic navigation system ceases movement. In an embodiment that employs a joystick, for example, the button could be a trigger button on the joystick. In practice, the apply field button used in the discrete navigation mode and the continuous navigation button used in the continuous navigation mode can be the same.

[0085] Alternatively, the user can make minor adjustments to the position using the buttons 204, 206, 208, and 210. Once the medical device is characterized, as prompted by the interface system, the system can automatically interpret further pointing and clicking on the display 202 as an indication that the user is identifying a target point 214. Alternatively, a button can be provided on the display or on the input device for the user to turn on the target mode.

[0086] As mentioned previously, rather than buttons 204, 206, 208, and 210, the user can input directional changes with a joystick. For example, moving the joystick forward and backward can increase and decrease the deflection, and moving the joystick left and right (or twisting the joystick) can cause rotation. Likewise, it is possible to employ other modes of mapping from the joystick to device actuation. As with using the buttons above, the user can operate in discrete or continuous mode. When operating in discrete mode, the user uses the joystick to position a virtual image 212 on the display 202, and when satisfied with the position, operates a control (such as a button on the joy stick) to cause the navigation system to apply the magnetic field. When operating in the continuous mode, the user operates a control (such as holding down a button on the joystick) so that changes indicated by the movement of the joystick are automatically implemented, with the magnetic navigation system moving in response to change the fields as required. In this mode the medical device responds interactively as the user operates the joystick.

[0087] In a preferred embodiment, advancement of the medical device is preferably separately controlled by the user, for example with a toggle switch on the joystick so that a single joystick can be used for both device steering and device advancement or retraction, or with a separate button on the display 202. However in the target mode, the interface preferably controls both the magnetic navigation system and the advancer system, so that the device is directly computer controlled. In yet another preferred embodiment, advancement of the medical device and device deflections or shape changes are controlled by the user from separate joysticks. Various other embodiments and modes of use may be conceived by those skilled in the art.

[0088] FIG. 11 shows a display 300 from another implementation of the interface system of the second embodiment. The display 300 includes two bi-plane images 302 and 304 of the operating region. The display also includes a control pane 306. The control pane may include a box 308 in which the user can select or specify the stiffness of the medical device; alternatively the medical device type may be selected from a menu of medical devices whose stiffness value is pre-programmed into the system. A computer processor can use the stiffness to determine the configuration of the medical device in an applied magnetic field, both to generate virtual images of the device under specified magnetic fields, and to calculate the applied field to reach a particular target point.

[0089] The control pane 306 also includes a field display 309, with the coordinates of the current magnetic field direction, and pick boxes for selecting the point of application of the field. The control pane has free length box 310, which allows the user to select the free length of the medical device from the end of its sheath. Arrows 312 and 314 allow the user to increase and decrease the specified length, which is implemented by an advancer/retractor mechanism. The control pane 306 also includes slide controls 316 and 318 for controlling the degree of deflection and rotation (like buttons 204 206, 208 and 210). An apply field button 320 allows the user to apply the field specified by the virtual device in the displays. A conform to field button 322 causes the virtual device model to predict and show the shape that would be caused by application of an input magnetic field. A get live button 324 updates the images 302 and 304. Finally, a target mode button 326 allows the user to enter the target mode (as described above), so that pointing and clicking on the displays 302 and 304 identifies a point which the virtual medical device moves to in the discrete mode, or which the actual medical device moves in the continuous mode.

[0090] In still another embodiment of user interface according to the principles of this invention, the user interaction can be simplified further for ease of use. In FIG. 12, graphical buttons 411 and 412 are provided as alternate modes of control of a magnetic navigation system. Button 411 when selected enables defining the direction of the magnetic field in three dimensional space by specification of the direction in two projections (commonly the LAO and RAO perspectives familiar to interventional surgeons). When the corresponding magnetic field is applied to steer a medical device, the device tends to align itself with the magnetic field but with a certain angular lag as dictated by the elastic and magnetic properties of the device. In contrast, button 412 enables direct selection of a target location that the device can move to. In order to use this mode, the user first selects a “Mark catheter base” button 413 (shown highlighted in FIG. 12) with a mouse click. This enables the user to indicate (by drawing with a pen-tablet or by suitably dragging a mouse) the pivot or point of support of the device as well as the pivot direction, by suitably drawing a line in two projected X-ray views. In FIG. 12, a catheter 419 can be seen to extend from a guide sheath 422. The line 420 drawn by the user indicates the pivot point to be the distal tip of the sheath and the direction in which the device 419 is extended at the pivot point. FIG. 13 shows the pivot point and direction marked by lines 505 and 506 respectively in RAO and LAO views 501 and 502. This process in effect provides x₁ and n₁ to the system. Next the user selects the target button 412 as shown in FIG. 14. This provides to the user a “Target” cursor 601 that enables selection of a point 602 in three dimensional space from two X-ray projections. This point is the desired anatomical target point that the user wishes to steer the medical device to. The computer then calculates the requisite field direction and insertion length from equations (25) and (32). The corresponding shape of the device is also computed and displayed as a dashed curve 603. FIG. 15 shows a desired target point in RAO and LAO projections respectively as 602 and 602a together with associated projections of the device shape 603 and 603a11 respectively.

[0091] The required field direction and insertion length may be automatically applied by the system upon the press of a button. Alternatively, the field direction may be applied directly by the system while the user continuously controls the advancement or retraction of the device. The device curve may be updated to be shown as a solid curve, perhaps in a different color, to indicate that the desired change of controls has been effected. FIG. 16 shows an LAO perspective of such an updated device curve 650 together with the user-defined pivot direction 648 and the desired target location 652. Further the device shape may be interactively manipulated by the user as described above. One convenient implementation uses keyboard buttons for increasing or decreasing deflection of the virtual device and for rotating it clockwise or anti-clockwise about the pivot direction. The device shape is continuously computed and updated in the projected displays as well as possibly in a three dimensional display during this process of user adjustment. The corresponding manipulated virtual device 702 is shown in an LAO perspective in FIG. 17 together with the current device configuration 701 and an updated location 703 for the device tip as the use manipulates the virtual device. Once a desired device shape or tip location is achieved the corresponding controls may be applied at the touch of an “Apply” button, or alternatively the system may apply the controls (magnetic field and insertion/retraction in one embodiment) in user-defined steps every time a keyboard button is pressed. Another convenient implementation of continuous control uses a joystick or other input device together with a mapping function to convert joystick movements to changes in control inputs, causing the device to respond continuously to joystick movements. In one implementation the user could, together with moving or deflecting the joystick, press and hold a “trigger” button to cause the changes in control inputs to be applied. The user observes (on an X-ray image or other mode of visualization) the device response to joystick movements and may interactively stop device response when a desired device configuration has been attained, for example by releasing a “trigger” button on the joystick.

[0092] The actuation system employed to steer or orient the flexible device can take several different forms such as magnetic actuation by external magnetic fields, mechanical actuation through the use of steering cables or other force transmission elements within the device, device actuation using piezoelectric materials or electrostrictive polymers such as silicone elastomers, actuation with magnetostrictive elements embedded in the device, or other technologies known to practitioners of the art.

[0093] In a preferred embodiment, the actuation system can use magnetic fields generated by external magnets to orient magnets incorporated in the distal portion of the device. The interaction of the external field with the device magnets serves to orient the device in controllable fashion by controlling the externally applied magnetic field. The external magnets could employ either permanent magnets or electromagnets possibly employing superconductivity to produce strong magnetic fields. FIG. 1 shows a schematic illustration of such a system and device.

[0094] In an alternate preferred embodiment employing electrostrictive actuation, the device could use electrically activated electrostrictive materials to bend or steer the device. FIG. 18 shows a flexible device 851 which incorporates electrostrictive elements 854 and 855. Electrical leads 858, 859 and 865, 866 connect the elements 854 and 855 respectively to a voltage source 856. The electrostrictive element 854 is sandwiched between rings 860 and 862. When a voltage is applied across the leads 858 and 859, the electrostrictive element 854 decreases in length, forcing the flexible device to bend in the plane defined by the device axis and the orientation of the electrostrictive element relative to the device axis. The amount of bending can be controlled by varying the applied voltage. When a voltage is applied across leads 865 and 866, the shortening of element 855 forces the device to bend in the opposite direction.

[0095] In some cases, it may be useful to determine the bending plane of the device. This is relevant where a non-magnetic steering mechanism is used to orient the device. This may be done by employing an imaging method such as X-ray imaging or by using a localization sensor embedded in the device. If a localization sensor which gives complete position and orientation of the device is used, the orientation of the actuation mechanism such as steering cable or electrostrictive element is known in a reference configuration, the corresponding orientation in a general configuration may be obtained by processing the data obtained from the localization sensor. Thus the bending plane of the device in three dimensional space may be determined. Alternatively, from imaging such as fluoroscopy, the change in the tangent vector to the device tip upon a small change in an actuation control variable may be recorded, which also provides information needed to determine the bending plane in three dimensional space.

[0096] Knowledge of the bending plane can be used to control the device for targeting purposes and for configurational changes with the aid of a computational model in a manner similar to that described above.

[0097] The embodiments discussed here are for purposes of example only, and other embodiments of actuation systems such as mechanical actuators or others known in the field can be used for device steering. In the following a computational model for magnetic actuation with externally applied magnetic fields is described as a non-limiting example, with the understanding that the computational model and its use can generally incorporate a variety of possible actuation systems.

[0098] The actuation system coordinates may be registered to the imaging system coordinates and to the localization system coordinates by means of several methods known to those skilled in the art. For example, the actuation system may employ external magnetic fields, in which case the source of the fields such as an external magnet could be mechanically registered to the imaging system by the use of suitably placed markers placed in known positions with respect to the actuation system. These positions may also be ascribed coordinates in the reference frame of the imaging system since information from multiple imaging projections or from three dimensional imaging would be available from the imaging system. The positions of three markers determined in both coordinate systems suffices to generate a rigid coordinate transformation (rotation and translation) which is the requisite registration. Additional markers may be used for improved accuracy as desired.

[0099] Likewise if a device localization system is used, it could be used to determine the coordinates of known markers with respect to the localization system as well, thence providing a mutual registration of the various distinct coordinate frames employed by the various systems. In practice, such a registration would be performed at the start of a medical procedure. Subsequent movement of the patient table may be tracked to within a suitable accuracy to permit updated registrations as may be required on a regular basis during a clinical procedure.

[0100] While a magnetic actuation system has been discussed in the above for non-limiting illustrative purposes, various alternative actuation systems may be deployed and registered as needed according to the methods taught here. For example, a device employing piezoelectric actuators may use a coordinate frame in common with an X-ray imaging system, requiring only registration to a device localization system. Likewise, the markers in question may be either mechanically placed by the user or may be based on anatomical locations as pinpointed by the user.