US 6,714,841 B1Grant

Head cursor control interface for an automated endoscope system for optimal positioning

Issue Date:2004-03-30

•25 Claims

•4 Drawing Sheets

Abstract

A medical system that is coupled to an endoscope which provides a video image to a monitor. The system includes an electrical circuit to overlay a graphic image onto the video image provided by the endoscope. The endoscope is moved by a robotic arm.

Metadata

Assignee

Computer Motion, Inc.

Inventors

James Wright
Hamid Wasti
Darrin R. Uecker

Application Information

Application Number:US 09/179,039

Filing Date:1998-10-26

Priority Date:1995-09-15

Art Unit:7

Classifications

IPC:

G05B 1500G05B 1904A61B 505A61B 100

Field of Search:

700259700264700258700251600407600118

Patent Drawings (4 sheets)

Description

[0002] This application is a continuation application of application Ser. No. 08/904,047, filed Jul. 31, 1997, U.S. Pat. No. 5,911,036, which is a continuation application of application Ser. No. 08/529,095, filed Sep. 15, 1995, now U.S. Pat. No. 5,825,982.

Background of the Invention

[0003] 1. Field of the Invention

[0004] The present invention relates to a graphical user interface that can be remotely controlled by a surgeon to control various devices and conditions of an operating room.

[0005] 2. Description of Related Art

[0006] To reduce the invasiveness of surgery, endoscopes are commonly utilized to view the internal organs of a patient. One end of the endoscope contains a lens which is inserted into the patient through a small incision of the skin. The lens focuses an image that is transmitted by fiber optic cable to a camera located at the opposite end of the endoscope. The camera is coupled to a monitor that displays a video image of the patient.

[0007] The endoscope can be used in conjunction with another surgical instrument that is inserted into the patient. An assistant typically holds the endoscope while the surgeon manipulates the surgical instrument. The assistant moves the endoscope in response to instructions from the surgeon. Any mis-communication between the surgeon and the assistant may result in an error in the movement of the endoscope, thereby requiring the surgeon to repeat the instruction. Additionally, holding the endoscope for a significant amount of time may cause the assistant to become fatigued.

[0008] U.S. application Ser. No. 07/927,801 discloses a robotic arm that holds and moves an endoscope in response to commands from the surgeon. The commands are provided through a hand controller or a foot pedal. The controller and pedal require coordinated movements which may detract the surgeon from the surgical procedure. It would be desirable to provide an interface that manipulates a robotically controlled surgical device while requiring minimal physical coordination by the surgeon. Additionally, it would be desirable to provide a single interface that allows the surgeon to control a number of devices such as an operating table, laparoscopic camera, laser tool, etc.

Summary of the Invention

[0009] The present invention is an interface that allows a surgeon to remotely control surgical devices and conditions of an operation room. The surgeon views a video image that is displayed by a monitor. The monitor may be coupled to a video device such as a laparoscopic camera that is attached to the end of an endoscope. Static graphic images and a dynamic graphic cursor are overlayed onto the video image. The graphic cursor has a pixel location on the monitor which corresponds to a spatial location of a pointer signal. The pointer signal is transmitted by a transmitter worn on the head of the surgeon. The pointer signal may be a laser which is directed to a screen that is located adjacent to a detection camera. The surgeon may move the graphic cursor relative to the video image by tilting his head and varying the spatial location of the pointer signal. The interface may have a controller which generates output signals in response to the movement of the pointer signal. The output signals may move a robotic arm which controls the position of the endoscope. The controller may also generate command signals when the graphic cursor is moved into a static graphic image. The command may vary a condition of the operating room such as the position of the operating table.

Brief Description of the Drawings

[0010] The objects and advantages of the present invention will become more readily apparent to those ordinarily skilled in the art after reviewing the following detailed description and accompanying drawings, wherein:

[0011] FIG. 1 is a perspective view of a robotic system that controls an endoscope;

[0012] FIG. 2 is a schematic of an endoscope within two different coordinate systems;

[0013] FIG. 3 is a schematic of a head cursor interface electrical circuit.

[0014] FIG. 4 is a front view of a monitor which displays a video image and a plurality of graphical overlays;

[0015] FIG. 5 is a schematic of an endoscope within various coordinate frames;

[0016] FIG. 6 is a schematic of a robotic arm.

Detailed Description of the Invention

[0017] Referring to the drawings more particularly by reference numbers, FIG. 1 is a robotic system 10 that controls a surgical instrument 12. The surgical instrument 12 is typically an endoscope that is inserted into a patient. The tip of the endoscope typically has a lens(es) that focuses an image of the patient. The endoscope 12 may also have fiber optic cable that transmits the image to a camera 16 located at the end of the scope. The camera 16 is typically a charge coupled device (CCD). The camera 16 is coupled to a monitor 18 which displays the image.

[0018] The instrument 12 is moved by a robotic arm assembly 20 that is coupled to a computer 22. In the preferred embodiment the robotic assembly 20 has a linear actuator 24 that is mounted to a surgical table 26. The linear actuator 24 moves a linkage arm assembly 28 in a linear manner relative to the table 26. The linear actuator 24 defines an origin of a fixed first coordinate system that has a first x axis, a first y axis and a first z axis.

[0019] The linkage arm assembly 28 contains a first linkage arm 30 attached to an end effector 32. The first linkage arm 30 is mounted to a first rotary actuator 34 which can rotate the arm. The first rotary actuator 34 is attached to a second linkage arm 36. The second linkage arm 36 is mounted to a second rotary actuator 38 that can rotate the arms. The rotary actuator 38 is attached to the output shaft of the linear actuator 24.

[0020] The end effector 32 is typically coupled to a pair of passive joints (not shown) that allow rotation of the instrument as indicated by the arrows in FIG. 1. The end effector 32 may also have a worm gear (not shown) that rotates the endoscope about the longitudinal axis of the instrument. As shown in FIG. 2, the junction of the instrument 12 and the end effector 32 define the origin of a second coordinate system which has a second x axis (x′), a second y axis (y′) and a second z axis (z′). The junction of the end effector 32 and the instrument 12 also define a third coordinate system which has a third x axis (x″), a third y axis (y″) and a third z axis (z″). The zag axis is always parallel with the longitudinal axis of the instrument 12. The actuators receive input signals from the computer 22 to control the movement of the robotic arm assembly 20.

[0021] Referring to FIG. 1, the surgeon wears a transmitter unit 40 that transmits a pointer signal 42 which is received by a receiver unit 44. The transmitter unit 40 is preferably a laser pointer which emits a laser beam 42. The laser pointer may have a blow switch 46 that allows the surgeon to turn the laser on and off by blowing or drawing in the air of a tube located adjacent to the surgeons mouth. The transmitter 40 may be a laser switch sold by Point Source, Inc. of Germantown, Ohio. Although a laser transmitter is shown and described, the transmitter may be an acoustic or electromagnetic device that generates a wave that is detected by an appropriate detector(s). It being understood that any system that can detect a physical movement of the surgeon is encompassed by the present invention.

[0022] The receiver unit 42 preferably includes a screen 48 that is in the field of view of a camera 50. The laser beam 42 creates an illuminated dot on the screen 48 which is then detected by the camera 50. The camera 50 is preferably a charged coupled device (CCD). When the surgeon moves his head, the pointer signal 42 moves to a new spatial location on the screen 48. The surgeon can therefore control the position of the illuminated dot by tilting his head.

[0023] As shown in FIG. 3, the CCD camera 50 is coupled to an image digitizer 52 which digitizes the images provided by the camera 50. The digitizer 52 provides digitally based values that correspond to the light intensity detected by each pixel of the camera 50. The digitizer 52 is coupled to a position detector 54 which detects the spatial location of the pointer signal 42 relative to the screen 48. The detector 54 first compares the intensity values of each pixel with a threshold value. The detector 54 provides an associated value of 1 for each pixel that has an intensity which exceeds the threshold value, and a value of 0 for each pixel which is below the threshold value. The threshold value is selected to correspond to the intensity of an illuminated dot created by the laser beam 42 striking the screen 50. The threshold value is preferably large enough to filter out background light.

[0024] After each pixel is assigned a 1 or 0 value, the x and y spatial coordinates of the pointer signal 42 relative to the screen 48 is computed by determining the center of mass of the pixels which have an assigned value of 1 in accordance with the following equations.

M x = \frac{\sum_{i - n, j - m}^{} x_{i} \cdot 0 (i, j)}{\sum_{i - n, j - m}^{} 0 (i, j)}

My = \frac{\sum_{i - n, j - m}^{} y_{j} \cdot 0 (i, j)}{\sum_{i - n, j - m}^{} 0 (i, j)}

[0025] where;

[0026] Mx=the x coordinate of the center of mass.

[0027] My=the y coordinate of the center of mass.

[0028] O(i,j)=the assigned value of the pixels i through j.

[0029] Xi=the x coordinate of the pixels i through n.

[0030] Yj=the y coordinate of the pixels j through m.

[0031] The x and y spatial coordinates generated by the detector 54 are provided to an interface controller 56. The interface controller 56 maps the x and y spatial coordinates generated by the detector to corresponding pixel locations on the monitor 18 . The interface controller 56 is coupled to a graphic overlay processor 58 and a robot controller 60. The graphic overlay processor 58 is coupled to the monitor 18. Although separate controllers are shown and described, it is to be understood that the blocks depicted are merely functional and that the operations may be performed by a single microprocessor or different combinations of processors.

[0032] As shown in FIG. 4, the monitor 18 displays a video image 62 provided by the camera 16 of the endoscope 12. The video image 62 is typically an internal organ of a patient. The graphic overlay processor 58 generates a series of static graphic images 64-70 that overlay onto the video image 62 displayed by the monitor 18. The overlay processor 58 also generates a dynamic graphic cursor 72 that can move across the monitor 18. The graphic cursor 72 may move in conjunction with any movement of the laser beam 42 emitted from the pointer 40 mounted to the surgeon's head.

[0033] To move the cursor 72, the surgeon may move his head and vary the spatial location of the pointer signal 42 on the screen 48. The new pointer location is detected by the CCD camera 50. The position detector 54 computes the x and y spatial coordinates which are then provided to the interface controller 56. The interface controller 56 maps the new x and y spatial coordinates to pixel locations on the video image 62. The controller 56 then provides the new pixel locations to the graphic overlay processor 58 which displays the cursor 72.

[0034] The interface controller 56 may also generate output signals to move the robotic arm assembly 20 in conjunction with the position of the cursor 72. For example, the interface controller 56 may generate output signals to move the robotic arm 20 and endoscope 12 and to move the video image in the direction of the cursor. In this manner, the surgeon can view a new location within the patient by merely moving his head. Although a cursor 72 is shown and described, it is to be understood that the surgeon may move the robotic arm 20 and the video image 62 without a cursor 72 by merely tilting his head and watching the displayed image on the monitor 18.

[0035] The static graphic images 64-70 may provide input commands to control various devices such as the robotic arm assembly 20. For example, the graphic images 64 and 66 provide ZOOM IN and ZOOM OUT commands for the video image. When the surgeon moves the cursor 72 into the area of the IN graphic image 64, the interface controller 56 generates output signals to move the robotic arm 20 so that the end of the endoscope moves closer to the object displayed by the monitor 18. Likewise, when the cursor 72 is moved into the OUT graphic 66, the controller 56 generates output signals to move the robotic arm 20 so that the endoscope moves away from the object shown on the monitor 18.

[0036] To determine the interaction between the cursor 72 and the graphic images 64-70, the interface controller 56 compares the pixel locations that correspond to the x and y coordinates provided by the detector 54 with a group of pixel locations associated with each graphic image. If the x and y pixel locations associated with the pointer signal equal a pixel location of a graphic image, the controller 56 generates a command associated with the graphic image. The graphic images 64-70 may be removed from the video image by drawing in air on the tube 46 and turning off the laser pointer 40.

[0037] The graphical image 68 may generate a command to create a “pointer” out of the cursor 72 so that any subsequent movement of the cursor 72 will not generate a corresponding movement of the robotic arm 20. The surgeon may use the pointer as an instructional aid for other personnel viewing the monitor 18.

[0038] The robotic arm 20 can be manipulated by initially placing the cursor 72 in the PAN graphic 70 and then moving the cursor 72 about the monitor 18. The interface controller 56 generates new pixel locations associated with the cursor movement which are then provided to the robot controller 60 to move the robotic arm so that the video image moves in conjunction with the movement of the cursor and the spatial location of the laser beam on the screen.

[0039] The process of moving the endoscope is performed by initially subtracting the new pixel position from an arbitrary reference pixel position to determine a Ax and a Ay pixel movement of the cursor 72 within the video image 62. The computed movement (Δx and Δy) is multiplied by a weighted pseudoinverse of the following Jacobean matrix with reference to the coordinate system shown in FIG. 5.

[\begin{matrix} \frac{- xy \sin φ}{f} + y \cos θ & \frac{- f ρ}{Z_{c}} - (f + \frac{x^{2}}{f}) & \frac{x}{Z_{c}} \\ - x \cos θ - \sin φ (f + \frac{y^{2}}{f}) - \frac{f ρ \sin φ}{Z_{c}} & - \frac{xy}{f} & \frac{y}{Z_{c}} \end{matrix}]

[0040] where;

[0041] the angles θ, φ and ρ are measured by robotic position sensors (not shown). The angles provide spherical coordinates of the endoscope within a scope frame coordinate system that has an origin at the pivot point of the instrument and the patient.

[0042] x, y=the new pixel coordinates of the reference point.

[0043] Z_c=is a constant.

[0044] f=the focal length of the endoscope lens.

[0045] The product (Vθ, Vφ and Vρ) of the reference point movement (Δx and Δy) and the Jacobean matrix is the computed movement of the endoscope by the robotic arm assembly in a spherical coordinate frame. The spherical coordinates (Vθ, Vφ and Vρ) are converted into Cartesian coordinates (Vx, Vy and Vz) by a transformation. The movement of the endoscope within the scope frame is converted to the fixed first coordinate system by an additional transformation matrix or matrices.

[0046] Referring to FIG. 6, the controller 60 typically computes the movement of the robotic arm assembly 20 in accordance with the following equations.

a3 = π - \cos^{- 1} (\frac{x^{2} + y^{2} - {L1}^{2} + {L2}^{2}}{- 2 \cdot L1L2})

Δ = \cos^{- 1} (\frac{x^{2} + y^{2} - {L1}^{2} - {L2}^{2}}{2 L1 \sqrt{x^{2} + y^{2}}})

a0 = \tan^{- 1} 2 (\frac{y}{x})

[0047] where;

[0048] a2=angle between the second linkage arm 36 and the x axis.

[0049] a3=angle between the first linkage arm 30 and the longitudinal axis of the second linkage arm 36.

[0050] L1=length of the second linkage arm.

[0051] L2=length of the first linkage arm.

[0052] x=x coordinate of the end effector in the first coordinate system.

[0053] y=y coordinate of the end effector in the first coordinate system.

[0054] To move the end effector to a new location of the x-y plane, the computer computes a change in the angles a2 and a3, and then provides output signals to move the actuators accordingly. The original angular position of the end effector is provided to the computer by the position sensors. The computer moves the linkage arms an angle that corresponds to the difference between the new location and the original location of the end effector. A differential angle Δa2 corresponds to the amount of angular displacement provided by the third actuator 38 and a differential angle Δa3 corresponds to the amount of angular displacement provided by the second actuator 34.

[0055] To improve the effectiveness of the system 10, the system is constructed so that the desired movement of the surgical instrument correlates to a direction relative to the image displayed by the monitor. Thus when the robotic arm moves the endoscope 12 up, the scope always appears to move in the up direction relative to the image displayed by the monitor. To accomplish this result, the computer converts the desired movement of the end of the endoscope in the third coordinate system to coordinates in the second coordinate system, and then converts the coordinates of the second coordinate system into the coordinates of the first coordinate system.

[0056] Referring to FIG. 2, the desired movement of the endoscope is converted from the third coordinate system to the second coordinate system by using the following transformation matrix;

(\begin{matrix} Δ x^{'} \\ Δ y^{'} \\ Δ z^{'} \end{matrix}) = (\begin{matrix} \cos (a6) & 0 & - \sin (a6) \\ - \sin (a5) \sin (a6) & \cos (a5) & - \sin (a5) \cos (a6) \\ \cos (a5) \sin (a6) & \sin (a5) & \cos (a5) \cos (a6) \end{matrix}) (\begin{matrix} Δ x^{″} \\ Δ y^{″} \\ Δ z^{″} \end{matrix})

[0057] where;

[0058] Δx″=the desired incremental movement of the scope along the x″ axis of the third coordinate system.

[0059] Δy″=the desired incremental movement of the scope along the y″ axis of the third coordinate system.

[0060] Δz″=the desired incremental movement of the scope along the z″ axis of the third coordinate system.

[0061] a5=the angle between the z′ axis and the scope in the y′-z′ plane.

[0062] a6=the angle between the z′ axis and the scope in the x′-z′ plane.

[0063] Δx′=the computed incremental movement of the scope along the x′ axis of the second coordinate system.

[0064] Δy′=the computed incremental movement of the scope along the y′ axis of the second coordinate system.

[0065] Δz′=the computed incremental movement of the scope along the z′ axis of the second coordinate system.

[0066] The angles a5 and a6 are provided by position sensors coupled on the end effector 32.

[0067] The desired movement of the endoscope is converted from the second coordinate system to the first coordinate system by using the following transformation matrix;

(\begin{matrix} Δ x \\ Δ y \\ Δ z \end{matrix}) = (\begin{matrix} \cos (π) & - \sin (π) & 0 \\ \sin (π) & \cos (π) & 0 \\ 0 & 0 & 1 \end{matrix}) (\begin{matrix} Δ x^{'} \\ Δ y^{'} \\ Δ z^{'} \end{matrix})

[0068] where;

[0069] Δx′=the computed incremental movement of the scope along the x′ axis of the second coordinate system.

[0070] Δy′=the computed incremental movement of the scope along the y′ axis of the second coordinate system.

[0071] Δz′=the computed incremental movement of the scope along the z′ axis of the second coordinate system.

[0072] π=is the angle between the first linkage arm and the x axis of the first coordinate system.

[0073] Δx=the computed incremental movement of the scope along the x axis of the first coordinate system.

[0074] Δy=the computed incremental movement of the scope along the y axis of the first coordinate system.

[0075] Δz=the computed incremental movement of the scope along the z axis of the first coordinate system.

[0076] The incremental movements Δx and Δy are inserted into the algorithms described above for computing the angular movements (Δa2 and Δa3) of the robotic arm assembly to determine the amount of rotation that is to be provided by each actuator. The value Δz is used to determine the amount of linear movement provided by the linear actuator 24.

[0077] The endoscope 12 is typically coupled to the camera 16 such that any spinning of the instrument about its own longitudinal axis will result in a corresponding rotation of the video image 62 on the monitor 18. Rotation of the instrument and video image may disorient the viewer. It is therefore desirable to maintain the orientation of the video image. In the preferred embodiment, the end effector has a worm gear which rotates the surgical instrument about the longitudinal axis of the instrument. To insure proper orientation of the endoscope, the worm gear rotates the instrument about its longitudinal axis an amount Δθ6 to insure that the y″ axis is oriented in the most vertical direction within the fixed coordinate system. Δθ6 is computed from the following cross-products.

[0078] Δθ6=z_i″×(yo″×yi″)

[0079] where;

[0080] Δθ6=the angle that the instrument is to be rotated about the z″ axis.

[0081] yo″=is the vector orientation of the y″ axis when the instrument is in the first position.

[0082] yi″=is the vector orientation of the y″ axis when the instrument is in the second position.

[0083] zi″=is the vector orientation of the z″ axis when the instrument is in the second position.

[0084] The vectors of the yi″ and zi″ axis are computed with the following algorithms.

[{zi}^{″}] = a5 [\begin{matrix} \cos a6 & 0 & - \sin a6 \\ - \sin a5 \sin a6 & \cos a5 & - \sin a5 \cos a6 \\ \cos a5 \sin a6 & \sin a5 & \cos a5 \cos a6 \end{matrix}] [\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}]

xi″=z×zi″

[0085] yi″=zi″×xi″

[0086] where;

[0087] a6=is the angle between the instrument and the z axis in the y-z plane.

[0088] a5=is the angle between the instrument and the z axis in the x-z plane.

[0089] z=is the unit vector of the z axis in the first coordinate system.

[0090] The angles a5 and a6 are provided by the joint position sensors of the end effector. The vector yo″ is computed using the angles a5 and a6 of the instrument in the original or first position. For the computation of yi″, the angles a5 and a6 of the second position are used in the transformation matrix. After each arm movement yo″ is set to yi″ and a new yi″ vector and corresponding Δθ6 angle are computed and used to re-orient the endoscope. Using the above described algorithms, the worm gear continuously rotates the instrument about its longitudinal axis to insure that the pivotal movement of the endoscope does not cause a corresponding rotation of the viewing image.

[0091] While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.

[0092] For example, although graphic images which provide commands to control a robotic arm are shown and described, it is to be understood that the graphics may generate commands that control other devices. The graphic overlay processor 58 may provide an entire menu that allows the surgeon to adjust the operating table or the lighting of the camera 16. Additionally, surgical instruments such as laser cutters or electrode coagulators may be controlled by the surgeon through the head activated graphical interface provided by the present invention. The present invention generally provides a remotely controlled graphically based interface that allows the surgeon to control various devices and conditions at a surgical site.

Claims

What is claimed is:

1. A medical system adapted to be coupled to an endoscope that is coupled to a monitor which displays a video image provided by the endoscope, the medical system comprising:

a robotic arm configured to be coupled to the endoscope; and

an electrical circuit that is coupled to said robotic arm and being configured to overlay a graphic image onto the video image;

wherein the overlaid graphic image is adjustable to manipulate the video image.

2. The system of claim 1, wherein said electrical circuit overlays a dynamic graphic cursor onto the video image.

3. The system of claim 2, wherein the dynamic graphic cursor can be moved into the graphic image to select a function.

4. The system of claim 3, wherein the function is a movement of said robotic arm.

5. The system of claim 1 further comprising a controller configured to adjust the endoscope to manipulate the video image in response to adjustments to the overlaid graphic image.

6. The system of claim 1 wherein the graphic image comprises at least one of “in” for video zoom in, “out” for video zoom out, and “pan” for video pan movement.

7. A medical system adapted to be coupled to an endoscope that is coupled to a monitor which displays a video image provided by the endoscope, the medical system comprising:

movement means for moving the endoscope; and

overlay means for overlaying a graphic image onto the video image;

wherein the overlaid graphic image is adjustable to manipulate the video image.

8. The system of claim 7, wherein said overlay means overlays a dynamic graphic cursor onto the video image.

9. The system of claim 8, wherein the dynamic graphic cursor can be moved into the graphic image to select a function.

10. The system of claim 9, wherein the function is a movement of said movement means.

11. The system of claim 7 further comprising control means for adjusting the endoscope to manipulate the video image in response to adjustments to the overlaid graphic image.

12. A method for operating a medical system, comprising:

moving an endoscope within a patient;

displaying a video image provided by the endoscope on a monitor coupled to the endoscope; and

overlaying a graphic image onto the video image;

wherein the overlaid graphic image is adjustable to manipulate the video image.

13. The method of claim 12, further comprising selecting a function by moving a dynamic graphic cursor into the graphic image.

14. The method of claim 13, further comprising adjusting the endoscope in response to the selection of the function.

15. The method of claim 13 wherein the graphic image comprises at least one of an “in” function for video zoom in, an “out” function for video zoom out, and a “pan” function for video pan movement.

16. The method of claim 12 further comprising adjusting the endoscope to manipulate the video image in response to adjustments to the overlaid graphic image.

17. A medical system adapted to be coupled to an endoscope that is coupled to a monitor which displays a video image provided by the endoscope, the medical system comprising:

a robotic arm configured to be coupled to the endoscope;

an electrical circuit that is coupled to said robotic arm and being configured to overlay a dynamic graphic cursor and graphic image onto the video image; and

a cursor input device coupled to said electrical circuit;

wherein the overlaid graphic image is adjustable by the dynamic graphic cursor to manipulate the video image.

18. The system of claim 17, wherein the dynamic graphic cursor can be moved into the graphic image to select a function.

19. The system of claim 18, wherein the function is a movement of said robotic arm.

20. The system of claim 17 further comprising a controller configured to adjust the endoscope to manipulate the video image in response to adjustments to the overlaid graphic image.

21. The system of claim 17 wherein the graphic image comprises at least one of “in” for video zoom in, “out” for video zoom out, and “pan” for video pan movement; and wherein the “in,” “out,” and “pan” are selectable by the dynamic graphic cursor.

22. A medical system adapted to be coupled to an endoscope that is coupled to a monitor which displays a video image provided by the endoscope, the medical system comprising:

movement means for moving the endoscope;

overlay means for overlaying a dynamic graphic cursor and a graphic image onto the video image; and

input means for moving the dynamic graphic cursor;

wherein the overlaid graphic image is adjustable by the dynamic graphic cursor to manipulate the video image.

23. The system of claim 22, wherein the dynamic graphic cursor can be moved into the graphic image to select a function.

24. The system of claim 23, wherein the function is a movement of said movement means.

25. The system of claim 22 further comprising control means for adjusting the endoscope to manipulate the video image in response to adjustments to the overlaid graphic image.

Patent Citations (189)

Patent	Date	Inventor	Cited By
US977825(A)	1910-12-01	Murphy	Applicant
US3171549(A)	1965-03-01	Orloff	Applicant
US3280991(A)	1966-10-01	Melton et al.	Applicant
US4058001(A)	1977-11-01	Waxman	Applicant
US4128880(A)	1978-12-01	Cray, Jr.	Applicant
US4221997(A)	1980-09-01	Flemming	Applicant
US4367998(A)	1983-01-01	Causer	Applicant
US4401852(A)	1983-08-01	Noso et al.	Applicant
US4456961(A)	1984-06-01	Price et al.	Applicant
US4460302(A)	1984-07-01	Moreau et al.	Applicant
US4474174(A)	1984-10-01	Petruzzi	Applicant
US4491135(A)	1985-01-01	Klein	Applicant
US4503854(A)	1985-03-01	Jako	Applicant
US4517963(A)	1985-05-01	Michel	Applicant
US4523884(A)	1985-06-01	Clement et al.	Applicant
US4586398(A)	1986-05-01	Yindra	Applicant
US4604016(A)	1986-08-01	Joyce	Applicant
US4616637(A)	1986-10-01	Caspari et al.	Applicant
US4624011(A)	1986-11-01	Watanabe et al.	Applicant
US4633389(A)	1986-12-01	Tanaka et al.	Applicant
US4635292(A)	1987-01-01	Mori et al.	Applicant
US4641292(A)	1987-02-01	Tunnell et al.	Applicant
US4655257(A)	1987-04-01	Iwashita	Applicant
US4672963(A)	1987-06-01	Barken	Applicant
US4676243(A)	1987-06-01	Clayman	Applicant
US4728974(A)	1988-03-01	Nio et al.	Applicant
US4762455(A)	1988-08-01	Coughlan et al.	Applicant
US4791934(A)	1988-12-01	Brunnett	Applicant
US4791940(A)	1988-12-01	Hirschfeld et al.	Applicant
US4794912(A)	1989-01-01	Lia	Applicant
US4815006(A)	1989-03-01	Andersson et al.	Applicant
US4815450(A)	1989-03-01	Patel	Applicant
US4837734(A)	1989-06-01	Ichikawa et al.	Applicant
US4852083(A)	1989-07-01	Niehaus et al.	Applicant
US4853874(A)	1989-08-01	Iwamoto et al.	Applicant
US4854301(A)	1989-08-01	Nakajima	Applicant
US4860215(A)	1989-08-01	Seraji	Applicant
US4863133(A)	1989-09-01	Bonnell	Applicant
US4883400(A)	1989-11-01	Kuban et al.	Applicant
US4930494(A)	1990-06-01	Takehana et al.	Applicant
US4945479(A)	1990-07-01	Rusterholz et al.	Applicant
US4949717(A)	1990-08-01	Shaw	Applicant
US4954952(A)	1990-09-01	Ubhayakar et al.	Applicant
US4965417(A)	1990-10-01	Massie	Applicant
US4969709(A)	1990-11-01	Sogawa et al.	Applicant
US4969890(A)	1990-11-01	Sugita et al.	Applicant
US4979933(A)	1990-12-01	Runge	Applicant
US4979949(A)	1990-12-01	Matsen, III et al.	Applicant
US4980626(A)	1990-12-01	Hess et al.	Applicant
US4989253(A)	1991-01-01	Liang et al.	Applicant
US4996975(A)	1991-03-01	Nakamura	Applicant
US5019968(A)	1991-05-01	Wang et al.	Applicant
US5020001(A)	1991-05-01	Yamamoto et al.	Applicant
US5065741(A)	1991-11-01	Uchiyama et al.	Applicant
US5078140(A)	1992-01-01	Kwoh	Applicant
US5086401(A)	1992-02-01	Glassman et al.	Applicant
US5091656(A)	1992-02-01	Gahn	Applicant
US5097829(A)	1992-03-01	Quisenberry	Applicant
US5097839(A)	1992-03-01	Allen	Applicant
US5098426(A)	1992-03-01	Sklar et al.	Applicant
US5105367(A)	1992-04-01	Tsuchihashi et al.	Applicant
US5109499(A)	1992-04-01	Inagami et al.	Applicant
US5123095(A)	1992-06-01	Papadopoulos et al.	Applicant
US5131105(A)	1992-07-01	Harrawood et al.	Applicant
US5142930(A)	1992-09-01	Allen et al.	Applicant
US5145227(A)	1992-09-01	Monford, Jr.	Applicant
US5166513(A)	1992-11-01	Keenan et al.	Applicant
US5175694(A)	1992-12-01	Amato	Applicant
US5182641(A)	1993-01-01	Diner et al.	Applicant
US5184601(A)	1993-02-01	Putman	Applicant
US5187574(A)	1993-02-01	Kosemura et al.	Applicant
US5196688(A)	1993-03-01	Hesse et al.	Applicant
US5201325(A)	1993-04-01	McEwen et al.	Applicant
US5201743(A)	1993-04-01	Haber et al.	Applicant
US5217003(A)	1993-06-01	Wilk	Applicant
US5221283(A)	1993-06-01	Chang	Applicant
US5228429(A)	1993-07-01	Hatano	Applicant
US5230623(A)	1993-07-01	Guthrie et al.	Applicant
US5236432(A)	1993-08-01	Matsen, III et al.	Applicant
US5251127(A)	1993-10-01	Raab	Applicant
US5257999(A)	1993-11-01	Slanetz, Jr.	Applicant
US5271384(A)	1993-12-01	McEwen et al.	Applicant
US5279309(A)	1994-01-01	Taylor et al.	Applicant
US5282806(A)	1994-02-01	Haber	Applicant
US5289273(A)	1994-02-01	Lang	Applicant
US5289365(A)	1994-02-01	Caldwell et al.	Applicant
US5299288(A)	1994-03-01	Glassman et al.	Applicant
US5300926(A)	1994-04-01	Stoeckl	Applicant
US5303148(A)	1994-04-01	Mattson et al.	Applicant
US5304185(A)	1994-04-01	Taylor	Applicant
US5305203(A)	1994-04-01	Raab	Applicant
US5305427(A)	1994-04-01	Nagata	Applicant
US5309717(A)	1994-05-01	Minch	Applicant
US5313306(A)	1994-05-01	Kuban et al.	Applicant
US5320630(A)	1994-06-01	Ahmed	Applicant
US5337732(A)	1994-08-01	Grundfest et al.	Applicant
US5339799(A)	1994-08-01	Kami et al.	Applicant
US5343385(A)	1994-08-01	Joskowicz et al.	Applicant
US5343391(A)	1994-08-01	Mushabac	Applicant
US5345538(A)	1994-09-01	Narayannan et al.	Applicant
US5357962(A)	1994-10-01	Green	Applicant
US5368015(A)	1994-11-01	Wilk	Applicant
US5368428(A)	1994-11-01	Hussey et al.	Applicant
US5371536(A)	1994-12-01	Yamaguchi	Applicant
US5382885(A)	1995-01-01	Salcudean et al.	Applicant
US5388987(A)	1995-02-01	Badoz et al.	Applicant
US5395369(A)	1995-03-01	McBrayer et al.	Applicant
US5397323(A)	1995-03-01	Taylor et al.	Applicant
US5402801(A)	1995-04-01	Taylor	Applicant
US5403319(A)	1995-04-01	Matsen, III et al.	Applicant
US5408409(A)	1995-04-01	Glassman et al.	Applicant
US5410638(A)	1995-04-01	Colgate et al.	Applicant
US5417210(A)	1995-05-01	Funda et al.	Applicant
US5417701(A)	1995-05-01	Holmes	Applicant
US5422521(A)	1995-06-01	Neer et al.	Applicant
US5431645(A)	1995-07-01	Smith et al.	Applicant
US5434457(A)	1995-07-01	Josephs et al.	Applicant
US5442728(A)	1995-08-01	Kaufman et al.	Applicant
US5443484(A)	1995-08-01	Kirsch et al.	Applicant
US5445166(A)	1995-08-01	Taylor	Applicant
US5451924(A)	1995-09-01	Massimino et al.	Applicant
US5455766(A)	1995-10-01	Scheller et al.	Applicant
US5458547(A)	1995-10-01	Teraoka et al.	Applicant
US5458574(A)	1995-10-01	Machold et al.	Applicant
US5476010(A)	1995-12-01	Fleming et al.	Applicant
US5490117(A)	1996-02-01	Oda et al.	Applicant
US5490843(A)	1996-02-01	Hildwein et al.	Applicant
US5506912(A)	1996-04-01	Nagasaki et al.	Applicant
US5512919(A)	1996-04-01	Araki	Applicant
US5515478(A)	1996-05-01	Wang	Applicant
US5524180(A)	1996-06-01	Wang et al.	Examiner
US5544654(A)	1996-08-01	Murphy et al.	Applicant
US5553198(A)	1996-09-01	Wang et al.	Applicant
US5562503(A)	1996-10-01	Ellman et al.	Applicant
US5571110(A)	1996-11-01	Matsen, III et al.	Applicant
US5572999(A)	1996-11-01	Funda et al.	Applicant
US5609560(A)	1997-03-01	Ichikawa et al.	Applicant
US5626595(A)	1997-05-01	Sklar et al.	Applicant
US5629594(A)	1997-05-01	Jacobus et al.	Applicant
US5630431(A)	1997-05-01	Taylor	Applicant
US5631973(A)	1997-05-01	Green	Applicant
US5636259(A)	1997-06-01	Khutoryansky et al.	Applicant
US5649956(A)	1997-07-01	Jensen et al.	Applicant
US5657429(A)	1997-08-01	Wang et al.	Applicant
US5658250(A)	1997-08-01	Blomquist et al.	Applicant
US5676673(A)	1997-10-01	Ferre et al.	Applicant
US5695500(A)	1997-12-01	Taylor et al.	Applicant
US5696837(A)	1997-12-01	Green	Applicant
US5718038(A)	1998-02-01	Takiar et al.	Applicant
US5735290(A)	1998-04-01	Sterman et al.	Applicant
US5749362(A)	1998-05-01	Funda et al.	Applicant
US5754741(A)	1998-05-01	Wang et al.	Applicant
US5776126(A)	1998-07-01	Wilk et al.	Applicant
US5779623(A)	1998-07-01	Bonnell	Applicant
US5800423(A)	1998-09-01	Jensen	Applicant
US5807284(A)	1998-09-01	Foxlin	Applicant
US5807378(A)	1998-09-01	Jensen et al.	Applicant
US5808665(A)	1998-09-01	Green	Examiner
US5810880(A)	1998-09-01	Jensen et al.	Applicant
US5813813(A)	1998-09-01	Daum et al.	Applicant
US5814038(A)	1998-09-01	Jensen et al.	Applicant
US5815640(A)	1998-09-01	Wang et al.	Examiner
US5817084(A)	1998-10-01	Jensen	Applicant
US5825982(A)	1998-10-01	Wright et al.	Examiner
US5841950(A)	1998-11-01	Wang et al.	Examiner
US5859934(A)	1999-01-01	Green	Applicant
US5876325(A)	1999-03-01	Mizuno et al.	Applicant
US5878193(A)	1999-03-01	Wang et al.	Examiner
US5882206(A)	1999-03-01	Gillio	Applicant
US5887121(A)	1999-03-01	Funda et al.	Applicant
US5907664(A)	1999-05-01	Wang et al.	Examiner
US5920395(A)	1999-07-01	Schulz	Examiner
US5931832(A)	1999-08-01	Jensen	Applicant
US5950629(A)	1999-09-01	Taylor et al.	Applicant
US6024695(A)	2000-02-01	Taylor et al.	Applicant
US6201984(B1)	2001-03-01	Funda et al.	Examiner
US6463361(B1)	2002-10-01	Wang et al.	Examiner
DEU9204118.3	1992-07-01		Applicant
DE4310842(C2)	1995-01-01		Applicant
EP239409(A1)	1987-09-01		Applicant
EP424687(A1)	1991-05-01		Applicant
EP776738(A2)	1997-06-01		Applicant
WO91/04711	1991-04-01		Applicant
WO92/20295	1992-11-01		Applicant
WO93/13916	1993-07-01		Applicant
WO94/18881	1994-09-01		Applicant
WO94/26167	1994-11-01		Applicant
WO97/15240	1997-05-01		Applicant
WO98/25666	1998-06-01		Applicant

Non-Patent Literature (41)

“Endocorporeal Surgery Using Remote Manipulators” (Ned S. Rasor and J.W. Spickler) Remotely Manned Systems—Exploration and Operation in Space, California Institute of Technology 1973.Applicant
“A Survey Study of Teleoperators, Robotics, and Remote Systems Technology” (Arthur D. Alexander, III) Remotely Manned Systems—Exploration and Operation in Space, California Institute of Technology 1973.Applicant
“Impacts of Telemation on Modern Society” (Arthur D. Alexander, III), On the Theory and Practice of Robots and Manipulators vol. II, 1974.Applicant
Transcript of a video presented by SRI at the 3rd World Congress of Endoscopic Surgery in Bordeaux on Jun. 18-20, 1992, in Washington on Apr. 9, 1992, and in San Diego, CA on Jun. 4-7, 1992 entitled “Telepresence Surgery—The Future of Minimally Invasive Medicine”.Applicant
Statutory Declaration of Dr. Philip S. Green, presenter of the video entitled “Telepresence Surgery—The Future of Minimally Invasive Medicine”.Applicant
Abstract of a presentation “Telepresence: Advanced Teleoperator Technology for Minimally Invasive Surgery” (P. Green et al.) given at the 3rd World Congress of Endoscopic Surgery in Bordeaux, Jun. 18-20, 1992.Applicant
Abstract of a presentation “Telepresence: Advanced Teleoperator Technology for Minimally Invasive Surgery”, (P. Green et al.) given at “Medicine meets virtual reality” symposium in San Diego, Jun. 4-7, 1992.Applicant
Abstract of a presentation “Camera Control for Laparoscopic Surgery by Speech-Recognizing Robot: Constant Attention and Better Use of Personnel” (Colin Besant et al.) given at the 3rd World Congress of Endoscopic Surgery in Bordeaux, Jun. 18-20, 1992.Applicant
“A Literature Review: Robots in Medicine” (B. Preising et al.) IEEE Jun. 1991.Applicant
“Robots for the Operating Room” (Elizabeth Corcoran), The New York Times, Sunday Jul. 19, 1992, Section 3, p. 9, col. 1.Applicant
“Taming the Bull: Safety in a Precise Surgical Robot” (Russell H. Taylor et al.), IEEE 1991.Applicant
Abstract of a presentation “Design Considerations of a New Generation Endoscope Using Robotics and Computer Vision Technology” (S.M. Krishnan et al.) given at the 3rd World Congress of Endoscopic Surgery in Bordeaux, Jun. 18-20, 1992.Applicant
Abstract of a presentation “3-D Vision Technology Applied to Advanced Minimally Invasive Surgery Systems” given at the 3rd World Congress of Endoscopic Surgery in Bordeaux, Jun. 18-20, 1992.Applicant
“Analysis of the Surgeon's Grasp for Telerobotic Surgical Manipulation” (Frank Tendick and Lawrence Stark), IEEE 1989.Applicant
“Kinematic Control and Visual Display of Redundant Teleoperators” (Hardi Das et al.), IEEE 1989.Applicant
“A New System for Computer Assisted Neurosurgery” (S. Lavallee), IEEE 1989.Applicant
“An Advanced Control Micromanipulator for Surgical Applications” (Ben Gayed et al.), Systems Science vol. 13 1987.Applicant
“Force Feedback-Based Telemicromanipulation for Robot Surgery on Soft Tissues” (A.M. Sabatini et al.), IEEE 1989.Applicant
“Six-Axis Bilateral Control of an Articulated Slave Manipulator Using a Cartesian Master Manipulator” (Masao Inoue), Advanced Robotics 1990.Applicant
“On a Micro-Manipulator for Medical Application—Stability Consideration of its Bilateral Controller” (S. Majima et al.), Mechatronics 1991.Applicant
“Anthropomorphic Remote Manipulator”, NASA Tech Briefs 1991.Applicant
“Controlling Remote Manipulators through Kinesthetic Coupling” (A.K. Bejczy), Computers in Mechanical Engineering 1983.Applicant
“Design of a Surgeon-Machine Interface for Teleoperated Microsurgery” (Steve Charles M.D. et al.), IEEE 1989.Applicant
“A Robot in an Operating Room: A Bull in a China Shop” (J.M. Dolan et al.), IEEE 1987.Applicant
Abstract of a presentation “Concept and Experimental Application of a Surgical Robotic System the Steerable MIS Instrument SMI” given at the 3rd World Congress of Endoscopic Surgery in Bordeaux, Jun. 18-20, 1992.Applicant
Abstract of a presentation given at the 3rd World Congress of Endoscopic Surgery in Bordeaux, Jun. 18-20, 1992, entitled “Session 15/1”.Applicant
Abstract of a presentation “A Pneumatic Controlled Sewing Device for Endoscopic Application the MIS Sewing Instrument MSI” given at the 3rd World Congress of Endoscopic Surgery in Bordeaux, Jun. 18-20, 1992.Applicant
Abstract of a presentation given at the 3rd World Congress of Endoscopic Surgery in Bordeaux (Jun. 18 to 20, 1992), entitled “Session 15/2”.Applicant
Abstract of a presentation given at the 3rd World Congress of Endoscopic Surgery in Bordeaux (Jun. 18 to 20, 1992), entitled Session 15/4.Applicant
Abstract of a presentation given at the 3rd World Congress of Endoscopic Surgery in Bordeaux (Jun. 18 to 20, 1992), entitled “Session 15/5”.Applicant
“Properties of Master-Slave Robots” (C. Vibet), Motor-con 1987.Applicant
“A New Microsurgical Robot System for Corneal Transplantation” (Noriyuki Tejima), Precision Machinery 1988.Applicant
“Human/Robot Interaction via the Transfer of Power and Information Signals—Part I: Dynamics and Control Analysis” (H. Kazerooni), IEEE 1989.Applicant
“Human/Robot Interaction via the Transfer of Power and Information Signals—Part II: An Experimental Analysis” (H. Kazerooni), IEEE 1989.Applicant
“Power and Impedance Scaling in Bilateral Manipulation” (J. Edward Colgate), IEEE 1991.Applicant
“S.M.O.S.: Stereotaxical Microtelemanipulator for Ocular Surgery” (Aicha Guerrouad and Pierre Vidal), IEEE 1989.Applicant
“Motion Control for a Sheep Shearing Robot” (James P. Trevelyan et al.), Proceedings of the 1st International Symposium on Robotics Research, MIT, Cambridge, Massachusetts, USA, 1983.Applicant
“Robots and Telechirs” (M.W. Thring), Wiley 1983.Applicant
Industrial Robotics (Gordon M. Mair), Prentice Hall 1988 (pp. 41-43, 49-50, 54, 203-209 enclosed).Applicant
“Student Reference Manual for Electronic Instrumentation Laboratories” (Wolf et al.), Prentice Hall, New Jersey 1990, pp. 498 amd 499.Applicant
“Surgery in Cyberspace” (Taubes), Discover Magazine, Dec. 1994.Applicant

Abstract

Description

Background of the Invention

[0003] 1. Field of the Invention

[0004] The present invention relates to a graphical user interface that can be remotely controlled by a surgeon to control various devices and conditions of an operating room.

[0005] 2. Description of Related Art

Summary of the Invention

Brief Description of the Drawings

[0011] FIG. 1 is a perspective view of a robotic system that controls an endoscope;

[0012] FIG. 2 is a schematic of an endoscope within two different coordinate systems;

[0013] FIG. 3 is a schematic of a head cursor interface electrical circuit.

[0014] FIG. 4 is a front view of a monitor which displays a video image and a plurality of graphical overlays;

[0015] FIG. 5 is a schematic of an endoscope within various coordinate frames;

[0016] FIG. 6 is a schematic of a robotic arm.

Detailed Description of the Invention

M x = \frac{\sum_{i - n, j - m}^{} x_{i} \cdot 0 (i, j)}{\sum_{i - n, j - m}^{} 0 (i, j)}

My = \frac{\sum_{i - n, j - m}^{} y_{j} \cdot 0 (i, j)}{\sum_{i - n, j - m}^{} 0 (i, j)}

[0025] where;

[0026] Mx=the x coordinate of the center of mass.

[0027] My=the y coordinate of the center of mass.

[0028] O(i,j)=the assigned value of the pixels i through j.

[0029] Xi=the x coordinate of the pixels i through n.

[0030] Yj=the y coordinate of the pixels j through m.

[\begin{matrix} \frac{- xy \sin φ}{f} + y \cos θ & \frac{- f ρ}{Z_{c}} - (f + \frac{x^{2}}{f}) & \frac{x}{Z_{c}} \\ - x \cos θ - \sin φ (f + \frac{y^{2}}{f}) - \frac{f ρ \sin φ}{Z_{c}} & - \frac{xy}{f} & \frac{y}{Z_{c}} \end{matrix}]

[0040] where;

[0042] x, y=the new pixel coordinates of the reference point.

[0043] Z_c=is a constant.

[0044] f=the focal length of the endoscope lens.

[0046] Referring to FIG. 6, the controller 60 typically computes the movement of the robotic arm assembly 20 in accordance with the following equations.

a3 = π - \cos^{- 1} (\frac{x^{2} + y^{2} - {L1}^{2} + {L2}^{2}}{- 2 \cdot L1L2})

Δ = \cos^{- 1} (\frac{x^{2} + y^{2} - {L1}^{2} - {L2}^{2}}{2 L1 \sqrt{x^{2} + y^{2}}})

a0 = \tan^{- 1} 2 (\frac{y}{x})

[0047] where;

[0048] a2=angle between the second linkage arm 36 and the x axis.

[0049] a3=angle between the first linkage arm 30 and the longitudinal axis of the second linkage arm 36.

[0050] L1=length of the second linkage arm.

[0051] L2=length of the first linkage arm.

[0052] x=x coordinate of the end effector in the first coordinate system.

[0053] y=y coordinate of the end effector in the first coordinate system.

[0056] Referring to FIG. 2, the desired movement of the endoscope is converted from the third coordinate system to the second coordinate system by using the following transformation matrix;

(\begin{matrix} Δ x^{'} \\ Δ y^{'} \\ Δ z^{'} \end{matrix}) = (\begin{matrix} \cos (a6) & 0 & - \sin (a6) \\ - \sin (a5) \sin (a6) & \cos (a5) & - \sin (a5) \cos (a6) \\ \cos (a5) \sin (a6) & \sin (a5) & \cos (a5) \cos (a6) \end{matrix}) (\begin{matrix} Δ x^{″} \\ Δ y^{″} \\ Δ z^{″} \end{matrix})