METHOD FOR HANDLING AN OPERATOR COMMAND EXCEEDING A MEDICAL DEVICE STATE LIMITATION IN A MEDICAL ROBOTIC SYSTEM

[0001] This application claims priority to U.S. provisional application Ser. No. 60/752,174 filed Dec. 20, 2005, which is incorporated herein by reference.

[0002] The present invention generally relates to medical robotic systems and in particular, to a method for handling an operator command exceeding a medical device state limitation in a medical robotic system.

[0003] Medical robotic systems such as those used in performing minimally invasive surgical procedures offer many benefits over traditional open surgery techniques, including less pain, shorter hospital stays, quicker return to normal activities, minimal scarring, reduced recovery time, and less injury to tissue. Consequently, demand for minimally invasive surgery using such medical robotic systems is strong and growing.

[0004] Examples of medical robotic systems include the davinci® Surgical System and the davinci® S™ Surgical System from Intuitive Surgical, Inc., of Sunnyvale, Calif. Each of these systems includes a surgeon's console, a patient-side cart, a high performance three-dimensional (“3-D”) vision system, and Intuitive Surgical's proprietary EndoWrist™ articulating instruments, which are modeled after the human wrist so that when added to the motions of the robotic arm assembly holding the surgical instrument, they allow at least a full six degrees of freedom of motion, which is comparable to or even greater than the natural motions of open surgery.

[0005] The patient-side cart typically includes three or more robotic arm assemblies each having a slave manipulator for holding and manipulating a medical device such as a surgical instrument or image capturing device for performing and viewing a medical procedure at a surgical site within a patient. To manipulate these medical devices, the surgeon's console also includes master input devices which may be selectively associated with the slave manipulators to manipulate their respectively held medical devices.

[0006] Whenever one of the slave manipulators reaches one or more of its workspace limits, the surgeon loses control of corresponding degrees of freedom movement of the medical device being held by that slave manipulator. Thus, unless special action is taken by a master-slave control system, operator manipulation of the associated master input device does not produce the expected response for the slave manipulator and/or its held medical device. Further, the slave manipulator response in this case may prove to be not only unintuitive to the surgeon, but also unpredictable and potentially dangerous during a surgical procedure.

[0007] One approach to mitigate this problem is to provide haptic feedback back to the master input device so as to urge the surgeon away from the workspace limitation. Although highly effective, such an approach may still be improved upon to not only ensure operation within workspace limitations, but also to provide operation that does not result in unintuitive or surgically dangerous responses.

[0008] Accordingly, one object of aspects of the present invention is a method for handling an operator command that would result in a medical device exceeding one of its state limitations so as to avoid exceeding the limitation.

[0009] Another object of aspects of the present invention is a method for handling an operator command that would result in a medical device exceeding one of its state limitations so as to avoid exceeding the limitation in such a manner so as not to result in unintuitive or dangerous behavior.

[0010] Still another object of aspects of the present invention is a method for handling an operator command that would result in a medical device exceeding one of its state limitations so as to prevent a surgeon from forceably exceeding the limitation.

[0011] These and additional objects are accomplished by the various aspects of the present invention, wherein briefly stated, one aspect is a method for handling an operator command exceeding a medical device state limitation in a medical robotic system, comprising: determining whether a robotically controlled medical device is being commanded to a state that exceeds a state limitation by a threshold amount; and if the medical device is being commanded to such a state, then disengaging control of the medical device by a master input device so that the medical device remains in its current state, providing a message to an operator of the master input device to lighten the hold of the master input device, detecting the lightened hold of the master input device by the operator, and upon such detection, setting at least one parameter associated with the master input device so that the medical device is commanded to a different state that does not exceed the state limitation and reengaging the control of the medical device by the master input device.

[0012] Another aspect is a medical robotic system comprising: a robotic arm adapted with a medical device; a master input device associated with the robotic arm for operator manipulation of the medical device; and a processor configured to determine whether the medical device is being commanded to a state that exceeds a state limitation by a threshold amount in response to the operator manipulation of the master input device, and if the medical device is being commanded to such a state, then disengage control of the medical device by the master input device so that the medical device remains in its current state, provide a message to an operator of the master input device to lighten hold of the master input device, detecting the lightened hold of the master input device by the operator, and upon such detection, set at least one parameter associated with the master input device so that the medical device is commanded to a different state that does not exceed the state limitation and reengage the control of the medical device by the master input device.

[0013] Additional objects, features and advantages of the various aspects of the present invention will become apparent from the following description of its preferred embodiment, which description should be taken in conjunction with the accompanying drawings.

[0014] FIG. 1 illustrates a top view of an operating room employing a medical robotic system utilizing aspects of the present invention.

[0015] FIG. 2 illustrates a simplified side view of a robotic arm assembly employing various aspects of the present invention.

[0016] FIG. 3 illustrates a block diagram of a master/slave control system utilizing aspects of the present invention.

[0017] FIG. 4 illustrates a flow diagram of a method for handling an operator command exceeding a medical device state limitation in a medical robotic system, utilizing aspects of the present invention.

[0018] FIG. 1 illustrates, as an example, a top view of an operating room employing a medical robotic system. The medical robotic system in this case is a minimally invasive robotic surgical system 100 including a Console (“C”) utilized by a Surgeon (“S”) while performing a medical procedure, such as a diagnostic or surgical procedure, with assistance from one or more Assistants (“A”), on a Patient (“P”) who is lying down on an Operating table (“O”).

[0019] The Console includes a 3-D monitor 104 for displaying a 3-D image of a surgical site to the Surgeon, left and right manipulatable control devices 108, 109, a foot pedal 105, and a processor 102. The control devices 108, 109 may include any one or more of a variety of input devices such as joysticks, gloves, trigger-guns, hand-operated controllers, or the like. The processor 102 may be a dedicated computer integrated into the Console or positioned next or near to it, or it may be broken up into a number of processing or controller components that are distributed in a distributed processing fashion throughout the System 100.

[0020] The Surgeon performs a medical procedure by manipulating the input devices 108, 109 (also referred to herein as “master manipulators”) so that the processor 102 causes slave manipulators of their respectively associated robotic arm assemblies 128, 129 to manipulate their respective removably coupled surgical instruments 138, 139 (also referred to herein as “tools”) accordingly, while the Surgeon views the surgical site in 3-D on the Console monitor 104 as it is captured by a stereoscopic endoscope 140.

[0021] Each of the tools 138, 139, as well as the Endoscope 140, is conventionally inserted through a tool guide (not shown) into the Patient so as to extend down to the surgical site through a corresponding minimally invasive incision such as Incision 166. The number of surgical tools used at one time and consequently, the number of robotic arms being used in the system 100 will generally depend on the medical procedure being performed and the space constraints within the operating room, among other factors. If it is necessary to change a tool being used during a procedure, the Assistant may remove the tool no longer being used from its robotic arm assembly, and replace it with another tool 131 from a Tray (“T”) in the operating room.

[0022] Each of the robotic arm assemblies 122, 128, 129 includes a slave manipulator and setup arms. The slave manipulators are robotically moved using motor controlled joints (also referred to herein as “active joints”) in order to manipulate and/or move their respectively held medical devices. The setup arms may be manually manipulated by releasing normally braked joints (also referred to herein as “setup joints”) to horizontally and vertically position the robotic arm assemblies 122, 128, 129 so that their respective medical devices may be inserted into their respective tool guides.

[0023] Preferably, the monitor 104 is positioned near the Surgeon's hands so that it will display a projected image that is oriented so that the Surgeon feels that he or she is actually looking directly down onto the operating site. To that end, images of the tools 138, 139 preferably appear to be located substantially where the Surgeon's hands are located.

[0024] The processor 102 performs various functions in the system 100. One important function that it performs is to translate and transfer the mechanical motion of control devices 108, 109 to their respective slave manipulators of robotic arm assemblies 128, 129 through control signals over bus 110 so that the Surgeon can effectively manipulate their respective tools 138, 139. Another important function is to implement various control system processes as described herein.

[0025] Although described as a processor, it is to be appreciated that the processor 102 may be implemented in practice by any combination of hardware, software and firmware. Also, its functions as described herein may be performed by one unit, or divided up among different components, each of which may be implemented in turn by any combination of hardware, software and firmware.

[0026] For additional details on the construction and operation of medical robotic systems such as described herein, see, e.g., commonly owned U.S. Pat. No. 6,493,608 “Aspects of a Control System of a Minimally Invasive Surgical Apparatus,” and commonly owned U.S. Pat. No. 6,671,581 “Camera Referenced Control in a Minimally Invasive Surgical Apparatus,” which are incorporated herein by reference.

[0027] FIG. 2 illustrates, as an example, a side view of a simplified (not necessarily in proportion or complete) robotic arm assembly 200 (which is representative of the robotic arm assemblies 128, 129) holding a surgical instrument 250 (which is representative of tools 138, 139) for performing a medical procedure. The surgical instrument 250 is removably held in tool holder 240. The robotic arm assembly 200 is mechanically supported by a base 201, which may be part of a patient-side movable cart or affixed to the operating table or ceiling. It includes links 202, 203 which are coupled together and to the base 201 through horizontal setup joints 204, 205.

[0028] The setup joints 204, 205 in this example are passive joints that allow manual positioning of the arm 200 when their brakes are released. For example, setup joint 204 allows link 202 to be manually rotated about axis 206, and setup joint 205 allows link 203 to be manually rotated about axis 207. The setup arm or portion of the robotic arm assembly 200 includes these setup joints.

[0029] Although only two links and two setup joints are shown in this example, more or less of each may be used as appropriate in this and other robotic arm assemblies in conjunction with the present invention. For example, although setup joints 204, 205 are useful for horizontal positioning of the arm 200, additional setup joints may be included and useful for limited vertical and angular positioning of the arm 200. For major vertical positioning of the arm 200, however, the arm 200 may also be slidably moved along the vertical axis of the base 201 and locked in position.

[0030] The robotic arm assembly 200 also includes three active joints driven by motors. A yaw joint 210 allows arm section 230 to rotate around an axis 261, and a pitch joint 220 allows arm section 230 to rotate about an axis perpendicular to that of axis 261 and orthogonal to the plane of the drawing. The slave manipulator of the robotic arm assembly 200 includes these active joints.

[0031] The arm section 230 is configured so that sections 231, 232 are always parallel to each other as the pitch joint 220 is rotated by its motor. As a consequence, the instrument 250 may be controllably moved by driving the yaw and pitch motors so as to pivot about the pivot point 262, which is generally located through manual positioning of the setup joints 204, 205 so as to be at the point of incision into the patient. In addition, an insertion gear 245 may be coupled to a linear drive mechanism (not shown) to extend or retract the instrument 250 along its axis 263.

[0032] Although each of the yaw, pitch and insertion joints or gears, 210, 220, 245, is controlled by an individual joint or gear controller, the three controllers are controlled by a common master/slave control system so that the slave manipulator of the robotic arm assembly 200 may be controlled through user (e.g., surgeon) manipulation of its associated master manipulator.

[0033] FIG. 3 illustrates, as an example, a block diagram of a master/slave control system 300 for controlling movement of the slave manipulator of the robotic arm assembly 128 and consequently, the position and orientation of its attached tool 138, as commanded by movement of the master manipulator 108 by a surgeon. A similar control system may also be provided for the slave manipulator of the robotic arm assembly 129 and its associated master manipulator 109.

[0034] Both the master and slave manipulators include a number of linkages connected by joints so as to facilitate multiple degrees-of-freedom movement. As the surgeon moves the master manipulator 108 from one position to another during the course of performing a surgical procedure, sensors associated with the master manipulator joints provide information indicating such command movement in master joint space, and sensors associated with the slave manipulator joints provide information indicating slave manipulator and consequently, tool 138 movement in slave joint space for feedback purposes.

[0035] l A master input processing unit 301 receives the information of the master joint positions, which are sampled at the control system processing rate (e.g., 1300 Hz in the present example), from the master joint sensors in the master manipulator 108, and computes joint velocities from the sensed joint positions. A master forward kinematics processing unit 302 receives the master joint positions and velocities from the master input processing unit 301, transforms them from master joint space to corresponding positions and velocities of the master frame (i.e., the frame associated with the master manipulator 108) in Cartesian space relative to the eye reference frame (i.e., the reference frame associated with the position of the surgeon's eyes), using, for example, a Jacobian matrix and eye related information separately determined and provided in block 303.

[0036] A scale and offset processing unit 304 receives the Cartesian position and velocity commands from the master forward kinematics processing unit 302, scales the commanded movement according to a scale factor selected to perform the surgical procedure, and takes into account programmable offsets to generate desired slave tool frame (i.e., the frame associated with the tool 138) positions and velocities. The scale adjustment is useful where small movements of the slave manipulator of the robotic arm assembly 128 are desired relative to larger movement of the master manipulator 108 in order to allow more precise movement of the slave tool 138 at the surgical site. The offsets, on the other hand, determine, for example, the corresponding position and/or orientation of an end effector frame (e.g., the frame associated with an end effector of the tool 138) in the camera reference frame (i.e., the frame associated with the distal tip of the endoscope 140) relative to a position and orientation of the master frame in the eye reference frame.

[0037] A simulated slave processing unit 308 receives desired slave tool frame position and velocity commands from the scale and offset processing unit 304, and limits the desired slave tool frame position, orientation and velocities, to assigned Cartesian Limits for instance to enforce correct and intuitive operation of the tool 138 by keeping it within its dexterous workspace. The simulated slave processing unit 308 generates simulated slave joint positions and velocities corresponding to the limited slave tool frame positions and velocities, while making sure that the generated slave joint positions and velocities do not exceed the actual slave joint's range of motion and maximum velocities (i.e., joint limits) even in the vicinity of kinematic singularities for the slave kinematics.

[0038] An inverse scale and offset processing unit 306 receives the simulated joint position and velocity commands from the simulated slave processing unit 308, and performs an inverse function to that of the scale and offset processing unit 304 on them. A Cartesian controller 307 receives as first inputs, the inputs to the scale and offset processing unit 304 and as second inputs, the outputs of the inverse scale and offset processing unit 306. The Cartesian controller 307 then generates an error signal as a difference of the first and second inputs, and a Cartesian force “F_CART” from the error signal such as with the following formula:
F_CART=K(Δx)+B(Δx)
where “K” is a spring constant, “B” is a damping constant, “Δx” is the difference between the Cartesian velocity inputs to the Cartesian controller 307 and “Δx” is the difference between the Cartesian position inputs to the Cartesian controller 307. For an orientation error, a corresponding torque in Cartesian space is determined.

[0039] As will be further described in reference to FIG. 4, when the Cartesian force “F_CART” (and/or its corresponding torque in Cartesian space) exceeds a programmed threshold level, a special method for handling operator commands resulting in such force may optionally be performed for an additional level of patient safety during a minimally invasive surgical procedure.

[0040] A master transpose kinematics processing unit 315 receives the Cartesian force F_CART through a summation node 314, and generates a corresponding torque in joint space using, for example, the Jacobian transpose matrix and kinematic relationships associated with the master manipulator 108. A master output processing unit 316 receives the master torque signals from the master transpose kinematics processing unit 315, generates electrical currents corresponding to the master torque signals, and supplies the electrical currents to corresponding master joint motors of the master manipulator 108. As a result, a surgeon operating the master manipulator 108 feels the Cartesian force, F_CART, whenever the surgeon is commanding a position or velocity which exceeds system Cartesian or slave joint limits, or would result in a kinematic singularity condition for the slave manipulator of the robotic arm assembly 128.

[0041] As the master input processing unit 301 is receiving master joint positions from sensors in the master manipulator 108, a slave input processing unit 309 is also receiving slave joint positions from position sensors in the slave manipulator at the control system processing rate. A joint control unit 400 receives the slave joint positions from the slave input processing unit 309 and the simulated joint position commands provided from the simulated slave processing unit 308, and generates slave torque command signals for the slave joint motors and master torque feedback command signals for the master joint motors.

[0042] The slave torque command signals are generated by the joint control unit 400 so as to drive joints of the slave manipulator until feedback errors calculated in the joint control unit 400 zero out. A slave output processing unit 310 receives the slave torque command signals from the joint control unit 400, converts them into appropriate electrical currents, and supplies the electrical currents to the joint motors of the slave manipulator so as to drive the motors accordingly.

[0043] The master torque feedback command signals are generated by the joint control unit 400 as a function of the slave joint position and velocity tracking errors so as to reflect forces being exerted against the tool 138 or its slave manipulator back to the master manipulator 108 so that they may be felt by the surgeon. A kinematic mapping unit 311 receives the master torque feedback command signals from the joint control unit 400, and generates the corresponding Cartesian force at the tip of the tool 138 relative to the camera frame of the endoscope 140 using the slave kinematic configuration and the previously calculated slave fulcrum (e.g., pivot point) position information provided in block 312.

[0044] A gain 313 adjusts the magnitude of the Cartesian force so as to ensure system stability while providing adequate force sensation to the surgeon. The gain adjusted Cartesian force is then passed through the summation node 314, and processed along with the Cartesian force provided by the Cartesian controller 307 through the Master transpose kinematics processing unit 315 and Master output processing 316 as previously described in reference to their processing of the Cartesian force provided by the Cartesian controller 307.

[0045] Additional details related to conventional aspects of the master/slave control system 300, such as the various reference frames referred to herein and the calculation of the surgeon eye related information provided in block 303 and the slave fulcrum information provided in block 312, which are based upon well-known mathematics, are described, for example, in the previously incorporated by reference U.S. Pat. No. 6,424,885, “Camera Referenced Control in a Minimally Invasive Surgical Apparatus.”

[0046] The joint control unit 400 includes a joint controller for each active joint of the slave manipulator of the robotic arm assembly 128 that is being controlled by the master/slave control system 300. In particular, where the slave manipulator 128 includes a yaw joint 210, a pitch joint 220, and an insertion axis gear 245, such as the robotic arm assembly 200 of FIG. 2, each of these joints or gears will have its own controller.

[0047] FIG. 4 illustrates, as an example, a flow diagram of a method for handling an operator command exceeding a medical device state limitation in a medical robotic system such as the medical robotic system 100. Although the method is described herein as supplementing the haptic feedback Cartesian force “F_CART”, resulting from the slave tool frame command provided to the simulated slave processing unit 308 exceeding one or more state limitations (e.g., a singularity, position or velocity limitation) as described in reference to FIG. 3 above, it may also be employed in other medical robotic systems with or without such haptic feedback. Further, although the method is described herein as responding to one or more of the state limitations described in reference to FIG. 3 being exceeded, it may also be employed in other medical robotic systems to respond to other or additional state limitations being exceeded.

[0048] In 401, the master/slave control system 300 is operating in its normal “following mode” as described above in reference to FIG. 3.

[0049] In 402, the method periodically checks to determine whether the current slave tool command would result in one or more tool state limitations being exceeded beyond programmed threshold levels. If it doesn't matter which state limitation is being exceeded and by how much, such checking may be performed in the Cartesian controller 307 by simply checking the current magnitude of the Cartesian force “F_CART”. On the other hand, if it does matter which state limitation is being exceeded and/or by how much, then this task may be performed in the simulated slave processing unit 308 on each of the slave limitations for corresponding programmed threshold levels. Once it is determined in 402 that one or more state limitations is being exceeded beyond corresponding programmed threshold levels, then the method performs 403-408 as described below.

[0050] In 403, the normal “following mode” is interrupted so that the slave and master manipulators, 128 and 108, may be respectively locked in certain positions. Such interruption in this case takes the form of disengaging control of the slave manipulator 128 (and its tool 138) by the master manipulator 108.

[0051] In 404, the slave manipulator 128 is then position servoed (i.e., “locked”) at its current position by, for example, switching the input to the joint control unit 400 from the output of the simulated slave processing unit 308 (as in normal “following mode” operation) to a latched value of the output of the slave input processing unit 309, so that the input to the joint control unit 400 is the same as its feedback (i.e., the output of the slave processing unit 309) at the time the ‘lock’ command is executed and will resist any external force tending to displace it

[0052] In 405, the master manipulator 108 is position servoed (i.e., “locked”) so as to maintain the current output of the Cartesian controller 307 (e.g., the Cartesian force “F_CART”) being commanded on the master manipulator 108. Although shown as following 404, the order in which 404 and 405 are performed may be reversed, or preferably, they may be performed substantially concurrently.

[0053] In 406, a message is provided to the operator to lighten his or her grip on the master manipulator 108. The message may be conveyed in any conventional manner such as displaying it on a computer screen viewable to the operator, causing a pre-recorded audio message to be played on speakers (or earphones) hearable by the operator, and causing a vibration on the master manipulator 108 so as to indicate to the operator to lighten his or her hold of the master manipulator.

[0054] In 407, lightened hold of the master manipulator 108 by the operator is detected, for examples, by detecting the operator's release of a control activation button on the master manipulator 108, by a reading of a grip force sensor, or by detecting that the output of the Cartesian controller 307 has changed from F_CART to a smaller threshold value F_THREAS in response to a change in the readings of the master manipulator joint position sensors.

[0055] In 408, once it is determined that the operator has lightened his or her grip on the master manipulator 108, at least one parameter associated with the master manipulator 108 is set so that the slave manipulator 128 (and consequently, its tool 138) is commanded to a different state that does not exceed any of its state limitations. As an example, the programmable offset in the scale and offset processing unit 304 and its corresponding programmable offset in the inverse scale and offset processing unit 306 may be reprogrammed so that output of the Cartesian controller 307 becomes zero (from its current value that is already less than F_THREAS) After completing 408, the method then reengages control of the slave manipulator 128 by the master manipulator 108 by jumping back to 401 and reentering normal “following mode” for the master/slave control system 300. At this time, another message may be sent to the operator so that the operator knows that control of the slave manipulator 128 may now be resumed.

[0056] Although the various aspects of the present invention have been described with respect to a preferred embodiment, it will be understood that the invention is entitled to full protection within the full scope of the appended claims.