Endoscope comprising a system with multiple cameras for use in minimal-invasive surgery

[0001] This application claims priority to German Patent Application Serial No. DE 10 2012 025 102.5, filed Dec. 20, 2012, incorporated by reference herein in its entirety.

[0002] The present invention concerns an endoscope with a multi-camera system for use in minimally invasive surgeries and a respective surgical robot, especially for use in minimally invasive surgery, such as laparoscopy.

[0003] Minimally invasive surgeries, such as laparoscopic operations, are performed with the use of surgical instruments, for example, forceps, cutting and sewing tools, which are inserted into the body of a patient via one or several trocars. Usually, two to four, in most cases three surgical instruments are used. In addition to these instruments, it is required that a visualization unit is available which allows the surgeon to observe the area of operation. Such a visualization unit always comprises a camera or an endoscope, which are also inserted into the body of the patient via a trocar. Usually such visualization is performed by means of an endoscope which represents on an external monitor image of the area of operation in 2D or 3D. Prior art has provided numerous endoscopes in which a visualization unit, for example, a camera, has been integrated in the distal end. However, an endoscope can be provided with a camera either on its distal or on its proximal end. The images obtained by means of an endoscope are represented on one or several external monitors via an image transmission system and an image processing unit. Numerous endoscopes have been described in prior art.

[0004] The camera systems or endoscopes described in prior art have the disadvantage that even if two cameras are provided for taking images of the area of operation, it is not possible for these cameras to represent simultaneously in each constellation all surgical instruments due to the varying positions of the surgical instruments and the position of the endoscope near the surgical procedure, as well as the object field angle (FoV “field of view”), wherein merely the immediate area of the surgical procedure is represented. When a surgical instrument is removed from the surgical field of view, it is no longer captured by the camera or cameras and is no longer under the visual control of the surgeon or his assistant.

[0005] In one aspect, the invention provides an endoscope, which comprises

• • [0006] a main support device, which basically extends over the entire length of the endoscope from the outside to the interior of the body, and which comprises at its distal end at least one lighting unit and two imaging devices, wherein each of the imaging devices is basically arranged in such a way that it can be rotated to the outside on the same level as the main support device,
  • [0007] a trocar, by means of which the endoscope can enter the body, and
  • [0008] an additional support device, which is provided at the trocar and/or the main support device, wherein the additional support device comprises at its distal end an additional imaging device, which can be rotated from the additional support device to the outside and wherein the additional imaging device comprises an additional lighting unit and at least an additional image sensor, which comprises a monitoring area, which encompasses the two monitoring areas of the imaging devices of the main support device.

[0009] In one embodiment, the additional image sensor comprises a wide-angle lens which is located near the distal end of the trocar when pivoted. In another embodiment, each of the two imaging devices can be rotated about a rotation axis at the distal end of the main support device, wherein the rotation axes are aligned in a plane parallel to one another. In a further embodiment, the additional support device is located between the trocar and the main support device, especially resting directly at the main support device, wherein in particular the main support device and the additional support device have a cylindrical design. In another embodiment, each of the imaging devices is arranged in such a way that it can be tilted by means of joints about the rotation axis, as well as about a further pivot axis, in orthogonal direction toward the longitudinal extension of the support device, wherein the rotary movements about the rotation axes and the rotation axes are decoupled independently of one another.

[0010] In another aspect, the invention provides a surgical robot system having at least one robotic arm on which at least one surgical instrument and/or one endoscope can be arranged, which comprises

• • [0011] a main support device that basically extends over the entire length of the endoscope from the outside to the interior of the body, and which comprises at its distal end at least one lighting unit and two imaging devices, wherein each of the imaging devices is basically arranged in such a way that it can be rotated to the outside on the same level as the main support device,
  • [0012] a trocar, by means of which the endoscope can enter the body, and
  • [0013] an additional support device, which is provided at the trocar and/or the main support device, wherein the additional support device comprises at its distal end an additional imaging device, which can be rotated from the additional support device to the outside and wherein the additional imaging device comprises an additional lighting unit and at least an additional image sensor, which comprises a monitoring area, which encompasses the two monitoring areas of the imaging devices of the main support device, wherein provision has been made for an image processing unit, which is connected with the two imaging devices and the additional imaging device, and a visualization unit, which represents 2D image data and/or 3D image data of the imaging devices and/or the additional imaging device.

[0014] In one embodiment, the additional image sensor comprises a wide-angle lens which is located near the distal end of the trocar when pivoted. In another embodiment, the two imaging devices are arranged at the distal end of the main support device, respectively, in such a manner that they can be rotated about a rotation axis, wherein the rotation axes are aligned in a plane parallel to one another. In a further embodiment, the additional support device is located between the trocar and the main support device, especially resting directly at the main support device, wherein in particular the main support device and the additional support device have a cylindrical design. In another embodiment, each of the imaging devices is arranged in such a way that it can be tilted by means of joints about the rotation axis, as well as about a further rotation axis, in orthogonal direction toward the longitudinal extension of the support device, wherein the rotary movements about the rotation axes and the rotation axes are decoupled independently of one another.

[0020] The invention is therefore based on the objective of providing an improved visualization system for minimally invasive surgeries, such as laparoscopic surgeries, which allows the surgeon to coordinate the instruments in a simple manner using a single trocar to access the body without having to use an additional trocar to access the body. According to the present invention, this objective is achieved by means of the endoscopes and surgical robot systems disclosed herein.

[0021] The present invention provides an endoscope with a multi-camera system for use in minimally invasive surgery, such as laparoscopy.

[0022] A first object of the present invention concerns an endoscope for minimally invasive surgery, especially for use within a surgical robot system, which basically extends over the entire length of the endoscope from the outside to the interior of the body, and which comprises at its distal end at least one lighting unit and two imaging devices, wherein each of the imaging devices is basically arranged in such a way that it can be rotated to the outside on the same level as the main support device, a trocar by means of which the endoscope can enter the body, and an additional support device, which is provided at the trocar and/or the main support device, wherein the additional support device comprises at its distal end an additional imaging device, which can be rotated from the additional support device to the outside and wherein the additional imaging device comprises an additional lighting unit and at least an additional image sensor, which comprises a monitoring area, which encompasses the two monitoring areas of the imaging devices of the main support device.

[0023] The present invention has the advantage that by means of providing and simultaneously using 2 imaging systems, an at least 2D overview camera and a 3D detail camera, which are inserted into the body of a patient via a single trocar (also combination trocar), it is possible to generate an at least 2D overview image with a high object field angle (wide angle of typically >90°) and a 3D detail image with a general object field angle of up to 70°. As a result, it is possible to represent the immediate area of operation and its surrounding area during an entire minimally invasive surgery, such as laparoscopic surgery. In this way, it is possible to represent all surgical instruments simultaneously even if due to their varying positions and the position of the camera or the endoscope, as well as the object field angle (FoV “field of view”), they are outside of the surgical field of view of both imaging devices at the distal end of the endoscope, because the additional imaging device can also capture instruments which are outside of the surgical field of view of both imaging devices. For example, this can be the case when a surgical instrument is temporarily not needed and “deposited”. In most cases, such “depositing” takes place outside of the immediate surgical procedure and outside of the surgical field of view so that the instruments are not in the way during surgery. According to the invention, such “deposited” surgical elements are captured by the invention-based 2D overview camera and are therefore constantly under visual control of the surgeon or his assistant. Since the additional imaging device designed as a 2D overview camera and the imaging device designed as 3D detail camera are arranged respectively on an endoscope, for example, in the form of 2 image sensors, the surgeon has no problem to watch the picture recordings of the 2D overview camera and the 3D detail camera via a mutual or separate monitor. For the surgeon it is easy to coordinate the images because the monitoring area of the 2D overview camera, which possibly can comprise a 3D lens, encompasses the monitoring area of the 2D detail camera or is larger than the object field angle of the 3D detail camera. In this regard, it should be mentioned that an arrangement of 2 separate cameras, which are positioned completely independent and have an overlapping monitoring area not attached to the endoscope, would result in unfavorable coordination for the surgeon, possibly causing him to become “seasick” because of the two completely independent cameras. This problem is solved in a simple manner by means of the invention-based endoscope with the 2D overview camera and the adjusted 3D detail camera.

[0024] Furthermore, the lighting unit at the main support device together with the additional lighting unit at the additional support device improves the illumination of the 3D images, improving the quality of the representation of the images of the 3D detail camera.

[0025] According to a preferred embodiment of the invention, the additional image sensor comprises a wide-angle lens which is located near the distal end of the trocar when pivoted.

[0026] In particular, it is advantageous when each of the two imaging devices can be rotated about a rotation axis at the distal end of the main support device, wherein the rotation axes are aligned in a plane parallel to one another, thus minimizing the constructional expenses.

[0027] A further structural simplification involves that the additional support device is located between the trocar and the main support device, especially resting directly at the main support device, wherein in particular the main support device and the additional support device have a cylindrical design.

[0028] Furthermore, it is advantageous when each of the imaging devices is arranged in such a way that it can be tilted by means of joints about the rotation axis, as well as about a further rotation axis, in orthogonal direction toward the longitudinal extension of the support device, wherein the rotary movements about the rotation axes and the pivot axes are decoupled independently of one another.

[0029] Consequently, according to the present invention, it is possible when using only one trocar that the 3D detail camera can be moved in 4 degrees of freedom independent of the 2D overview camera. The 2D overview camera and the 3D detail camera, in turn, are connected with one another in 2 degrees of freedom, which represent the movements in x and y direction (pivot movement of the trocar) about the insertion point of the trocar into the body.

[0030] A second subject matter of the present invention concerns a surgical robot system having at least one robotic arm on which at least one surgical instrument and/or one endoscope for minimally invasive surgery can be arranged, which comprises a main support device that basically extends over the entire length of the endoscope from the outside to the interior of the body, and which comprises at its distal end at least one lighting unit and two imaging devices, wherein each of the imaging devices is basically arranged in such a way that it can be rotated to the outside on the same level as the main support device,

• • [0031] a trocar, by means of which the endoscope can enter the body, and
  • [0032] an additional support device, which is provided at the trocar and/or the main support device, wherein the additional support device comprises at its distal end an additional imaging device, which can be rotated from the additional support device to the outside and wherein the additional imaging device comprises an additional lighting unit and at least an additional image sensor, which comprises a monitoring area, which encompasses the two monitoring areas of the imaging devices of the main support device,
  • [0033] wherein provision has been made for an image processing unit, which is connected with the two imaging devices and the additional imaging device, and a visualization unit, which represents 2D image data and/or 3D image data of the imaging devices and/or the additional imaging device.

[0034] In particular, the invention-based surgical robot system has the advantage that depending on the needs the image data for the surgeon can be represented as 2D image data and/or 3D image data, i.e., the image data of the overview camera can be coupled with the image data of the 3D detail data by means of the image processing unit, providing the surgeon with a considerably improved overview through a single frame sequence on the visualization unit.

[0035] In particular, it is advantageous that the additional image sensor comprises a wide-angle lens which is located near the distal end of the trocar when pivoted.

[0036] The sub-claims show further advantageous embodiments of the invention. Accordingly, the entire disclosure of the present invention refers to the two camera systems, as well as the endoscope encompassing the two camera systems.

[0037] In minimally invasive surgery, such as laparoscopic surgery, the body of a patient is accessed via a trocar (usually through the abdominal wall or into the thorax). A surgical instrument or a camera or endoscope can be inserted into the body through such a trocar. As mentioned before, according to the invention, via a trocar two cameras are simultaneously inserted. A surgical procedure usually involves 2 to 4 surgical instruments and at least one camera. Therefore, 3 to 5 trocars are required for such a surgical procedure.

[0038] Subsequently, the present invention is described with reference to the figures:

[0039] FIG. 1 shows the invention-based multi camera system being integrated in an endoscope. The body of a patient is accessed by inserting a trocar 1 into the body tissue 2. Via a trocar 1 an additional support device 3 for a 2D overview camera is inserted into the body. The additional support device 3 is designed in such a way that a further rotationally symmetric rod-shaped main support device 4 for a 3D detail camera can be inserted into the tubular device. Alternatively, the tubular device can be used also for surgical instruments. A camera bracket 5 is attached by means of a joint 6 at the additional support device 3 in such a way that the camera bracket can be folded out through a pivot movement 7 basically by 90° to the rotation axis after being inserted into the trocar 1. The camera bracket 5 comprises an additional image sensor consisting of image sensor 9 and wide-angle lens-imaging optics 8. To illuminate the object field, the camera bracket 5 is also equipped with an additional lighting unit, consisting of a light source 11 and respective wide-angle imaging optics 10. These wide-angle imaging optics 10 are designed in such a way that the complete object field captured by the image sensor 9 and the connected wide-angle imaging optics 8 is illuminated. The camera bracket 5 with the additional image sensor and the additional lighting unit form the 2D overview camera for generating a 2D overview image. Preferably, the image sensor 9 is designed as CCD or CMOS sensor with a resolution of 1920×1080 pixels or higher.

[0040] The recorded image data are supplied via the data link 23 to a processing unit 25, which processes the image data for representation, and via a further data link 26 they are supplied to a visualization unit 27. The visualization unit 27 can represent 2D and 3D image data separately but also combined in a single image or a single frame sequence. A control unit 32 determines which image data is to be represented in what way, depending on the preferences of the surgeon.

[0041] At the end of the rotationally symmetric main support device 4 are two camera modules or two imaging devices 12a, 13a, 14a, 12b, 13b, 14b consisting in particular of 2 imaging optics 14a and 14b, respectively, mounted on two camera brackets 12a and 12b. Via the joints 17a and 17b which form the pivot axes, the camera brackets 12a and 12b are connected with the main support device 4 in such a way that they can be folded out by 90° to the rotation axis of the main support device 4 after being inserted into the body in swivel direction 18a or 18b. To illuminate the object field, a lighting unit consisting of light source 15 and imaging optics 16 is mounted at the end of the main support device 4 on which also the foldable camera brackets are attached. Furthermore, the camera brackets 12a and 12b comprise image receptors consisting of image sensors 13a and 13b and imaging optics 14a and 14b. Together these two imaging devices 12a, 13a, 14a, 12b, 13b, 14b form the 3D detail camera.

[0042] Preferably, the lighting unit consisting of light source 15 and imaging optics 16 can be designed as LED light source in the way that the radiation angle of the LED combined with appropriate imaging optics 16 is selected in such a way that the object field represented by both image sensors 13a and 13b and the associated imaging optics 14a and 14b is completely illuminated. In a modified embodiment the lighting unit at the proximal end can also comprise only imaging optics 16. In this case, the light source 30 is arranged outside the main support device 4 and therefore outside of the patient. The control commands to the light source 30 are transmitted from the processing unit 25 via the data link 31. Then the light is fed via a light conductor to an appropriate branch circuit mechanism 28. Preferably, said branch circuit mechanism 28 is designed, for example, as optical fiber bundle and conducts the light to the imaging optics 16. In a further embodiment, the branch circuit mechanism 28 can be implemented also by means of appropriate rod optics.

[0043] In a further embodiment (FIG. 3), a camera bracket 12c is located at the end of the rotationally symmetric main support device 4. Said camera bracket comprises both imaging devices 13a, 14a, 13b, 14b, which in particular consist of 2 imaging optics 14a and 14b, respectively, and mounted on a camera bracket 12c. In this arrangement, the light source 15 connected with the imaging optics 16 is also mounted on the camera bracket 12c in the center between the two imaging devices 13a, 14a, 13b, 14b. Via a joint 17c, which forms the swivel axis, the camera bracket 12c is connected with the main support device 4 in such a way that the camera bracket 12c can be folded out in a pivot movement 18c by 90° to the rotation axis of the main support device 4 after being inserted in the body.

[0044] In addition, the joint 17c is designed in such a way that the camera bracket 12c can be pivoted in the direction 20 by at least +/−90° about the pivot axis 19 located orthogonally to the rotation axis 21 of the main support device 4.

[0045] The recorded image data are supplied via the data link 24 to a processing unit 25, which processes the image data of the 3D detail camera for stereo representation, and via the data link 26 they are supplied to a visualization unit 27. The visualization unit 27 can represent 2D and 3D image data. A processing unit 25 controls which image data is to be represented in what way.

[0046] FIG. 2 shows the folding out movements of the 3D detail camera with regard to the rotation and pivot axis. The joints 17a and 17b (see FIG. 1) allow the camera brackets 12a and 12b to be folded out (see FIG. 1) by 90°, starting from the rotation axis 21 of the main support device 4.

[0047] Furthermore, the joints 17a and 17b can allow for a synchronized pivot movement or tilting by approximately +/−90° of the camera brackets 12a and 12b about the pivot axis 19 orthogonally to the rotation axis of the main support device 4. In this way, without changing the position of the main support device 4, it is possible to record respective 3D images angular to the rotation axis 21 of the main support device 4.

[0048] This invention-based synchronous pivot movement of the camera brackets 12a and 12b about the pivot axis 19 orthogonally to the rotation axis of the main support device 4 has advantages in comparison to structures of endoscopes known from prior art in which the rotation axis and the optical axis are identical, i.e., a view “with an upward inclination” or “downward inclination” requires the endoscope to be tilted by the respective angle which, in turn, requires sufficient space for movement. This can put a strain on the patient's tissue and even injure the patient. Because of the possibility of pivoting the camera bracket about the pivot axis 19, the invention-based endoscope does not have to be tilted in the customary manner. Alternatively, endoscopes known from prior art use different lenses with rigid angles, which differ from 0° and typically comprise 30°. To change the lenses, the surgeon has to interrupt the surgery, remove the endoscope lens, connect a different lens with the endoscope and again insert the endoscope into the patient by way of the endoscope trocar.

[0049] Furthermore, the joints 17a and 17b allow for an independent, decoupled pivot movement by basically +/−90°, respectively, of the camera brackets 12a and 12b about the pivot axis 19 orthogonally to the rotation axis 21 of the main support device 4. In this way, without changing the position of the main support device 4, it is possible to record respective 2D images over a larger object field angle. For this purpose, after being transmitted via the data link 24 (see FIG. 1) to a processing unit 25 (see FIG. 1) the two 2D images are composed in such a way that it is possible to represent on the visualization unit 27 (see FIG. 1) a 2D image over a larger object field angle, which is not possible when using customary endoscopes. If the visualization unit 27 is also suitable for representing 3D images, the processing unit 25 can also calculate a 3D image, which is then represented to the surgeon on the visualization unit 27 as a three-dimensional image. The surgeon is able to comprehend such a three-dimensional image on the visualization unit 27 by means of an optical aid, for example, shutter glasses or polarization glasses. Alternatively, the visualization unit 27 by means of an optical imaging system is designed in such a way that it projects a left image for the left eye and a right image for the right eye, respectively. In this embodiment, it is not required to use additional optical aids, for example, shutter glasses or polarization glasses.

[0050] In the alternative embodiment according to FIG. 3, the camera bracket 12c comprises both imaging devices 13a, 14a, 13b, 14b, as well as the lighting unit consisting of light source 15 and imaging optics 16. Consequently, pivoting the joint 17c of the camera bracket basically by +/−90° about the pivot axis 19 located orthogonally to the rotation axis 21 of the main support device 4 has an effect on both imaging devices 13a, 14a, 13b, 14b, as well as the lighting unit consisting of light source 15 and imaging optics 16. In this embodiment it is advantageous that the imaging optics 16 can be optimally adapted to the object field angle recorded by the imaging devices 13a, 14a, 13b, 14b, because in this embodiment the lighting unit is pivoting together with the imaging devices. In this embodiment, the light source 15 is preferably designed as LED illumination.

[0051] In the processing unit 25, the position and trajectories of the inserted instruments can be calculated from the data of the 2D overview camera. This information of the trajectories can be shown as additional information when representing in an appropriate manner, for example, as overlay representation, the 3D image on the visualization unit 27.

[0052] FIG. 4 shows the exemplary use of the invention-based multi-camera system 43 in a telemanipulator or robot system. The surgeon controls the actuators via a display or control unit 32. The control commands generated by means of the display and control unit 32 are transmitted to a control unit 34 via data transmission 33. Said control unit 34 is connected with the robot system 37 via a further data line 35 and, equipped with a support shaft 40, bowing 42 can be pre-positioned via joint mechanics according to the position of the patient on the OR table 38 in such a way that the robotic arm 44 allows for an optimal position of the multi-camera system 43. The image data recorded by the 2D overview camera are supplied via the data link 23 to a processing unit 25, which processes the image data, and via a further data link 26 they are supplied to a visualization unit 27. The visualization unit 27 can represent 2D and 3D image data, for example separately, but also combined in a single image or a single frame sequence. A control unit 32 determines which image data is to be represented in what way, depending on the preferences of the surgeon. The control commands generated by the control unit 32 are transmitted by means of the data link 39 to the processing unit 25.

[0053] According to FIG. 1, two images from different positions of a scene are recorded by two imaging devices 12a, 13a, 14a, 12b, 13b, 14b, which in particular consist of 2 imaging optics 14a and 14b, respectively, and which are mounted on 2 camera brackets 12a and 12b. The recorded image data are supplied via the data link 24 to a processing unit 25, which processes the image data of the 3D detail camera for stereo representation, and via the data link 26 they are supplied to a visualization unit 27. The visualization unit 27 can represent 2D and 3D image data. A control unit 32 determines which image data is to be represented in what way, depending on the preferences of the surgeon. The control commands generated by the control unit 32 are transmitted by means of the data link 39 to the processing unit 25.

[0054] Minimally invasive surgery, such as laparoscopic surgery, is often performed by means of surgical manipulators, telemanipulators or robot systems. The invention-based endoscopes and cameras can be used in such telemanipulators or robot systems for minimally invasive surgery, especially for use within a surgical robot system

[0055] The present invention is not limited to the fact that an invention-based endoscope or invention-based camera is used in a telemanipulator or robot system in order to perform minimally invasive surgery. It can also be used in the medical field apart from such systems.