Z sharpening for fibre confocal microscopes

[0002] The present invention relates to an apparatus and method for performing image sharpening in confocal microscopy, of—in particular—the Z axis of confocal data sets in real time with a single scan. The present invention is of particular but by no means exclusive application in increasing the density of information storage of optical date storage devices, particularly of three dimensional digital data store devices.

[0003] In confocal microscopy it is generally desirable to minimise the thickness of the focal plane section. This is achieved by reducing the return pinhole to the smallest size which will give a reasonable signal.

[0004] With a 1.4 NA oil immersion objective lens, the XY resolution is approximately 200 μm while the Z resolution is approximately 500 μm. This means that the voxels or boxels making up the image have a long axis that is 2.5 times the two orthogonal voxel dimensions. This is true for all laser scanning confocal microscopes (LSCMs) and affects all 3D reconstructions.

[0005] This ratio is greater for lower NA lenses and the result has a deleterious effect in 3D reconstructions. Rotations of images show a lack of resolution in the Z direction and perhaps more seriously, an artefact in which give a perception an anisotropy in views of tissues which include a Z dimension.

[0006] Image processing software can be used to improve the image. For example, in a first existing technique, Z sharpness is increased by concentrating on a voxel and then deconvolving it to a sharper value by subtracting from it a small proportion of the value of the voxel above it and below it.

[0007] A second existing technique utilises a similar principle in conjunction with XY sharpening algorithms. This is actually marketed as a synthetic aperture confocal system which can deconvolve sharp pictures from successive depth blurred low contrast brightfield images. However, it has been suggested that confocal data sets would also benefit from this approach. More sophisticated correction takes into account the brightness of pixels two levels above and below the focal plane being Z sharpened.

[0008] These are in effect a digital versions of unsharp masking techniques by means of which a correction is provided for the brightness of each individual voxel, which takes into account the brightness and ‘spillover’ addition of light from voxels above and below. The successful use of the aforementioned methods also depends on the operator having a fairly good understanding of the nature of the sample, the lens characteristics, the pixel sampling interval, the distance between successive image planes and other factors and entering these into the variables and chart of the algorithm.

[0009] A third existing technique that effectively achieves an identical Z sharpening result involves carrying out two separate scans of each plane, one scan being with the pinhole stopped right down and a second scan with the pinhole opened to about double the XY resolution optimum size. The second scan includes light from fluorescence from objects in the adjacent planes above and below and gives an analog sum of light intensities which can be used to obtain a correction factor equivalent to the digital correction algorithm used in the technique described above (in which one concentrates on a voxel and then deconvolves it to a sharper value by subtracting from it a small proportion of the value of the voxel above it and below it).

[0010] However, the above methods are time consuming and require a knowledge of the lens characteristics and sampling intervals. They require more than one scan to be made together with post acquisition processing. The software deconvolution (which is effectively digital unsharp masking) requires 3 or 5 scan depths to obtain corrections for 1 and 2 planes above and below the plane to be sharpened and, in some techniques, 2 or 3 scans with 2 or 3 different pinhole sizes.

[0011] Similarly, many methods have been proposed for high density digital storage using optically addressable elements within the three dimensional structure. Typical of these is the work by Rentzepis and by Min Gu. Previously proposed methods use confocal techniques to address the individual bit storage elements. The resolution in XY and Z of these methods has pretty thoroughly been established by Sheppard, Gu and others.

[0012] FIG. 1 illustrates the formation of a Gaussian Waist 10 when a TEM₀₀beam 12 comprising a set of plane parallel wavefronts 13 from a laser 14 passes through a beamsplitter 16 (in which the first reflection is omitted for clarity) and objective lens 18. The lens 18 produces a convergent concentric wave front 20. If the Gaussian Waist 10 is focussed in a uniform fluorescent medium (not shown) then the points of re-emission of light which will return more than a given percentage of the excitation light energy through the return pinhole 22, after reflection and re-direction by beamsplitter 16 and focussing by lens 23, will constitute a volume 24 which is roughly football or elliptically shaped, symmetrically located in the waist 10. This elliptical volume 24 could be termed an isofluorescence boundary for confocal pinhole return. In fact for a perfect lens the ‘football’ has two haloes above and below it (not shown). These do not affect the discussion and have been omitted for clarity. The 1/e²Gaussian profile is also indicated in this figure, as is the region 28 shown in subsequent figures and encompassing the Gaussian Waist 10 and environs.

[0013] Clearly the principle of unsharp masking involves the subtraction of return light from just above and just below the pixel to be sharpened in which the ‘overlap’ return light is taken away from the central pixel.

[0014] Two such prior art techniques (such as those employed in the first and second existing techniques discussed above respectively) are illustrated in FIGS. 2A and 2B, in which all the boxels are the same size. The pinhole is not altered but the ‘overlap’ required for the unsharp masking is obtained from the pixels in the scans on either side. FIG. 2A illustrates a prior art digital image sharpening technique using three scans at three separate levels within a specimen. In FIG. 2A, the plane to be sharpened is indicated at 30, and cross sections of the Gaussian Waist and confocal volume (or isofluorescence intensity voxel perimeter) for each of three scans are shown at 32, 34 and 36; the Gaussian Waist and confocal volume are respectively on, above and below the desired focal plane. In the sharpening procedure (see schematic representation at 38), a portion of both dotted volumes 40 and 42 (corresponding to the confocal volumes of the second and third scans 34 and 36) are removed from the central volume 44 (corresponding to the confocal volume of the first scan 32), leaving a sharpened voxel 46.

[0015] In the prior art technique illustrated in FIG. 2B, the central voxel 50 is sharpened by removing a portion of a 3×3 voxel matrix 52 from above and another 3×3 voxel matrix 54 from below the desired focal plane. The schematic image of FIG. 2B is shown undersampled from the Nyquist point of view to increase clarity.

[0016] FIG. 3 illustrates the traditional unsharp masking of the third existing technique discussed above, in which—after a first scan 60 is made with the pinhole stopped down—a second scan 62 is made with the pinhole opened but at the same focal plane. Next the pixel values for the image produced in the second scan 62 are subtracted—where an overlap exists—from the image produced in the first scan 60 (with the pinhole stopped down); the resulting difference signal contains the ‘overlap’ information 64 and is used to correct each of the pixels to be sharpened to produce the sharpened voxel 66.

[0017] It is an object of the present invention, therefore, to provide a method that avoids the necessity for multiple scanning and post acquisition processing.

[0018] It is another object of the present invention to provide a method and apparatus for reducing the Bit Error Rate (BER) of reading and of increasing the storage capacity (typically measured in gigabits per cubic millimeter) of a data storage material.

[0019] In a first broad aspect, therefore, the present invention provides a method of image sharpening in a confocal microscopy or endoscopy observation, comprising:

[0020] collecting true confocal return light emanating from an observational field of an object;

[0021] focussing said true confocal return light into a core of a fiber wave-guide;

[0022] collecting near confocal return light from a volume partially overlapping said observational field and thereby defining an overlap volume;

[0023] focussing said near confocal return light into said fiber wave-guide so as to be transmitted principally in a cladding of said fiber wave-guide;

[0024] separately detecting said true confocal return light and said near confocal return light to produce a true confocal output signal and a near confocal output signal; and

[0025] adjusting said true confocal output signal on the basis of said near confocal output signal to substantially eliminate from said true confocal output signal a component due to said near confocal output signal;

[0026] whereby the effective volume of said observational field is reduced and the resolution of said observation is effectively increased.

[0027] Preferably said overlap volume is in the Z axis of said observational field.

[0028] Preferably said true confocal return light and said near confocal return light are collected and focussed by means of a light condenser. More preferably said light condenser comprises a lens or a compound lens.

[0029] In one embodiment, adjusting said true confocal output signal comprises subtracting said near confocal output signal from said true confocal output signal.

[0030] This may constitute an over-correction, but the component of the near confocal signal due to light from other than the overlap volume will be small compared to the component of the near confocal signal due to light from the overlap volume, so the adjustment of the true confocal output signal will nevertheless improve, overall, the resolution of the observation.

[0031] More preferably said method includes absorbing or otherwise excluding higher angle rays from said near confocal return light, whereby said near confocal return light comprises principally light from said overlap volume.

[0032] Preferably said method includes excluding higher angle rays from said near confocal return light by transmitting said near confocal return light through a region of said fiber provided with an outer cladding with a refractive index such that said higher angle rays are transmitted into said outer cladding while lower angle rays of said near confocal light are internally reflected and thereby retained in a glass inner cladding of said fiber.

[0033] Preferably said method includes absorbing light transmitted within said outer cladding.

[0034] In one embodiment, said fiber is a single moded optic fiber with a glass inner cladding and an outer cladding having a low refractive index such that modes of said near confocal return light in said glass cladding are normally internally reflected by said outer cladding, wherein said method includes cooling said outer cladding within a region of said fiber so that within said region said higher angle rays are transmitted into said outer cladding. More preferably said outer cladding comprises silicone rubber.

[0035] Preferably said outer cladding is surrounded at least partially within said region with an optically absorbing medium.

[0036] Preferably said cooling is by means of a Peltier effect cooler.

[0037] In one embodiment, said object is a data storage medium.

[0038] In a second broad aspect, the present invention provides an image sharpening apparatus for use in making a confocal microscopy or endoscopy observation, comprising:

[0039] a light condenser for collecting true confocal return light emanating from an observational field of an object, for focussing said true confocal return light into a core of a fiber wave-guide, for collecting near confocal return light from a volume partially overlapping said observational field and thereby defining an overlap volume, and for focussing said near confocal return light into said fiber wave-guide so as to be transmitted principally in a cladding of said fiber wave-guide;

[0040] detection means for detecting said true confocal return light and said near confocal return light, and to produce respectively a true confocal output signal and a near confocal output signal; and

[0041] signal processing means for adjusting said true confocal output signal on the basis of said near confocal output signal to substantially eliminate from said true confocal output signal a component due to said near confocal output signal;

[0042] whereby the effective volume of said observational field is reduced and the resolution of said observation is effectively increased.

[0043] Preferably said overlap volume is in the Z axis of said observational field.

[0044] Preferably said light condenser comprises a first light condenser and a second light condenser, wherein said first light condenser is arranged to collect and focus said true confocal return light and a second light condenser is arranged to collect and focus said near confocal return light.

[0045] More preferably said light condenser comprises a lens or a compound lens.

[0046] Preferably said detection means comprises a first detector and a second detector, wherein said first detector is arranged to detect said true confocal return light and said second detector is arranged to detect said near confocal return light.

[0047] In one embodiment, said signal processing means is operable to adjust said true confocal output signal by subtracting said near confocal output signal from said true confocal output signal.

[0048] Preferably said apparatus includes absorption means for extracting and absorbing higher angle rays from said near confocal return light, whereby said near confocal return light comprises principally light from said overlap volume.

[0049] Preferably said fiber has an glass inner cladding and a region provided with an outer cladding with a refractive index such that within said region higher angle rays of said near confocal return light are transmitted into said outer cladding while lower angle rays of said near confocal light are internally reflected and thereby retained in said glass cladding.

[0050] In one embodiment, said fiber is a single moded optic fiber with a glass inner cladding and an outer cladding having a low refractive index such that modes of said near confocal return light in said glass cladding are normally internally reflected by said outer cladding, wherein said apparatus includes means for increasing said refractive index of said outer cladding within a region of said fiber so that within said region said higher angle rays are transmitted into said outer cladding. More preferably said outer cladding comprises silicone rubber.

[0051] Preferably said outer cladding is surrounded at least partially within said region with an optically absorbing medium.

[0052] Preferably said means for increasing said refractive index of said outer cladding within a region comprises a cooling means, and more preferably a Peltier effect cooler.

[0053] Preferably said apparatus includes optical path varying means for varying the optical path of said true and near confocal return light to compensate for variations in said optical path due to changes in the depth of said observational field within said object, said optical path varying means having regions of greater and lesser optical path, whereby said optical path varying means can be located with a region of lesser optical path in said optical path when said observational field is deep within said object and with a region of greater optical path in said optical path when said observational field is less deep within said object.

[0054] Preferably said optical path varying means comprises an optical wedge.

[0055] In a third broad aspect, the present invention provides a data reading apparatus for reading data from a data storage medium, including the image sharpening apparatus described above.

[0056] In order that the present invention may be more clearly ascertained, a preferred embodiment will now be described, by way of example, with reference to the accompanying drawing, in which:

[0057] FIG. 1 illustrates the formation of a Gaussian Waist;

[0058] FIG. 2A is a schematic view of a prior art Z sharpening technique in which each voxel is sharpened by subtracting from it a small proportion of the value of the voxel above it and below it;

[0059] FIG. 2B is a schematic view of a prior art Z sharpening technique similar to that shown in FIG. 2B, but in conjunction with XY sharpening algorithms;

[0060] FIG. 3 is a schematic view of a prior art Z sharpening technique that employs a traditional unsharp masking technique;

[0061] FIG. 4 is a schematic view illustrating a preferred embodiment of the present invention;

[0062] FIG. 5 illustrates the relative locations of the Gaussian Waist, the core and near confocal fluorescences, and the sharpened voxel according to a preferred embodiment of the present invention;.

[0063] FIG. 6 is a schematic view of a Z sharpening confocal microscope according to a preferred embodiment of the present invention;

[0064] FIG. 7 is a schematic view of a Z sharpening confocal microscope according to a further preferred embodiment of the present invention;

[0065] FIG. 8 is a schematic view of a confocal microscope without Z sharpening, modified from the apparatus of FIGS. 6 or 7;

[0066] FIG. 9 is a schematic representation of a data reading apparatus according to another preferred embodiment of the present invention;

[0067] FIG. 10 is a schematic representation of another data reading apparatus according to a further preferred embodiment of the present invention; and

[0068] FIG. 11 is a schematic view of an apparatus for use with the apparatus of FIG. 9 or FIG. 10, for compensating for spherical aberration on focussing into an optical data storage medium.

[0069] Referring to FIG. 4, in a Z sharpening method according to a preferred embodiment of the present invention, the near confocal fluorescence or reflection is conveyed back simultaneously via separate coded channels with the one fiber to photodetectors and the intensity converted to an electric signal. The voltage produced from these other channels can be convolved with the confocal return channel voltage and an on-line single scan sharpened image can be generated. FIG. 4 illustrates the volumes around the Gaussian Waist from which confocal and near confocal fluorescence originates and how the overlap can be used for voxel sharpening. In this figure are shown the Gaussian Waist 70, the fluorescence 72 that goes back into the core of the fiber (not shown), the fluorescence 74a,bthat goes into the near confocal channel, and the sharpened voxel 76 (i.e. where there is no overlap between the ‘true’ confocal fluorescence 72 and near confocal fluorescence 74a,b).

[0070] FIG. 5 is a schematic view of an apparatus according to an embodiment of the present invention (though for simplicity using a long path variable pinhole). In this embodiment the light 80 from laser 82 passes through beamsplitter 84 and is focussed by lens 86 into the core 98 at the tip 88 of a silicone rubber clad single moded optic fiber 90. The cladding modes in the glass cladding 92 are internally reflected by the low refractive index silicone rubber 94 but are absorbed by cladding mode stripper 96. The light is transmitted along the core 98 of the fiber 90 to the end 100 of the fiber 90, from which it emerges as a beam 102 of low NA which is reflected by lens 104 to a beam 106 which is intercepted by lens 108 and focussed through a coverslip 110 to a spot (or Gaussian Waist) 112 within a specimen in the form of a cell 114 on a slide 116. The light re-emanating from “focal volume” of the Gaussian Waist 112 returning through lens 108 retraverses the same optical path as the excitation light 106. It is reconverged by lens 104 to enter back into the core 98 of the optic fiber 90 at the end 100 and retraverses the core 98 to the other end 88 at which point it remerges and is collimated by lens 86.

[0071] Passing to the beamsplitter 84 the beam is reflected by the dichroic or semireflective layer 118 to emerge as a beam 120 which impinges on photomultiplier 122. The electrical signal from this photomultiplier tube 122 provides the ‘true’ confocal data set bitmap information. Fluorescence which is generated above 124 or below the focal plane arrives back at lens 104 with a degree of lateral displacement 126 which is proportional to the distance of the light returning body from the focal plane (Gaussian Waist 112). Because of this lateral displacement it is converged to the core 98 at the tip 100 of the fiber 90 at angles which are unable to be carried as bound modes within the core 98. The light is therefore carried as cladding mode rays 128 and 130.

[0072] Ray 128 came from a plane which was closer to the Gaussian Waist 112 than ray 130. Ray 128 is therefore carried as a cladding mode of lower angle than ray 130. A Peltier effect cooler 132 reduces the temperature of an optically absorbing medium 134 surrounding the fiber 90 and also cools the silicone rubber and the glass in that vicinity 136. The silicone rubber has its refractive index increased by the cooling (relative to the silica) and the critical angle at the interface 138 will no longer guide the higher angle ray. Ray 130 therefore passes through the silicone rubber and is absorbed at 140. Ray 128 which is carried by the fiber 90 at a shallower glancing angle passes through area 136 and proceeds to area 142 where the fiber 90 is cooled by a second Peltier effect device 144, operated at a temperature that is lower than that of the first Peltier device 132.

[0073] This then allows the light to escape from the glass into the silicone rubber at 146 where it can proceed into a second fiber 148. The light then proceeds along this fiber to the tip 150 from which it emerges and impinges on the photosensitive surface of a photomultiplier tube 152. It is desirable that all cladding mode light is extracted and directed to this photomultiplier tube 152. The electrical output from photomultiplier tube 152 quantifies the intensity of this light. The two temperatures of the Peltier effect devices 132 and 144 can be varied to control the mode fraction which passes to photomultiplier tube 152.

[0074] FIG. 6 illustrates a variation of the embodiment of FIG. 5, in which two photomultipliers 160 and 162 are used in conjunction with three Peltier Coolers 164, 166 and 168. The temperature of the Peltier coolers is T(P164)<T(P166)<T(P168). The principle of operation is similar to the embodiment of FIG. 5 except that light from two separate successive planes above and below the focal plane is sampled by the two photomultiplier tubes 160 and 162, as follows.

[0075] FIG. 7 indicates the areas (representing volumes) within the Gaussian Waist 170 from which the sets of modes going to the photomultiplier tubes 160 and 162 of the embodiment of FIG. 6 are derived. Photomultiplier tube 160 derives light from the areas 174a,bcorresponding to the lowest order modes, while photomultiplier tube 162 derives light from the areas 174a,bcorresponding to the next set of modes. In carrying out the analog computation a portion of the output from photomultiplier tube 160 is subtracted from the ‘true’ confocal signal from area 176 (from photomultiplier tube 122), which effectively removes the signal from the overlap areas 178a,band thereby reduces the depth of field of the image obtained. A smaller fraction of the output of photomultiplier tube 162 is added to the ‘true’ confocal signal from area 176, to sharpen the overlap areas 180a,b. The temperatures of the three Peltier effect coolers 164, 166 and 168 are optimized to give a best sharpening effect.

[0076] FIG. 8 illustrates the change which would be made to convert the system into a conventional variable pinhole LSCM. The cladding mode coupler fiber 182 is relocated from photomultiplier tube 160 and its output is allowed to fall on photomultiplier tube 122. Electrical output addition of the two photomultiplier signals is also possible.

[0077] The above approach can also be employed to provide a method and apparatus by again utilising the return light from the regions which are slightly out of focus of a reading lens. According to the method of this embodiment, a part of the signal from these slightly out of focus regions is convolved or subtracted from the signal which has been generated by the confocal return light. The separation of the “near confocal” return light from the “true” confocal return is achieved using modes in an optical fiber.

[0078] FIG. 9 is a schematic representation of a data reading apparatus according to this embodiment of the invention, in which laser 210 generates laser beam 212, which is focussed by lens 214 into single mode optic fiber 216. The light 212 exits fiber 216, is collimated by lens 218 and directed into beam splitter 220. One portion of the light 212 is focussed by lens 222 onto digital data storage medium 224; the interrogated spot lies at focal point 226.

[0079] Return light from a digital data storage medium 224 is directed by beam splitter 220 to mirror 228, which directs the light through beam compressor 230 (including lenses 232aand 232a). Both true confocal rays 234 and near confocal rays 236 emerging from the beam compressor 230 are focussed by lens 238 into silica multimode fiber 242, encased in low refractive index silicone cladding 240. Towards the end of fiber 242, higher (relative to cladding 240) refractive index cladding 244 strips out higher order modes; the remainder of the light (confocal return light 246) is detected by photodetector 248, which the higher order modes (near confocal return light 250) is detected by photodetector 252.

[0080] Note that this method can operate in reflected light or in fluorescence mode, and in single photon (linear) mode or in non linear (multiphoton molecular fluorescence) mode. The method can be used with the one fiber delivering the laser light and returning the signal or with a separate fiber returning the signal (see FIG. 9). It can be made to operated in reflection (the preferred mode, as depicted in FIG. 9) or in transmission.

[0081] FIG. 10 depicts an alternative embodiment, which can be used to obtain “unsharp masking” data density improvement. In this figure, laser light from blue laser diode 260 is collimated by lens 262 into beam splitter 264. A portion of the light is focussed by lens 266 onto data storage medium 268 (as above). Return light is directed by beam splitter 264 through beam compressor 270; then, both true confocal light 272 and near confocal light 274 are focussed by lens 276 into silica fiber 278. Again, confocal return light exits fiber 278 and is detected by photodetector 282, while higher modes are stripped out by higher refractive index cladding 280 and detected by photodetector 284. In this way, “unsharp masking” data density improvement can be obtained by subtracting a part of the output of photodetector 284 from the output of photodetector 282.

[0082] FIG. 11 depicts an apparatus for use with the apparatuses of FIGS. 9 and 10, for compensating for spherical aberration on focussing into the optical data storage medium or material. In this figure, light from optical data storage medium 290 passes through glass optical wedge 292; the optical data is then “read” by lens 294. As the focussed “reading” spot 296 moves deeper into the medium 290, the wedge 292 is moved to the left (in this view) so that a thinner portion of wedge 292 is in the beam path, thereby keeping the optical path effectively constant.

[0083] Modifications within the spirit and scope of the invention may readily be effected by persons skilled in the art, so it is to be understood that this invention is not limited to the particular embodiments described by way of example hereinabove. For example, it should particularly be noted that any reference to microscopy or to endoscopy is intended also to refer to endomicroscopy.