Spatially Light Modulated Confocal Microscope and Method

This invention relates generally to confocal microscopes and more particularly to a spatially light modulated confocal microscope and method which employs multiple pattern illumination.

The basic principle that distinguishes confocal microscopy from other forms of light microscopy is that discrete aperture spots are illuminated in the object plane of the microscope from which reflected or fluorescent light is then relayed for observation through conjugate apertures in the image plane. For diffraction limited spots, this principle results in spacial resolution 1.32 times better than the optimum resolution obtainable by conventional light microscopy. Furthermore, this method dramatically reduces the interference of stray, out-of-focus light from an observed specimen above or below the focal plane. This technique actually permits optical sectioning of living tissue (with depth restrictions) and high-resolution 3-D reconstruction with automated focussing controls. It is the only microscopic technique that can effectively resolve individual cells in living tissue without staining.

Presently known confocal microscopes employ a sequential scanning technique which involves scanning each illumination and image conjugate aperture pair one at a time until the object (and image) has been covered sufficiently to produce an adequate confocal image. A basic form of the confocal microscope is disclosed in U.S. Pat. No. 3,013,467 to Minsky.

Scanning technologies which have been developed for known confocal microscopes include a mechanical translation of the specimen with fixed optics, but scanning has also been accomplished using a fixed specimen and scanning beams manipulated by special rotating aperture disks. The use of such rotating disks is disclosed in U.S. Pat. Nos. 4,802,748 to McCarthy et al., 5,067,805 to Corle et al., 5,099,363 to Lichtman and 5,162,941 to Favro et al. Although these disks include a plurality of apertures, only one aperture at a time is used for scanning. Other known confocal scanning systems have included a laser beam rastered with rotating mirrors to scan a specimen, or a laser beam which scans a slit rather than a spot. Such slit scanning increases imaging speed but slightly degrades resolution.

Conventional confocal microscopes are subject to a number of limitations imposed by the scanning techniques employed. In some cases, confocal image acquisition is too slow for certain applications and becomes slower as the scan line density increases and as aperture separation decreases. Furthermore, these parameters are preset and are not practically adjustable in most commercial systems. Signal-to-noise ratio (SNR) must be severely sacrificed to increase imaging rate. Trying to improve SNR by increasing illumination intensity is often limited by the susceptibility of live cells to photodamage or rapid fluorophore photobleaching.

For most conventional confocal systems, proper alignment is critical and may be difficult to maintain. The same physical aperture must be used for illumination and detection thereby limiting this technique to the study of reflection and epifluorescence. Therefore, transmitted light confocal imaging through a translucent slice (as for many voltage-sensitive dyes) is not practically possible.

Finally, it is often difficult or impossible for the user to restrict illumination and imaging to any selected subfield and differentially control light exposure.

Laser scanning systems have become widely accepted because of their higher intensity and diffraction limited spot size which results in greater resolution in the confocal image. However, they are more expensive than white-light systems and do not offer the selection of illumination wavelengths needed for the growing varieties of fluorescence indicators now available. Furthermore, laser intensity often leads to problems with phytotoxicity and rapid photobleaching.

It is a primary object of the present invention to provide a novel and improved spatially light modulated confocal microscope which employs multiple pattern illumination to provide conjugate aperture patterns in the image plane.

Another object of the present invention is to provide a novel and improved method for confocal microscopy which includes creating complementary patterns of illumination spots and imaging these spots on a specimen.

A further object of the present invention is to provide a novel and improved method for confocal microscopy which includes creating a plurality of complementary patterns of illumination spots and superimposing these patterns sequentially on a specimen.

Yet another object of the present invention is to provide a novel and improved confocal light modulation system which can be conveniently combined with almost any conventional research microscope to convert the research microscope to a confocal microscope having unique capabilities.

A further object of the present invention is to provide a novel and improved spatially light modulated confocal microscope designed to image illuminate selected subsets of an entire cell field thereby enhancing image rate, signal-to-noise ratio, and light collection. The device will include user selectable program modules so that a user can fully control the optical transfer function of the confocal microscope in coordination with particular optical characteristics of a specimen.

Still another object of the present invention is to provide a novel and improved spatially light modulated confocal microscope wherein selected image subsets and entire field-by-field image acquisition can be completely controlled by an image processing system.

Yet another object of the present invention is to provide a novel and improved spatially light modulated confocal microscope which provides external gating and control of illumination and image acquisition giving the user the selection of illumination and imaging from partial view fields. White light, other non-coherent light sources or laser illumination may be selected and variable speed imaging is provided. Also, a choice may be made of reflection, fluorescence or transmission confocal imaging.

A further object of the present invention is to provide a novel and improved spatially light modulated confocal microscope which provides confocal illumination and imaging from user selected subsets of an object-plane down to the level of single-spaced confocal measurements.

Another object of the present invention is to provide a novel and improved spatially light modulated confocal microscope adapted for software-mediated tuning of the microscope optical transfer function to optical specimen characteristics to provide novel forms of contrast enhancement.

A further object of the present invention is to provide a novel and improved spatially light modulated confocal microscope with no moving parts to create vibration which employs advanced analog or digital spatial light modulators to provide user-selectable imaging rates faster than one thousand frames per second while maintaining a sufficient signal to noise ratio for high quality imaging.

A still further object of the present invention is to provide a novel and improved spatially light modulated confocal microscope wherein collimated light is patterned through a pixel shutter array having a high pixel density with patterned gating ability to achieve multiple pattern illumination of a specimen. Detection is accomplished with a 2-D detector and conjugate image masking.

FIG. 1 is a basic optical schematic of a spatially light modulated confocal microscope embodying the present invention;

FIG. 2 is a perspective view of a portion of an electrostatic shutter array for use with the confocal microscope of FIG. 1;

FIG. 3 is a cutaway view of a single pixel for a digital mirror device array used for the present invention;

FIG. 4 is an optical diagram illustrating the operation of a digital mirror device with the present invention; and

FIG. 5 is an optical diagram showing the combination of the confocal light modulation system of FIG. 1 with a research microscope.

Ordinary light microscopes are often not adequate for the examination of biological specimens which are generally three-dimensional. Therefore, confocal microscopy was developed where the specimen is illuminated in a very small area one point at a time and a detector receives light only from the illuminated point. The image of the whole sample is obtained by scanning point to point over the sample and then reconstructing the total image.

Referring now to FIG. 1, the spatially light modulated confocal microscope of the present invention indicated generally at 10 is schematically illustrated. In this device, the sequential spot scanning of conventional confocal microscopes is replaced by multiple pattern illumination (MPI), and the single channel detector, usually a photomultiplier tube, used with conventional sequential spot scanning is replaced by a two dimensional detector array masked by a conjugate aperture pattern which conforms with the multiple pattern illumination. Sequential complimentary patterns of illumination spots are imaged on an object.

A confocal light modulation system 12, which forms the basic component of the spatially modulated confocal microscope 10, may be combined with the basic optics present in most conventional research microscopes to convert the research microscope to a confocal microscope. This confocal light modulation system includes a multiple pattern aperture array 14 which operates under the control of a central processor unit 16 to transform illuminated light from a light source 18 into sequential complimentary patterns of illumination spots which are imaged on a specimen 20 by means of a dichroic beam splitter 22 and a microscope objective lens 24. In the image plane, a conjugate array detector system 26 detects the light from the illuminated spots on the specimen which are transmitted via the beam splitter 22 and a lens 28. This array detector system includes a two-dimensional detector 30 which is masked by a conjugate aperture pattern unit 32 controlled by the central processor 16. The conjugate aperture pattern unit can be a separate unit or can constitute an addressed masking section of the detector 30. Under the control of the central processor, the conjugate aperture pattern unit masks the detector to define a conjugate aperture image corresponding to the pattern of the illuminated spots on the specimen onto the detector 30 in synchrony with the corresponding complimentary pattern formed by the multiple pattern aperture array 14. Both the conjugate aperture pattern unit and the multiple pattern aperture array are preferably high speed, pixelated arrays having a high pixel density which provide patterns of very small spots.

The detector provides a video image output over lines 34 to the central processor unit which provides video image processing and display, or the detector could provide an output to a conventional video image processing and display system of known type.

To elaborate further, using the method and apparatus of the present invention, sequential complimentary patterns of illumination spots are imaged on the specimen 20. Each full complimentary set of patterns will illuminate an entire object field without voids. The corresponding conjugate aperture patterns formed in front of a two dimensional detector modulate the image and reject stray out-of-focus light indicated in broken lines at 36, in FIG. 1 from out-of-focus planes indicated by plane 38. This process can yield one complete image of the entire object field alter the storage and subsequent summing of the conjugate aperture patterns imaged on the detector 30.

The multiple pattern aperture array 14 can take two forms; namely, interference patterns or patterned throughput from an aperture pattern. Interference patterns are complex and difficult to control, and interference pattern integrity will be compromised by the differences in refraction and birefringence across specimens of living tissue. Therefore, a multiple pattern aperture array 14 which provides direct aperture throughput is preferred for its simplicity, versatility and ease of implementation. Such a multiple pattern aperture array can be formed by an array of ferroelectric liquid crystal devices, by a digital mirror device, or by electrostatic microshutters of the type illustrated at 40 in FIG. 2. At rest, the microshutters block light from the light source 18, but when selected shutters are gated from the central processor 16, they open as illustrated at E in FIG. 2 and pass light through to the specimen 20. Electronically addressed and gated microshutter arrays of the type shown in FIG. 2 are well known and have previously been employed in display devices. (See R. Vuelleumier, A-E Perret, F. Jurrez and P. Weiss (1984), Novel electromechanical microshutter display device. Proc. Eurodisplay '84, Display Research Conference Sept. SID).

Confocal imaging is possible by using two separate throughput aperture arrays operated in synchrony for the multiple pattern aperture array 14 and the conjugate aperture pattern unit 32. However, this requires extremely careful optical alignment and timing to implement. A simpler alternative is to employ an optical configuration that avoids use of two synchronously timed aperture arrays. This could be achieved by passing the image from the specimen back to the detector through the same aperture array used to form the multiple pattern aperture array 14, thus using this single array in place of the conjugate aperture pattern unit 32. For this purpose, the multiple pattern aperture array 14 would be positioned between the beam splitter 22 and the lens 24 in FIG. 1. However, this alternative requires precise optical alignment and would not provide a unit which could be easily attached to most conventional research microscopes.

The conjugate aperture pattern unit 32 and detector 30 may be advantageously formed electronically by a CID camera or a charge-coupled device detector array (CCD) which uses computer aided image processing to enable the detector to disregard light returns from detector pixels which are not activated. A charge-coupled detector has the ability to acquire an entire image at once rather than by raster scanning. A complete bit map must be constructed of where active pixels are located on the CCD after calibration for each aperture pattern to be provided by the multiple pattern aperture array 14 under the control of the central processing unit 16. Calibration simply requires training the processing unit 16 which controls the CCD to disregard pixels which should not be read by placement of a mirror in the objective focal plane of the microscope so that each illumination pattern from the multiple pattern aperture array can be imaged directly on the CCD and stored by the central processing unit. Additional electronic and computer aided filtering after calibration may be required to clarify aperture boundaries and improve spatial resolution and sectioning capability, but once this is done, only light transmitted to the CCD for an associated pattern from the multiple pattern aperture array 14 is imaged, while light spillage to other areas of the CCD is ignored and effectively masked. This technique provides practical transmission confocal microscopy because the illumination subsystem does not use the same optical path as the imaging subsystem as is evident from FIG. 1.

For a CCD unit to work at the high speeds desired for confocal microscopy, the detector pixel registers for the CCD device must be read rapidly in parallel for all lines. Therefore, special CCD pixel driver circuitry is required to optimize the detector for high speed applications.

The need for the special driver circuitry required with a CCD detector to read lines in parallel can be eliminated by forming the detector 30 from an active pixel sensor (APS) rather than from a CCD. The active pixel sensor operates in a manner similar to that of a random access memory with each pixel containing its own selection and readout transistors. These transistors are monolithically integrated in the pixel for readout selection and amplifying or buffering the pixel output signal. An APS pixel pattern selected by the central processor 16 receives the light pattern from the specimen 20 and a photogenerated charge is integrated in each active pixel under a photogate. The selected pixels can be addressed simultaneously to provide output signals to an image processing system or to the central processing unit 16 for image processing.

The multiple pattern aperture array 14 and, for some applications, the conjugate aperture pattern unit 32 may preferably be formed by a digital mirror device which includes a monolithic electrostatic, mirror-coated shutter array. These pixelated, micromechanical spatial light modulators are formed monolithically upon a silicon chip and include minor squares of highly reflective material suspended above the surface of silicon addressing circuitry by support pillar torsion bars. The addressing circuitry under each mirror pixel is a memory cell accessed by the central processor unit 16 which drives two electrodes under each mirror with complimentary signals. When a high logical level is applied to one of the electrodes, the mirror pixel rotates in a given direction with a total deflection of about 20.degree. between mirror states, 10.degree. on either side of the perpendicular to the silicon wafer.

FIG. 3 illustrates the basic structure of one pixel for a digital mirror device array. Here, a mirror coated shutter 42 is mounted in contact with a conducting support layer 44 by torsion hinges 46. A spacer layer 48 separates the support layer 44 from a silicon chip 50. Each pixel of the digital mirror device array includes an address electrode 52 and a control or landing electrode 54 which, when activated, deflects the mirror coated shutter 42 around the torsion hinges 46. When the mirror coated shutter 42 forms part of a multiple pattern aperture array mounted in the optical path of the illumination subsection 56 for a microscope, it may be positioned in a first position as shown in FIG. 4 to direct a pixel of light along an optical path 58 to a specimen tinder observation. However, when the mirror coated shutter 42 is deflected from the position of FIG. 4, the pixel of light is angled away from the specimen along an optical path 60.

Software for the central processor unit 16 will enable a pre-programmed selection of any possible two dimensional pixel pattern within predetermined limits for confocal imaging. This technology provides the user with the flexibility to choose the pattern which offers the best combination of temporal and spacial resolution for a given application. The ability to custom make patterns can be geared for the correction of field distortion. The individual complementary images formed from each pixel pattern received by the detector 30 will be stored in the central processor unit 16 and subsequently overlaid to form one complete image frame for real-time imaging.

The control software for the central processor 16 will be programmed to select stored aperture patterns designated by the user and, for some applications, to automatically select stored aperture patterns in response to sensed conditions such as the optical characteristics of a specimen. The processor software may also cause the processor to make a specimen-dependent background correction for each image frame. This is accomplished by sensing with the detector 30 a background image which may result with all illumination pixels of the multiple pattern aperture array 14 closed to block light from the light source 18, and then correcting in the processor the subsequent individual complimentary images formed from each pixel pattern by subtracting therefrom the effect created by the background image.

FIG. 5 illustrates one manner in which the confocal modulation system 12 may be incorporated into the structure of a conventional research microscope 62. The multiple pattern aperture array 14 is inserted into the optical path of the illumination subsystem 56 for the microscope while the conjugate aperture pattern unit 32 is inserted into the optical path of the microscope image subsection 64. The remaining optical components shown in FIG. 5 are components of the research microscope 62.

Industrial Applicability

The spatially light modulated confocal microscope and method of the present invention provides conjugate aperture image formation to counter the deficiencies of scanning confocal microscopes. The confocal light modulation system may be used to convert a conventional research microscope into a confocal microscope having enhanced capabilities. Since the unit has no moving parts and is vibration free, it is suitable for sensitive micromanipulation and intracellular recording techniques common in neurophysiological studies. The high speed imaging capability of the unit renders it suitable for use to follow dendritic and synaptic changes in neurons on the millisecond scale or to measure kinetics in response to drugs or fluorescent indicators for intracellular pH, calcium, sodium, etc. The spatially light modulated confocal microscope is particularly well adapted for use in the high speed inspection of silicon wafers for quality control purposes in integrated circuit production.