Method and apparatus for three dimensionally displaying eyeground and measuring coordinates thereof

[0001] This application claims priority of Japanese Application No. JP-2002-189265, filed on Jun. 28, 2002. The entire disclosure of the above application is hereby incorporated by reference.

[0002] 1. Technical Field of the Invention

[0003] The present invention relates to methods and apparatus for displaying an eyeground three dimensionally and measuring its coordinates from photos of the eyeground.

[0004] 2. Description of the Related Art

[0005] An eyeground means the inner portion of an eyeball, and is one of the few portions of a human body wherein blood vessels and nerves can be directly observed. Conventionally, dedicated eyeground cameras have been used to observe the eyeground and diagnose abnormalities in blood vessels, retinas, tumors, etc.

[0006] The eyeground camera illuminates the interior of an eye through a pupil of a patient, and at the same time the camera takes a picture of part of the eyeground on photo film or CCD. However, the field of view of the eyeground camera is narrow at about 50 degrees, so even a specialized physician cannot easily make a precise diagnosis using a photo.

[0007] Therefore, a number of eyeground photos are connected according to conventional practice. That is, because the imaging range of an eyeground camera is narrow, photos taken in a number of directions such as center, upper, lower, left, and right directions are connected together, and the entire eyeground is diagnosed. Such connecting work is done not only manually but also semi automatically using an image processing method. In addition, eyeground cameras connected to computers have been developed for electronic clinical cards and video images can be filed.

[0008] However, because the eyeground is spherical and a photo of it is a projection of part of the eyeground on a plane, photos taken are not continuous and cannot be precisely connected.

[0009] Eyeground photos connected as described above consist of planes developed from the spherical eyeground, so only a specialist physician can precisely diagnose such photos, applying extensive expertise of the construction of an eyeball, and a physician without specialized knowledge or a patient cannot understand the images.

[0010] Furthermore, even if a specialized physician can diagnose an abnormality in blood vessels, retinas, tumors, etc. at the eyeground, images on eyeground photos have already been deformed, therefore, sizes and positions of blood vessels, retina detachment areas, etc. can neither be quantified as data nor be used as time series data or as an epidemic database.

[0011] To solve the aforementioned problems, “Methods and apparatus for recovering three dimensional patterns of an eyeground by superimposing eyeground images taken in multiple directions” (unexamined Japanese patent publication No. 2002-34925) has already been proposed.

[0012] According to this proposal, the lens of a living eye is modeled with a single lens, wherein the shape of the eyeground is regarded to be substantially a sphere; then, an image of the sphere taken through the single lens lies on a quadratic surface, and it can be assumed that the camera has taken pictures of the quadratic surface from various directions. Consequently, as shown schematically in FIG. 1, a number of unique points commonly contained in a number of eyeground photos are superimposed on the quadratic surface, thus camera positions and poses, when the shape of the quadratic surface and pictures were taken and estimated, the parameters of a single lens are obtained from the shape of the quadratic surface, and the shape of the eyeground can be reproduced.

[0013] Nevertherless, actual sizes and shapes of eyeballs vary by person, and it is known that if a person who is very near or far sighted in particular, the eyeballs deform from a precise sphere. Therefore, if the lens of a living eye is modeled with a single lens and the shape of the eyeground is regarded to be substantially a sphere, in the case of a deformed eye ball, if any, the reproduced eyeground differs from the actual eyeground, so the fact that the reproduced eyeground cannot be used for precision diagnostics or a database of eyeground data, is a problem.

[0014] According to the above mentioned method, if an eye ball is deformed significantly, images cannot be superimposed on a virtual quadratic surface. As a result, camera positions and poses when each image is photographed can no longer be estimated, so parameters for a single lens also cannot be obtained, and even the shape of the eyeground might not be reproducible.

[0015] The present invention aims to solve the aforementioned problems. More explicitly, an object of the present invention is to provide methods and apparatus that even if an eye ball is deformed from being a precise spherical surface, a three dimensional image of the eyeground can be accurately reproduced, and quantitative data on sizes and positions of abnormal blood vessels, retina detachment, tumors, etc. can be created from the above mentioned three dimensional image, and the data can be used as time dependent data of the patient and for an epidemic database by cubicly displaying the eyeground and measuring coordinates three dimensionally.

[0016] The present invention provides methods and apparatus for three dimensionally displaying an eyeground and measuring the coordinates of the eyeground, and is composed of a shape measuring step (A) for measuring the size L of an eye ball and the shape R of the eyeground, an eyeball setting step (B) for setting an eyeball template according to the measured size and shape, an eyeground image taking step (C) for taking images of the eyeground including the superimposed portion d by shifting the imaging position by a predetermined amount h, a parameter setting step (D) for obtaining eyeball parameter g, which represents the positional relationship between the eyeground and a image from the position H of the superimposed portion on the image, an image pasting step (F) for pasting a number of images on the eyeball template according to the eyeball parameter, and a three dimensional image display step (H) for displaying the three dimensional eyeball image on the eyeball template, onto a display device.

[0017] According to the above mentioned methods of the present invention, because in the shape measuring step (A), the measuring device measures the size L of the eyeball (the length from the surface of the cornea to the eyeground) and the shape R of the eyeground (radius of curvature at the eyeground), even if the eyeball is deformed from being a perfect sphere, an accurate eyeball template can be set in the eyeball setting step (B).

[0018] Because the imaging position is shifted by a predetermined amount h and an eyeground including the superimposed portion d is photographed in the eyeground image taking step (C), an eyeball parameter g that represents the positional relationship between the eyeground and the image (for example, CCD of the imaging device) can be obtained from the position H of the superimposed portion on the image in the parameter setting step (D).

[0019] Because imaging positions are shifted each time by a predetermined amount h and a number of eyeground areas including superimposed portions d are photographed, and the number of images are pasted on the eyeball template based on the eyeball parameter g in the image pasting step (F), a three dimensional image of the eyeground can be precisely produced, so this image is displayed in the display device in the three dimensional image displaying step (H) to help specialized physicians and patients understand the situation.

[0020] According to a preferred embodiment of the present invention, there is an image/position recording step (E) in which an image photographed in the aforementioned eyeground image taking step (C) and the positional relationship between the eyeball and the imaging device related to the image are recorded in a storage medium.

[0021] This method uses the storage medium to store time dependent data of the patient and provide for an epidemic database.

[0022] The present invention also provides an image positioning step (G) in which eyeground images are positioned within the eyeground surface according to the respective points at the aforementioned superimposed portions d.

[0023] Positions of eyeground images can be precisely matched using the respective points of papillae, blood vessels, nerves, etc. on the eyeground.

[0024] According to another preferred embodiment of the present invention, there are a characteristic portion indicating step (J) for indicating a characteristic portion on the above mentioned three dimensional eyeground image displayed in the display device, a three dimensional coordinate defining step (K) for obtaining three dimensional coordinates of an indicated characteristic portion, a characteristic portion measuring step (L) for measuring the size of the characteristic portion, and a data saving step (M) for saving measured data in the recording medium.

[0025] A characteristic portion on a three dimensional eyeground image displayed on the display device is indicated in the characteristic portion indicating step (J), thereby the three dimensional coordinates of the indicated characteristic portion can be obtained in the three dimensional coordinate defining step (K); in addition, the size of the characteristic portion can be measured in the characteristic portion measuring step (L); therefore, based on the three dimensional image, sizes and positions of pathologic locations such as abnormal blood vessels, retina detachment, and tumors can be converted into quantitative data.

[0026] In addition, measured data are saved in the storage medium in the data saving step (M), thereby the data can be utilized as time dependent data of the patient and as an epidemic database.

[0027] The present invention also provides a computer program to execute the above mentioned methods, and a storage medium that stores the program that the computer can read.

[0028] Using this program and the storage medium, the aforementioned methods can be implemented with the computer.

[0029] Other objects and advantages of the present invention are revealed in the following description referring to the attached drawings.

[0044] The present invention is described in more detail referring to the drawings.

[0045] In each drawing, common portions are identified with the same numbers, and no duplicate description is given.

[0046] FIGS. 2A and 2B show the positional relationship between the eyeball 1 and the imaging device 14.

[0047] FIG. 2A is a view that also shows the optical system of the imaging device 14, and FIG. 2B is a schematic view omitting the optical system of the imaging device 14. The principles of the present invention are described using these figures.

[0048] In FIG. 2A, 14a represents the lens of the imaging device 14 (eyeground camera), and 14b is a CCD that records images. When the imaging device 14 takes a picture of the eyeground, the following equation (1) is known to hold by omitting the presence of the eyeball 1.
a⁻¹+b⁻¹=f⁻¹  (1)
Where f is the focal distance of the lens 14a.

[0049] Because the eyeball 1 has a refractive index n, which is substantially constant, when the size L of the eyeball 1 (length from the surface of cornea to the eyeground) is known, the equation (2) is valid, where b in equation (1) is replaced with b′ modified from b by the size L of eyeball 1 and the refractive index n.
a⁻¹+b′⁻¹=f⁻¹  (2)

[0050] From equation (2), the ratio of the size of the subject (eyeground) and the size of an image 4 on the CCD is shown by equation (3), from equation (2).
Subject: Image=b′:a  (3)

[0051] In the above, the subject is eyeground 2, which is normally a sphere. The image 4 on the CCD lies on a plane. If the patient is very near or far sighted, the eyeball 1 deforms from being a perfect sphere, but the size L of eyeball 1 (length from the surface of the cornea to the eyeground) and the shape R of eyeground 2 (radius of curvature at the eyeground) can be measured using a measuring device 12.

[0052] Therefore, if the image 4 and the relationship between the eyeball 1 and the imaging device 14 related to the image (the aforementioned a, b, L, and R) are known accurately and the image is pasted on the eyeground of the eyeball concerned, a three dimensional eyeground image (5) can be displayed on the display device of the computer.

[0053] If the crystal lens is taken into account as shown in FIG. 9, the above mentioned equations (2) and (3) should be modified by setting a function that considers the shape of the lens.

[0054] FIG. 3 is a flow chart showing the methods of the present invention, and FIG. 4 is another flow chart showing the same.

[0055] Referring to FIG. 3, the methods of the present invention include shape measuring step (A), eyeball setting step (B), eyeground image taking step (C), parameter setting step (D), image/position recording step (E), image pasting step (F), image positioning step (G) and three dimensional image displaying step (H).

[0056] In the shape measuring step (A), the measuring device 12 measures the size L of eyeball 1 (length from the surface of cornea to the eyeground) and the shape R of eyeground 2 (radius of curvature at the eyeground).

[0057] The size L of eyeball 1 at least includes the length from the surface of cornea to the eyeground. The shape R of eyeground 2 also covers at least the radius of curvature at the eyeground. The measuring device 12 is, for example, an ultrasonic sensor that scans at least 3 points per image of an eyeground, or irradiates ultrasonic beams to precisely scan the sizes and shapes of the eyeball 1 as a whole and the eyeground 2.

[0058] In the eyeball setting step (B), an eyeball template 3 is set based on measured size L and shape R. This setting is achieved virtually on the computer. That is, the coordinate diagram of the eyeground 2 including retina, as shown in FIG. 5, is created three dimensionally.

[0059] In the eyeground image taking step (C), as shown in FIGS. 2A, 2B, 8A, and 8B, imaging positions of eyeground 2 are shifted each time by a predetermined amount d, and the eyeground including superimposed portions d are photographed a number of times.

[0060] In this imaging operation, first a number of images are shot at positions slightly displaced from each other, and according to deviations between the images, an eyeball parameter g that represents the positional relationship between the eyeground and a image from position H on the image 4 at a superimposed portion is obtained in the parameter setting step (D). For example, as shown in FIG. 2B, if a unique point A of the eyeground 2 is displaced by ΔH on the image 4 when the patient stares at 2 from points T, T′, provided the 2 points T, T′ are known, the eyeball parameter g, which shows the positional relationship between the eyeground and the image, can be obtained from the above mentioned equations (1) to (3).

[0061] In the image/position recording step (E), image 4 photographed in the eyeground image taking step (C) and the related positional relationship between the eyeball 1 and the imaging device 14 are recorded in a recording medium 15.

[0062] In the eyeground image taking step (C), as shown in FIGS. 8A and 8B in continuation, the imaging positions of the eyeground 2 are shifted by a predetermined amount h, and the eyeground including superimposed portions D is photographed a number of times. Next, as shown in FIG. 6, in the image pasting step (F), a number of images are pasted on the eyeball template 3 according to the eyeball parameter g. At that time, as shown in FIG. 7, the template on a projected image should be changed to the eyeball template 3 set in the eyeball setting step (B).

[0063] In the image positioning step (G), the positions of eyeground images 8 are adjusted appropriately within the plane of the eyeground 2 at the related points 7 in superimposed portions d (papilla, blood vessels, nerves, etc. at the eyeground).

[0064] In the three dimensional image displaying step (H), a three dimensional eyeground image 5 on the eyeball template 3 is displayed on a display device 16 (for instance, a CRT display). It is also possible to use a three dimensional display, CAVE or other devoted VR device (virtual reality device). The achieved three dimensional model can be easily three dimensionally visualized on a PC by converting the format to VRML etc.

[0065] According to the aforementioned methods of the present invention, in the shape measuring step (A), the measuring device 12 measures the size L of eyeball 1 (length from the surface of the cornea to the eyeground) and the shape R of eyeground 2 (radius of curvature at the eyeground), so even if the eyeball is deformed from a precision sphere, a precise eyeball template 3 can be set in the eyeball setting step (B).

[0066] As the eyeground, including superimposed portions d, is photographed by shifting the imaging positions each time by a predetermined amount h in the eyeground image taking step (C), an eyeball parameter g that shows the positional relationship between the eyeground and a image (for instance, CCD in the imaging device) according to the position H on the image 4 in a superimposed portion, can be ascertained in the parameter setting step (D).

[0067] Moreover, in the eyeground image taking step (C), the eyeground including superimposed portions d is photographed by shifting the imaging positions each time by the predetermined amount h, and a number of images taken are pasted on the eyeball template based on the eyeball parameter g in the image pasting step (F), thus a three dimensional image of the eyeground can be produced precisely and displayed on a display device 16 in the three dimensional image displaying step (H), to help unspecialized physicians and patients understand the clinical situation.

[0068] As shown in FIG. 4, the methods of the present invention further include three dimensional image displaying step (I), characteristic portion indicating step (J), three dimensional coordinate defining step (K), characteristic portion measuring step (L) and data saving step (M).

[0069] The three dimensional image displaying step (I) is substantially the same as the three dimensional image displaying step (H), wherein a three dimensional image of the eyeground is displayed on a display device 16.

[0070] In the characteristic portion indicating step (J), a characteristic portion 9 on the three dimensional eyeground image 5 is displayed on the display device 16 (CRT etc.). Such an indicateion is given by enclosing the characteristic portion 9 on a two dimensional display device such as CRT display, using an input device such as a mouse, as shown in FIG. 10.

[0071] In the three dimensional coordinate defining step (K), three dimensional coordinates 10 are obtained from two dimensional coordinates designated by the characteristic portion as indicated. In the characteristic portion measuring step (L), the size of the characteristic portion is measured. These steps K and L are processed automatically by the computer.

[0072] In the data saving step (M), measured data 11 (three dimensional coordinates 10 and sizes of characteristic portions) are saved in a storage medium 15. The storage medium 15 may be either tape, drum, hard disk, CD, or any other storage device known in the prior art.

[0073] Using any of the above mentioned methods, three dimensional coordinates 10 of indicated characteristic portions can be obtained in the characteristic portion indicating step (J); furthermore, in that step, the sizes of characteristic portions can be measured, and based on three dimensional images, the sizes and the positions of pathological portions such as abnormal blood vessels, retina detachment and tumors can be converted into quantitative data.

[0074] By saving measured data (11) to the storage medium (15), the data can be used as time dependent data of the patient or in an epidemic database.

[0075] In addition, the above mentioned methods can be implemented by means of a program with the computer and a storage medium that stores the program and can be read by the computer.

Preferred Embodiments

[0076] Preferred embodiments of the present invention are described below.

[0077] FIG. 11 is an image on a display, showing an embodiment of the eyeball template in an intermediate tone. This embodiment is an example of a normal eyeball, and the eyeball can be regarded forming a spherical surface. In fact, the sizes and the shapes of eyeballs vary by person; a patient having strong near or far sight, is known to have eyeballs that are deformed from a perfect sphere. The present invention is characterized in that based on the size L of the eyeball 1 and the shape R of the eyeground 2, measured in the shape measuring step (A), a precise eyeball template 3 is set in the eyeball setting step (B).

[0078] FIG. 12 is an image on a display, showing an example of an eyeground photo in an intermediate tone. Obviously, the field of view in this photo is narrow at about 50 degrees, so it is understood that even a specialized physician cannot easily make a precise diagnosis from this one photo.

[0079] FIGS. 13 and 14 show images on a display, that is, examples of three dimensional eyeground images in an intermediate tone. FIG. 13 shows a three dimensional eyeground image 5 consisting of a number of eyeground photos (9 photos in this example) pasted on an eyeball template; actually, the field of view was changed in various directions on a computer CRT from inside the eyeground, and the eyeground was displayed stereoscopically.

[0080] For FIG. 14, the eyeground image 8 was positioned more precisely than in FIG. 13, in the image positioning step (G); in the same way as for FIG. 13, the field of view can be changed freely from inside the eyeball on CRT of a computer, and the eyeground can be displayed three dimensionally.

[0081] As described above, according to the present invention, information on an eyeground having a three dimensional structure is converted into images, that is, two dimensional information, when the eyeground of the eyeball is photographed. Although these images alone cannot form a three dimensional model of the eyeground, the three dimensional model composed uses most of the features of the eyeground images.

[0082] In practice, based on eyeground images of a patient and the size of an eyeball of the patient, a three dimensional model is constructed by virtually projecting images onto the eyeball and adding other one dimensional information in the direction of depth to the two dimensional images.

[0083] The three dimensional model thus constructed can be displayed stereoscopically using a computer graphic (CG) method. The model visualized helps to obtain informed consent because the patient can understand the construction of the eyeground without requiring any special expertise.

[0084] In addition, the three dimensional model thus formed can be measured; therefore, based on this three dimensional model, the location of damaged blood vessels, range of retina detachment, and tumors can be designated by physician, engineer, or automated diagnostic system, and its three dimensional coordinates can be ascertained. More explicitly, three dimensional coordinates can easily be calculated from the information on images projected two dimensionally (on the monitor) and visualized, by assigning a two dimensional location using a mouse etc.

[0085] From the three dimensional coordinates determined, it is also possible to define numerical data such as volumes of blood vessels and retina detached location, and their areas and shapes on the retina. With images obtained by a conventional eyeground camera, the necessary dimensions cannot be easily measured because magnification scales differ between surroundings and center of image. However, using the methods of the present invention, numerical data of such dimensions can be easily obtained because a three dimensional model is constructed. These numerical data such as for areas can be collated with time dependent data of the patient, so therapy criteria such as the progress of a disease can be presented. Also, a database can be constituted by gathering data from many patients, thereby epidemic data including diseases (frequently occurring locations, relationship with ages) can be provided. Moreover, it is also possible to configure basic data for a computer simulated analysis of a disease, which is expected to be developed further in the future.

[0086] The above mentioned methods and apparatus for displaying an eyeground three dimensionally and measuring related coordinates enable the precise production of three dimensional images of the eyeground, even if the eyeball is deformed from being a perfect sphere, so the methods and the apparatus provide great advantages such as presenting quantitative data on sizes and locations of pathologic locations of abnormal blood vessels, retina detachment, tumors, etc. based on the aforementioned three dimensional images, and use the data for time dependent therapy on the patient and in an epidemic database.

[0087] Although the present invention has been described referring to several preferred embodiments, it should be understood that the scope of rights covered by the present invention is not limited only to these embodiments. Conversely, the scope of rights of the present invention should include all modifications, corrections, and similar entities included in the claims attached.