Determining mammographic image view and laterality

[0001] This invention generally relates to techniques for processing mammogram images, and more particularly to techniques for automatic detection of view type and laterality for digital mammographic images.

[0002] Screening mammography is a method for detecting signs of breast cancer in otherwise asymptomatic women. The American College of Radiology recommends annual mammography for women over forty. Women undergo an X-ray exam in which X-ray films of the breast are exposed and then developed for later review. A radiologist reads the films and assesses the likelihood of the presence of signs of breast cancer. If a suspicious finding is present, the woman may be invited for additional, more detailed diagnostic X-ray exams, ultrasonic exams, and/or biopsy.

[0003] In a typical screening exam in the United States, four X-rays of the breast are obtained. In conventional practice, two mammographic views are obtained for each breast: a cranio-caudal (CC) view is obtained by positioning the X-ray film horizontally under the compressed breast, and a medio-lateral oblique (MLO) view is obtained by positioning the X-ray film in a plane that is approximately orthogonal to the left-right axis. In some situations, more or fewer X-ray views may be obtained. The four views are typically labeled LCC (Left Cranio-Caudal), RCC (Right Cranio-Caudal), LMLO (Left Medio-Lateral Oblique) and RMLO (Right Medio-Lateral Oblique). Knowledge of the laterality (that is, Left/Right side, facing toward the patient) and type of mammographic view (that is, whether the view is a CC or MLO view) is critical for the review process.

[0004] It is common in radiological practice to display the different exposures for a patient in a particular format, often referred to as a hanging protocol. Initially devised when film media was used exclusively, the hanging protocol determined the specific spatial arrangement with which the radiologist or technician arranged the films on the light box, according to practitioner preferences. In a conventional hanging protocol for screening mammography, the LCC, RCC, LMLO, and RMLO views are typically displayed in mirrored fashion, such that the thorax edges of both breasts are centered, with the left breast image displayed on the right and the right breast image displayed on the left. However, both breasts images are acquired in a similar manner. It is, in general, not known which image corresponds to the left or the right breast; one image must be flipped before it can be positioned adjacent to the other image. In conventional screen-film imaging, X-ray-markers, for example comprised of opaque lead letters, are radiographed simultaneously (appropriately labeled RCC, LCC, RMLO and LMLO) with the breast tissue. As can be expected, not all films are correctly labeled or oriented properly for viewing, due to technician inexperience or error.

[0005] In order to suit practitioner expectations and to allow a smooth transition from film-based to digital radiography, the practice of using X-ray markers still applies for mammography imaging using images scanned from film or computed radiography (CR) cassettes. In place of the conventional light box, high resolution monitors can now be used to display patient exposures as digital images. The same hanging protocol can still be used by the practitioner for display of mammography images.

[0006] FIG. 1 shows a typical hanging protocol arrangement that is used. A display 10 has an RMLO image 20, an LMLO image 30, an RCC image 40, and an LCC image 50 arranged as shown. Each image has a corresponding marker 12, placed by the technician nearest the axilla of the patient prior to imaging.

[0007] It is desired that the mammography images, once obtained digitally, be displayed in the proper orientation and order in the hanging protocol. In order for this to happen, each image needs to be properly identified as to its type (MLO or CC) and side, or laterality (R or L).

[0008] U.S. Pat. No. 5,917,929 (Marshall et al.) entitled “User Interface for Computer Aided Diagnosis System” describes an operator interface panel that accepts operator commands on films for scanning and their proper identification and orientation. While such an approach may be workable for scanned films, however, this type of method requires operator interaction and is subject to operator error. It can be appreciated that automated solutions for detection of type and laterality of digital mammography images would be preferable to those requiring an operator.

[0009] There have been some attempts to solve this problem using image analysis techniques. Some examples are provided.

[0010] U.S. Publication No. 2004/0161164 (Dewaele) entitled “Method of Detecting the Orientation of an Object in an Image” describes orientation detection using mathematical moments of various axes or landmarks for shape detection.

[0011] PCT Application No. WO 02/45437 (Hartman et al.) entitled “Method and System for Automatic Identification and Orientation of Medical Images” describes a method using a set of templates for optical character recognition and other techniques for determining image orientation automatically.

[0012] PCT Application No. WO 05/052849 (Bamberger) entitled “Method for Computerized Analysis in Mammography” describes a workstation and method for identifying when an image from film has been inadvertently flipped with the incorrect orientation or rotation.

[0013] While these examples may be directed to image orientation, there is room for improvement, particularly with respect to noise effects, handling of operator error, and overall robustness. Thus, there is a need for a capable and robust solution for determining image type and laterality for digital mammography images.

[0014] One object of the present invention is to provide an automated method for identifying the view and laterality of mammography images. Another object of the present invention is to provide an automated method for detecting the orientation of mammography images in order to automatically display mammography images for diagnosis.

[0015] According to one aspect of the present invention there is provided a method for displaying a mammography image. Digital data of the mammography image is obtained. The mammography image is segmented to identify at least a first diagnostically relevant region comprising an image of the breast tissue and a second diagnostically relevant region. A view type is assigned for the image, either cranio-caudal or medio-lateral oblique view, according to a symmetry index calculated from the segmented first diagnostically relevant region. Right or left laterality is assigned to the image according to a laterality feature calculated according to the relative position of at least the second diagnostically relevant region within the image.

[0016] An advantage of the method of the present invention is that it does not require interpreting text characters from a marker used to identify image orientation; only the presence of the marker in a suitable location in the image is needed.

[0017] These and other objects, features, and advantages of the present invention will become apparent to those skilled in the art upon a reading of the following detailed description when taken in conjunction with the drawings wherein there is shown and described an illustrative embodiment of the invention.

[0026] The present description is directed to elements forming part of, or cooperating more directly with, apparatus in accordance with the invention. It is to be understood that elements not specifically shown or described may take various forms well known to those skilled in the art.

[0027] The method of the present invention uses analysis of segmented images and probability logic to identify type and laterality for mammography images. Using the method of the present invention, a system can accept, as input image data, the standard set of mammography images for a patient and can identify the image view type (MLO or CC) and laterality (R or L). These images can then be provided to a system for display using an appropriate hanging protocol or pattern that meets the needs of the radiology practitioner.

[0028] The method of the present invention makes some assumptions about digital mammography images. For example:

• • [0029] (i) The images are of the standard image set with four views (RCC, LCC, RMLO, and LMLO), or two views for a breast.
  • [0030] (ii) There is a marker, placed by the technician at the time of image capture in the proper position. However, the marker need not be the correct one. (For example, the LMLO marker may have been incorrectly used during capture of the RMLO image).

[0031] Referring to FIG. 2, there is shown a block diagram of the basic steps in logic flow for using the method of the present invention.

[0032] In an image acquisition step 100, the images of a patient are obtained as digital data. This can be image data generated/captured directly as digital data, such as, for example, from scanned film, computed radiography (CR), or digital radiography (DR).

[0033] A segmentation step 110 is executed to segment the radiographic images to identify at least two segmented regions; three basic regions are generally identified by segmentation, as described subsequently. A collimation region (that is, foreground) is the area of the image that is occluded to X-ray collimation during the exposure and normally presents salient borders surrounding the body part. Direct exposure regions (that is, background) are areas that have received direct X-ray exposure. Diagnosis useful regions (that is, anatomy) contain the breast tissue region and the marker region.

[0034] There are known segmentation techniques that could be applied in step 110. The method outlined in U.S. Publication No. 2005/0018893 entitled “Method of Segmenting a Radiographic Image into Diagnostically Relevant and Diagnostically Irrelevant Regions” by Wang et al., incorporated herein by reference, can be used. This process typically involves sub-sampling of the original image to generate an image of coarser resolution that can be more easily processed and segmenting the anatomy region from the sub-sampled image. Other segmentation techniques may obtain two thresholds from the image histogram, then segment the image into the foreground, background, and anatomy regions based on these thresholds.

[0035] FIGS. 3A, 3B, 4A, and 4B show typical results from applying segmentation techniques, such as described in U.S. Publication No. 2005/0018893 (Wang et al.). In FIGS. 3A and 3B, LMLO image 30 is segmented to provide segmented image 34. In FIGS. 4A and 4B, processing of LCC image 50 yields segmented image 54.

[0036] Once an image is segmented, the foreground and background can be removed from the original mammogram image by setting their pixel value to a pre-defined value. As the result, the segmented image only contains the set of diagnostically relevant regions that are useful for further processing, both to determine type and laterality, and to perform the diagnostic assessment. Among these regions, the breast region is the major region in the image, labeled Region 1 in FIG. 4B.

[0037] In addition, each of these images has a marker region (shown, for example, at element 12 in FIGS. 3A and 4A, and at element 22 in FIG. 4B) that is one of the diagnostically relevant regions according to the method of the present invention. It is noted that text characters from the marker region are not needed for image identification. In FIG. 4B, marker region 22 is labeled as Region 2. Under some circumstances, a variable collimation region, incorrectly placed markers, or embedded cassette information can appear as other diagnostically relevant regions due to imperfect segmentation, such as is shown as Region 3 in the example of FIG. 4B. Regions 1 (breast region) and 2 (marker region 22) are the minimum set of diagnostically relevant regions needed for the method of the present invention. Additional regions, as exemplified by Region 3 in FIG. 4, may also be diagnostically relevant according to the present invention.

[0038] Referring again to FIG. 2, an analysis step 120 is executed on the diagnostically relevant regions, based on segmented images obtained from segmentation step 110. For this step, each region in the segmented image can first be uniquely labeled and assigned an integer, then analyzed. According to one embodiment of the present invention, the label number of each region is associated with the region's area. For example, as shown in FIG. 4B, the region with the largest area is assigned as Region 1, and the second largest region is assigned as Region 2, and so forth.

[0039] Region 1 will include the image of breast tissue and is used to determine the type of image, whether CC or MLO. FIGS. 5A and 5C show example segmented images 54 and 34 and illustrate key features used for this procedure. Referring to FIGS. 5B and 5D, for Region 1, a vertical profile 14, 16 is first extracted from the image by using the following equation:

[0040] $\begin{array}{cc}P\left(y\right)=\sum _x^{}f\left(I\left(x,y\right)\right)f\left(I\left(x,y\right)\right)=\{\begin{array}{cc}1& I\left(x,y\right)\in \mathrm{Region}1\\ 0& I\left(x,y\right)\notin \mathrm{Region}1\end{array}& \left(1\right)\end{array}$
where I(x,y) is the labeled image. In the present invention, an assumption is used for differentiating the MLO and CC views. That is, it can be assumed that the extracted profile of a CC view image generally includes a self-symmetrical portion, or is at least substantially more symmetrical than is the MLO view. A symmetry index can be computed as a measure of relative symmetry. In accordance with one embodiment of the present invention, analysis of the profile symmetry obtained in Equation (1) is accomplished by computing the symmetry index S of the profile, which is obtained by the following equation:

[0041] $\begin{array}{cc}S=\frac{1}{\sum _m^{}\sum _i^mP\left(\mathrm{peak}-i\right)-P\left(\mathrm{peak}+i\right)}m=\{\begin{array}{cc}\mathrm{peak}& \mathrm{peak}<\mathrm{rows}-\mathrm{peak}\\ \mathrm{rows}-\mathrm{peak}& \mathrm{peak}\ge \mathrm{row}-\mathrm{peak}\end{array}& \left(2\right)\end{array}$
where:

[0042] peak is location of the profile peak, that is, the row with the maximal profile value, as indicated in FIGS. 5B and 5D;

[0043] i is the distance from the peak (row number); and

[0044] rows represents the total number of rows in the image.

[0045] By way of illustration, FIG. 5B shows an example vertical profile 14 for the LCC segmented image 54 of FIGS. 4B and 5A. FIG. 5D shows an example vertical profile 16 corresponding to the LMLO segmented image 34 of FIGS. 3B and 5C. According to Equation (2), the larger the symmetry index S, the more symmetrical will be the profile. In other words, the larger this index, the more likely that an image is of CC type, rather than MLO type.

[0046] It can be observed that the present invention is not limited to using this embodiment with Equation (2) or using profile symmetry in order to perform analysis for projection recognition. Other suitable algorithms may be known to those skilled in the art and can be employed, provided that they identify the difference between the two projection view types with some degree of accuracy.

[0047] Analysis for identifying the laterality of mammography images takes advantage of the other diagnostically relevant regions, that is, the region or regions other than that containing the breast image. For example, in one embodiment, Region 1 is first effectively removed from the segmented image by setting each of its pixels to a predefined value. FIG. 6A shows a segmented cranio-caudal image 56 of a left breast, but having Region 1 removed. Then, the remaining image is split vertically (that is, in the cranio-caudal direction) into two equal-sized portions, as shown by top and bottom portions 58a and 58b in FIGS. 6B and 6C.

[0048] For each of these top and bottom portions, a laterality feature L is computed using the equation:

[0049] $\begin{array}{cc}L=\sum _{x,y}^{}f\left(I\left(x,y\right)\right)f\left(I\left(x,y\right)\right)=\{\begin{array}{cc}1& I\left(x,y\right)\in \mathrm{any}\mathrm{Region}\\ 0& I\left(x,y\right)\notin \mathrm{any}\mathrm{Region}\end{array}& \left(3\right)\end{array}$
where I(x,y) is the labeled image after removing the largest region (that is, Region 1).

[0050] Since the portion 58a has the larger laterality feature L value, the image 56 represents the left side of the patient; otherwise, it would represent the right side. According to one conventional hanging protocol for screening, the images from the right side of the patient appear on the left, and the images from the left side of the patient appear on the right portion of the display. As with the symmetry index S described earlier, the laterality feature L could alternately be calculated using any of a number of other algorithms, as alternatives to that given by way of example in Equation (3).

[0051] A similar approach can be used to identify the correct/incorrect orientation of mammogram images. Generally, if lower portion 58b has a larger L value, the image has incorrect orientation and needs to be rotated.

[0052] Referring again to FIG. 2, an identification step 130 is executed. Although the methods described for analysis in step 120 provide an indication of image type and laterality, there can be some ambiguity, due to patient differences, technician practices, and equipment differences. For this reason, identification step 130 of the present invention uses a probability vector for each input mammogram image. This method can use the approach described in U.S. Publication No. 2004/0234125, entitled “System And Method Of Assigning Mammographic View And Laterality To Individual Images In Groups Of Digitized Mammograms” by Menhardt et al., incorporated herein by reference.

[0053] As shown in FIG. 7, the probability vector approach has four elements corresponding to the four possibilities for type and laterality (that is, LMLO, RLMO, RCC, and LCC). Each element entered in the table shown in FIG. 7 represents the probability of the input image having the specified view, based on analysis step 120 results. Using this approach, the symmetry index S (Equation 2) and laterality feature L (Equation 3) can be used to generate a probability vector for each image. The final decision on type and laterality is then made by evaluating the four probability vectors of images in the same study.

[0054] This method seeks global optimization by maximizing the sum of the probabilities of all mammograms of a patient. In the example of FIG. 7, Image 2 is clearly an LMLO image using this probability data. Images 1 and 3 can be determined with somewhat less confidence. Image 4 is assigned with a lower probability factor than other images. As a result, the sum of probabilities of all images is maximal among all possible sums.

[0055] At the conclusion of identification step 130, the image type and laterality of a set of standard mammography images can be automatically determined, along with its orientation. The detected type and laterality can then be assigned to each image and may be displayed along with the image or stored in a file header or in a separate file or other data structure that is associated with the image. Type and laterality assignment could also be displayed in an otherwise unused part of the image background. This assignment would allow the images to be displayed to a practitioner in suitable format, on one or more high-resolution display monitors, without the need for operator intervention or rearrangement. Image data stored with the assigned designation, such as in the image header or in some other manner, would then be available when an image is recovered from storage, such as from a PACS image storage system.

[0056] Unlike other approaches, the method of the present invention does not require that an operator use the correct lead marker when performing the image operation. The present invention uses a probabilistic model for decision-making and is thus adaptable to situations where there is somewhat less clarity about image type or there is ambiguity in the recognition results of one or two images in a study. While described primarily with regard to applications in mammography, the method of the present invention could be adapted to other types of diagnostic imaging, where it is a need to classify views taken from different perspectives or on different sides of the body.

[0057] The invention has been described in detail with particular reference to certain embodiments thereof, but it will be understood that variations and modifications can be effected within the scope of the invention as described above, and as noted in the appended claims, by a person of ordinary skill in the art without departing from the scope of the invention. For example, various types of image processing algorithms could be applied for segmentation and for determining image type and laterality. Any of a number of alternative approaches can be used for providing a symmetry index or computing a laterality feature.

[0058] Thus, what is provided is an apparatus and method for automatic detection of view type and laterality for digital mammographic images.

• [0059] 10 Display
• [0060] 12 Marker
• [0061] 14, 16 Vertical profile
• [0062] 20 RMLO image
• [0063] 22 Segmented marker image
• [0064] 30 LMLO image
• [0065] 34 Segmented image
• [0066] 40 RCC image
• [0067] 50 LCC image
• [0068] 54 Segmented image
• [0069] 56 Segmented image
• [0070] 58a, 58b Image portion
• [0071] 100 Image acquisition step
• [0072] 110 Segmentation step
• [0073] 120 Analysis step
• [0074] 130 Identification step