Identifying and quantizing system, operation method thereof, and non-transitory computer readable medium

[0001] This application claims priority to Taiwan Patent Application No. 112137958, filed Oct. 3, 2023, the entirety of which is herein incorporated by reference.

[0002] The present invention relates to systems and operation methods, and more particularly, identifying and quantizing systems and operation methods thereof.

[0003] Cancer is the first among the top ten causes of death in Taiwan. With the advancement of cancer treatment, the average life expectancy of various cancers has increased significantly. However, while the survival time of patients has been prolonged, brain metastasis of cancer has become the next treatment topics. Due to its special tissue structure, the brain is difficult for various therapeutic drugs to reach, making the brain a breeding ground for cancer to metastasize and survive after it is controlled at the original site. For the development of imaging technology, MRI can provide clear imaging examinations of the brain, so as to detect tiny brain metastases early, and therefore tumors can be treated before they expand through non-pharmacological treatment, such as various three-dimensional targeted radiation or focused ultrasound therapy. In addition to detecting tumors as early as possible, imaging examinations are also very important for evaluating the efficacy and side effects of tumors being treated. During the treatment process, the size and shape of the tumor may change in different ways, and the brain tissue next to the tumor may also produce varying degrees of cerebral edema, causing various side effects. Tracking these brain changes is critical to the effectiveness of treatments.

[0004] Effective treatment must be based on good imaging examinations. However, each MRI examination is composed of a large number of two-dimensional images stacked. Whether it is the discovery of lesions or the evaluation of treatment, it requires manual and careful interpretation. The large number of images not only makes it possible to miss small tumors, but also can only roughly estimate the size of brain tumors and edema, but cannot accurately quantify them, making the clinical assessment of treatment targets and side effects potentially inaccurate.

[0005] In one or more various aspects, the present disclosure is directed to identifying and quantizing systems and operation methods thereof.

[0006] An embodiment of the present disclosure is related to an identifying and quantizing system. The identifying and quantizing system includes a storage device and a processor. The storage device is configured to store at least one instruction. The processor is coupled to the storage device, and the processor configured to access and execute the at least one instruction for: splitting a T2 weighted image into a first group of two-dimensional images; inputting the first group of two-dimensional images into a mask R-CNN (mask region-based convolutional neural network) model to obtain a first group of two-dimensional parenchymal brain images; using a first group of two-dimensional parenchymal brain images to form a T2 weighted parenchymal brain images; performing a pre-process on the T2 weighted parenchymal brain image to obtain a pre-processed T2 weighted parenchymal brain image; and using a three-dimensional convolutional neural network model to segment and quantize a brain edema area in the pre-processed T2 weighted parenchymal brain image.

[0007] In one embodiment of the present disclosure, the processor accesses and executes the at least one instruction for: co-registering a contrast enhanced T1 (T1C) weighted image to the T2 weighted image to obtain a co-registered T1C weighted image; splitting the co-registered T1C weighted image into a second group of two-dimensional images; inputting the second group of two-dimensional images into the mask R-CNN model to obtain a second group of two-dimensional parenchymal brain images; using the second group of two-dimensional parenchymal brain images to form a T1C weighted parenchymal brain image; performing the pre-process on the T1C weighted parenchymal brain image to obtain a pre-processed T1C weighted parenchymal brain image; and using another three-dimensional convolutional neural network model to segment and quantize a metastatic brain tumor area in the pre-processed T1C weighted parenchymal brain image.

[0008] In one embodiment of the present disclosure, the pre-process includes an image resampling.

[0009] In one embodiment of the present disclosure, the pre-process includes image normalization.

[0010] In one embodiment of the present disclosure, the processor accesses and executes the at least one instruction for: pre-training the three-dimensional convolutional neural network model, where the three-dimensional convolutional neural network model has two convolution paths, one of the two convolution paths extracts each region in training data so as to perform a feature extraction on the each region, another of the two convolution paths selects each corresponding expanded area based on a center point of the each area and performs an under-sampling on the each corresponding expanded area for the feature extraction.

[0011] Another embodiment of the present disclosure is related to an operation method of an identifying and quantizing system. The operation method includes steps of: splitting a T2 weighted image into a first group of two-dimensional images; inputting the first group of two-dimensional images into a mask R-CNN model to obtain a first group of two-dimensional parenchymal brain images; using a first group of two-dimensional parenchymal brain images to form a T2 weighted parenchymal brain images; performing a pre-process on the T2 weighted parenchymal brain image to obtain a pre-processed T2 weighted parenchymal brain image; and using a three-dimensional convolutional neural network model to segment and quantize a brain edema area in the pre-processed T2 weighted parenchymal brain image.

[0012] In one embodiment of the present disclosure, the operation method further comprises: co-registering a T1C weighted image to the T2 weighted image to obtain a co-registered T1C weighted image; splitting the co-registered T1C weighted image into a second group of two-dimensional images; inputting the second group of two-dimensional images into the mask R-CNN model to obtain a second group of two-dimensional parenchymal brain images; using the second group of two-dimensional parenchymal brain images to form a T1C weighted parenchymal brain image; performing the pre-process on the T1C weighted parenchymal brain image to obtain a pre-processed T1C weighted parenchymal brain image; and using another three-dimensional convolutional neural network model to segment and quantize a metastatic brain tumor area in the pre-processed T1C weighted parenchymal brain image.

[0013] In one embodiment of the present disclosure, the step of performing the pre-process on the T2 weighted parenchymal brain image includes: performing an image resampling on the T2 weighted parenchymal brain image.

[0014] In one embodiment of the present disclosure, the step of performing the pre-process on the T2 weighted parenchymal brain image includes: performing an image normalization on the T2 weighted parenchymal brain image.

[0015] In one embodiment of the present disclosure, the operation method further includes: pre-training the three-dimensional convolutional neural network model, where the three-dimensional convolutional neural network model has two convolution paths, one of the two convolution paths extracts each region in training data so as to perform a feature extraction on the each region, another of the two convolution paths selects each corresponding expanded area based on a center point of the each area and performs an under-sampling on the each corresponding expanded area for the feature extraction.

[0016] Technical advantages are generally achieved, by embodiments of the present disclosure. With the identifying and quantizing system and its operation method of the present disclosure, the technology of image interpretation and segmentation using artificial intelligence allows computers to automatically segment not only metastatic brain tumors, but also brain edema in adjacent areas. With the assistance of artificial intelligence, clinicians can not only quickly and without omissions find the lesions that need treatment, but also quantitatively analyze the possible brain edema area before and after treatment, thereby improving the accuracy and safety of treatment.

[0017] Many of the attendant features will be more readily appreciated, as the same becomes better understood by reference to the following detailed description considered in connection with the accompanying drawings.

[0021] Reference will now be made in detail to the present embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

[0022] Referring to FIG. 1, in one aspect, the present disclosure is directed to an identifying and quantizing system 100. The identifying and quantizing system 100 may be easily integrated into a computer and may be applicable or readily adaptable to all technologies. Technical advantages are generally achieved by the identifying and quantizing system 100 according to embodiments of the present disclosure. Herewith the identifying and quantizing system 100 is described below with FIG. 1.

[0023] The subject disclosure provides the identifying and quantizing system 100 in accordance with the subject technology. Various aspects of the present technology are described with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It can be evident, however, that the present technology can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing these aspects. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0024] In practice, for example, the identifying and quantizing system 100 can be a computer server. The computer server can be remotely managed in a manner that substantially provides accessibility, consistency, and efficiency. Remote management removes the need for input/output interfaces in the servers. An administrator can manage a large data centers containing numerous rack servers using a variety of remote management tools, such as simple terminal connections, remote desktop applications, and software tools used to configure, monitor, and troubleshoot server hardware and software.

[0025] As used herein, “around”, “about”, “substantially” or “approximately” shall generally mean within 20 percent, preferably within 10 percent, and more preferably within 5 percent of a given value or range. Numerical quantities given herein are approximate, meaning that the term “around”, “about”, “substantially” or “approximately” can be inferred if not expressly stated.

[0026] In practice, in an embodiment of the present disclosure, the identifying and quantizing system 100 can selectively establish a connection with the MRI (magnetic resonance imaging) machine 190. It should be understood that in the embodiments and the scope of the patent application, the description involving “connection” can generally refer to a component that indirectly communicates with another component by wired and/or wireless communication through another component, or a component that is physically connected to another element without through another element. For example, the identifying and quantizing system 100 can indirectly communicate with the MRI machine 190 through wired and/or wireless communication via another component, or the identifying and quantizing system 100 can be physically connected to the MRI machine 190 without another component. Those with ordinary skill in the art may select the connection manner depending on the desired application.

[0027] FIG. 1 is a block diagram of the identifying and quantizing system 100 according to one embodiment of the present disclosure. As shown in FIG. 1, the identifying and quantizing system 100 includes a storage device 110, a processor 120 and a display device 130. For example, the storage device 110 can be a hard drive, a flash memory or another storage device, the processor 120 can be a central processing unit, and the display device 130 can be a built-in display or an external screen.

[0028] In structure, the identifying and quantizing system 100 is electrically connected to the MRI machine 190, the storage device 110 is electrically connected to the processor 120, the processor 120 is electrically connected to the display device 130. It should be noted that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present. For example, the storage device 110 may be a built-in storage device that is directly connected to the processor 120, or the storage device 110 may be an external storage device that is indirectly connected to the processor 120 through the network device.

[0029] In practice, in some embodiments of the present disclosure, the MRI machine 190 collects a plurality of original MRI data from the subject. In practice, for example, the MRI device 190 can capture a T2 weighted image and a contrast enhanced T1 (T1C) weighted image of the patient, and the T1C weighted image can be a T1 weighted image after contrast agent injection and development. Although only one MRI device 190 is shown in FIG. 1, this does not limit the present disclosure. In practice, the MRI device 190 can generally refer to one or more MRI devices. Those with ordinary skill in the art may flexibly select the amount of MRI device(s) depending on the desired application.

[0030] In use, the storage device 110 store the weighted image, the T1C weighted image and at least one instruction, and the processor 120 accesses and executes the at least one instruction for: splitting a T2 weighted image into a first group of two-dimensional images; inputting the first group of two-dimensional images into a mask R-CNN (mask region-based convolutional neural network) model to obtain a first group of two-dimensional parenchymal brain images; using a first group of two-dimensional parenchymal brain images to form a T2 weighted parenchymal brain images; performing a pre-process on the T2 weighted parenchymal brain image to obtain a pre-processed T2 weighted parenchymal brain image; and using a three-dimensional convolutional neural network model to segment and quantize a brain edema area in the pre-processed T2 weighted parenchymal brain image. The display device 130 can display the brain edema area in the pre-processed T2 weighted parenchymal brain image. In practice, compared with other images, the processor 120 uses T2 weighted images to perform the above processing, which is more conducive to identify and quantify the brain edema area.

[0031] Similarly, in some embodiments of the present disclosure, the processor 120 accesses and executes the at least one instruction for: the processor accesses and executes the at least one instruction for: co-registering a contrast enhanced T1 (T1C) weighted image to the T2 weighted image to obtain a co-registered T1C weighted image; splitting the co-registered T1C weighted image into a second group of two-dimensional images; inputting the second group of two-dimensional images into the mask R-CNN model to obtain a second group of two-dimensional parenchymal brain images; using the second group of two-dimensional parenchymal brain images to form a T1C weighted parenchymal brain image; performing the pre-process on the T1C weighted parenchymal brain image to obtain a pre-processed T1C weighted parenchymal brain image; and using another three-dimensional convolutional neural network model to segment and quantize a metastatic brain tumor area in the pre-processed T1C weighted parenchymal brain image. The display device 130 can display the metastatic brain tumor area in the pre-processed T1C weighted parenchymal brain image. In practice, compared with other images, the processor 120 uses T1C weighted images to perform the above processing, which is more conducive to identify and quantify the metastatic brain tumor areas. In addition, by co-registering the T1C weighted image to the T2 weighted image, the same mask R-CNN model can extract brain masks (i.e., substantial brain images) from the T2 weighted image and/or the co-registered T1C weighted image.

[0032] Regarding the above-mentioned Mask R-CNN, it can effectively analyze two-dimensional images to extract brain masks (i.e., substantial brain images). The original MR images include not only the cerebrospinal fluid, cerebrum, and cerebellum, but also the skull, scalp, and non-parenchymal brain regions. The brain edema and tumors to be segmented and quantified by the present disclosure occur in the brain parenchyma area. There are some noises or artifacts produced during magnetic resonance imaging, and some noises or artifacts show higher grayscale values in the T2 weighted image. In order to avoid misjudgment by the neural network and waste of training resources when segmenting cerebral edema, the present disclosure can only extract brain parenchymal areas for segmentation and quantification, excluding non-parenchymal brain areas.

[0033] The method of extracting brain masks is to use two-dimensional mask R-CNN. The mask R-CNN can achieve target detection, target classification and pixel-level target segmentation. It is a new convolutional neural network based on the Fast/Faster R-CNN architecture.

[0034] The training data for training to extract brain masks is to first use Statistical Parametric Mapping 12 (SPM 12). SPM software is specially designed to analyze brain imaging data. After locating the anterior commissure point in magnetic resonance imaging, the software can segment gray matter, white matter, cerebrospinal fluid, skull and soft tissue. The segmented gray matter, white matter and cerebrospinal fluid are then combined, and the brain mask generated by SPM is further manually repaired. After final inspection by a doctor, these data are used as the gold standard of supervised learning models.

[0035] The images used in training the brain extraction model are all T2 weighted images. The T1C weighted image is co-registered to the T2 weighted image to generate a co-registered T1C weighted image. The model segment mask trained through the T2 weighted image can be applied to co-registered T1C weighted image.

[0036] There is a total of 40 patients (a total of 1799 slices) in the data set of the brain extraction model, and the number of slices for each patient ranges from 42 to 54, in which 28 slices are the training set, 6 slices are the verification set, and 6 slices are the test set.

[0037] Regarding the above pre-process, specifically, in some embodiments of the present disclosure, the pre-process includes an image resampling.

[0038] Voxel spacing defines the physical size of each voxel in real situations. Although the size of each person's head is different, the image resolution of magnetic resonance imaging is fixed, which may cause the voxel spacing of each patient's image to be different.

[0039] In addition, since the three-dimensional convolutional neural network model (e.g., DeepMedic) model can randomly extract one of the areas of the data during training. In order to avoid that the size of the data extracted each time is different in the real situation to decrease the performance of the model, resampling the voxel spacing of all patients to 1×1×1 mm3 facilitates the learning of convolutional neural networks.

[0040] During the resampling process, the dimensions of each axis may change, so interpolation is used to make up for the extra voxels. The present disclosure adopts two interpolation methods: a B-Spline interpolation method and a nearest neighbor interpolation method. The B-Spline interpolation method can be applied to the T2-weighted and T1C-weighted images. The nearest neighbor interpolation method can be applied to the brain edema mask (i.e., the brain edema area).

[0041] Then, in one embodiment of the present disclosure, the pre-process includes image normalization.

[0042] The MRI data are taken sequentially by The MRI device 190 are different each time, and the gray scale value range of each data is not the same, which easily causes the convolutional neural network to make wrong judgments. The Z-score regularization method is used to rescale the grayscale value of each data within the actual brain range to data with a mean value of 0 and a standard deviation of 1. It enhances the consistency of each piece of data and helps improve the robustness of the model.

[0043] The formula defines z=(x−μ)/σ, where z is the normalized gray scale value, x is the original gray scale value, u is the average gray scale value, and σ is the standard deviation of the gray scale value.

[0044] Regarding the way to identify and quantify the brain edema area, In one embodiment of the present disclosure, the processor 120 accesses and executes the at least one instruction for: pre-training the three-dimensional convolutional neural network model, where the three-dimensional convolutional neural network model has two convolution paths, one of the two convolution paths extracts each region in training data so as to perform a feature extraction on the each region, another of the two convolution paths selects each corresponding expanded area (i.e., a larger area than the original area) based on a center point of the each area and performs an under-sampling on the each corresponding expanded area for the feature extraction.

[0045] For example, the network architecture for segmenting brain edema and tumors is a three-dimensional convolutional neural network (e.g., DeepMedic), and can accept multiple modalities. It was mentioned in the experiment that a model with good performance can be trained using only a data set of 28 subjects tracked multiple times.

[0046] The network architecture used by the three-dimensional convolutional neural network has two convolution paths, one of which is to extract an area in the data to extract features. Another path can expand and select a larger area from the center point of the original area, and perform under-sampling to reduce the resolution, and perform the same feature extraction. In this way, the network will learn the subtler and rougher features of the same area, to improve performance of the edema segmentation.

[0047] There is a total of 87 data sets in the segmented brain edema model, including images of patients before and after treatment. Among them, 53 data are used as the training set, 17 data are used as the verification set, and 17 data are used as the test set. And try to put the tracking images of the same patients in the same set to avoid the images in the test set being similar to the images in the training set.

[0048] For a more complete understanding of an operation method of the identifying and quantizing system 100, referring FIGS. 1-2, FIG. 2 is a flow chart of the operation method 200 of the identifying and quantizing system 100 according to one embodiment of the present disclosure. As shown in FIG. 2, the operation method 200 includes operations S201-S204. However, as could be appreciated by persons having ordinary skill in the art, for the steps described in the present embodiment, the sequence in which these steps are performed, unless explicitly stated otherwise, can be altered depending on actual needs; in certain cases, all or some of these steps can be performed concurrently.

[0049] The operation method 200 may take the form of a computer program product on a computer-readable storage medium having computer-readable instructions embodied in the medium. Any suitable storage medium may be used including non-volatile memory such as read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), and electrically erasable programmable read only memory (EEPROM) devices; volatile memory such as SRAM, DRAM, and DDR-RAM; optical storage devices such as CD-ROMs and DVD-ROMs; and magnetic storage devices such as hard disk drives and floppy disk drives.

[0050] In operation S201, a T2 weighted image is split into a first group of two-dimensional images. In operation S202, the first group of two-dimensional images is inputted into a mask R-CNN model to obtain a first group of two-dimensional parenchymal brain images, and a first group of two-dimensional parenchymal brain images is used to form a T2 weighted parenchymal brain images. In operation S203, a pre-process is performed on the T2 weighted parenchymal brain image to obtain a pre-processed T2 weighted parenchymal brain image. In operation S204, a three-dimensional convolutional neural network model (i.e., a three-dimensional convolutional neural network model trained in advance with a large number of other T2 weighted parenchymal brain images) is used to segment and quantize a brain edema area in the pre-processed T2 weighted parenchymal brain image.

[0051] In some embodiments of the present disclosure, in operation S201, a T1C weighted image is co-registered to the T2 weighted image to obtain a co-registered T1C weighted image, and the co-registered T1C weighted image is split into a second group of two-dimensional images; in operation S202, the second group of two-dimensional images is inputted into the mask R-CNN model to obtain a second group of two-dimensional parenchymal brain images, and the second group of two-dimensional parenchymal brain images is used to form a T1C weighted parenchymal brain image; in operation S203, the pre-process is performed on the T1C weighted parenchymal brain image to obtain a pre-processed T1C weighted parenchymal brain image; in operation S204, another three-dimensional convolutional neural network model (i.e., a three-dimensional convolutional neural network model trained in advance with a large number of other T1C weighted parenchymal brain images) is used to segment and quantize a metastatic brain tumor area in the pre-processed T1C weighted parenchymal brain image.

[0052] In some embodiments of the present disclosure, in operation S203, the image resampling is performed on the T2 weighted parenchymal brain image.

[0053] In some embodiments of the present disclosure, in operation S203, the image normalization is performed on the T2 weighted parenchymal brain image.

[0054] Regarding a pre-training method for the three-dimensional convolutional neural network model, in some embodiments of the present disclosure, the operation method 200 further includes: pre-training the three-dimensional convolutional neural network model, where the three-dimensional convolutional neural network model has two convolution paths, one of the two convolution paths extracts each region in training data so as to perform a feature extraction on the each region, another of the two convolution paths selects each corresponding expanded area based on a center point of the each area and performs an under-sampling on the each corresponding expanded area for the feature extraction.

[0055] In view of the above, technical advantages are generally achieved, by embodiments of the present disclosure. With the identifying and quantizing system 100 and its operation method 200 of the present disclosure, the technology of image interpretation and segmentation using artificial intelligence allows computers to automatically segment not only metastatic brain tumors, but also brain edema in adjacent areas. With the assistance of artificial intelligence, clinicians can not only quickly and without omissions find the lesions that need treatment, but also quantitatively analyze the possible brain edema area before and after treatment, thereby improving the accuracy and safety of treatment.

[0056] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims.