Labeling method and computing device by detecting lesion region in images using a learned network function

[0001] The present invention relates to a labeling method and a device therefor.

[0002] Recently, interests and demands for artificial intelligence have increased. Also, studies in the field of artificial intelligence have been developed. To perform supervised learning of an artificial neural network, above all, it is necessary to produce learning data. To produce the learning data, further, labeling for the learning data has to be performed. A large quantity of learning data is needed to enhance the accuracy of artificial intelligence, and if the labeling for the production of learning data is performed directly by a person, the time needed for the labeling increases exponentially.

[0003] Therefore, there is a definite need for a new method capable of decreasing the time and endeavor required for the labeling for learning data production.

[0004] Accordingly, it is an object of the present invention to provide a labeling method and a computing device therefor that are capable of labeling a lesion region on a medical image.

[0005] The technical problems of the multi-modality medical image analysis method and apparatus to be achieved through the present invention are not limited as mentioned above, and other technical problems not mentioned herein will be obviously understood by one of ordinary skill in the art through the following description.

[0006] According to an aspect of the present invention, a labeling method is provided. The labeling method may include the steps of acquiring a medical image, receiving the medical image, detecting regions of interest corresponding to at least one subject disease through a learned network function, and displaying the regions of interest on the medical image, and labeling a lesion region corresponding to the subject disease on the medical image according to the regions of interest and user inputs to the regions of interest.

[0007] The labeling method may further include the step of setting the subject disease according to a user input.

[0008] The labeling method may further include the step of correcting at least one of the regions of interest according to a user input and cropping and storing the corrected region of interest from the medical image.

[0009] Further, the step of labeling the lesion region corresponding to the subject disease may include the steps of: receiving a first user input for selecting a given region in the region of interest, setting a reference pixel value for the selected given region, detecting a region within the range of given pixel values from the reference pixel value as a lesion estimation region and displaying the detected lesion estimation region on the region of interest, and correcting the lesion estimation region according to a second user input to the lesion estimation region and thus labeling the lesion region on the region of interest.

[0010] Furthermore, the step of labeling the lesion region corresponding to the subject disease may further include the step of increasing a resolution for the region of interest, and the step of receiving the first user input may be performed after the step of increasing the resolution.

[0011] The step of labeling the lesion region corresponding to the subject disease may further include the step of returning the resolution of the region of interest on which the lesion region is labeled to an original resolution thereof.

[0012] The step of displaying the lesion estimation region may be performed by emphasizing the outer line of the lesion estimation region.

[0013] The labeling method may further include the step of storing the region of interest on which the lesion region is labeled.

[0014] According to another aspect of the present invention, a computing device for supporting labeling is provided. The computing device may include: at least one memory for storing a program for labeling, a communication unit for acquiring a medical image, and at least one processor for receiving the medical image, detecting regions of interest corresponding to at least one subject disease through a learned network function, and displaying the regions of interest on the medical image, and labeling a lesion region corresponding to the subject disease on the medical image according to the regions of interest and user inputs to the regions of interest.

[0015] Further, the processor may set the subject disease according to a user input.

[0016] Furthermore, the processor may correct at least one of the regions of interest according to a user input and crop and store the corrected region of interest from the medical image.

[0017] Also, the processor may receive a first user input for selecting a given region in the region of interest, set a reference pixel value for the selected given region, detect a region within the range of given pixel values from the reference pixel value as a lesion estimation region, display the detected lesion estimation region on the region of interest, correct the lesion estimation region according to a second user input to the lesion estimation region, and label the lesion region on the region of interest.

[0018] Moreover, the processor may increase a resolution for the region of interest and receive the first user input for selecting the given region in the region of interest whose resolution is increased.

[0019] Additionally, the processor may return the resolution of the region of interest on which the lesion region is labeled to an original resolution thereof.

[0020] Further, the processor may emphasizedly display the outer line of the lesion estimation region detected in the region of interest.

[0021] Moreover, the processor may store the region of interest on which the lesion region is labeled.

[0022] According to the present invention, the labeling method and the computing device therefor can detect the lesion estimation region corresponding to the subject disease through the learned network function, directly correct the lesion estimation region according to the user input, and more easily label the lesion region through the user.

[0023] The effectiveness of the labeling method and the computing device therefor according to the present invention is not limited as mentioned above, and it should be understood to those skilled in the art that the effectiveness of the disclosure may include another effectiveness as not mentioned above from the detailed description of the present invention.

[0031] The present invention may be modified in various ways and may have several exemplary embodiments. Specific exemplary embodiments of the present invention are illustrated in the drawings and described in detail in the detailed description. However, this does not limit the disclosure within specific embodiments and it should be understood that the disclosure covers all the modifications, equivalents, and replacements within the idea and technical scope of the disclosure.

[0032] In the description, if it is determined that the detailed explanation on the well-known technology related to the present invention makes the scope of the present invention not clear, the explanation will be avoided for the brevity of the description. Terms, such as the first, the second, and the like, may be used to describe various elements, but the elements should not be restricted by the terms. The terms are used to only distinguish one element from the other element.

[0033] When it is said that one element is described as being “connected” or “coupled” to the other element, one element may be directly connected or coupled to the other element, but it should be understood that another element may be present between the two elements.

[0034] The terms “unit”, “-or/er” and “module” described in the specification indicate a unit for processing at least one function or operation, which may be implemented by hardware, software or a combination thereof, such as a processor, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), an accelerate processor unit (APU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like.

[0035] Further, it should be appreciated that the division of the parts in the present invention is just made according to the principal functions the parts have. That is, two or more parts as will be discussed below, may be combined into one part, or one part may be divided into two or more parts according to more specified functions. Moreover, the respective parts as will be discussed in the specification, can additionally perform some or all of functions performed by other parts as well as their main functions, and of course, also, some of the main functions of the respective parts can be performed only by other parts.

[0036] Hereinafter, embodiments of the present invention will be described in detail sequentially.

[0037] In the description, a network function may be used with the same meaning as a neural network. In this case, the neural network is composed of interconnected calculation units, which are commonly called nodes, and the nodes are called neurons. Generally, the neural network is made up of a plurality of nodes. The nodes for constituting the neural network are connected to one another by means of one or more links.

[0038] In this case, some of the nodes constituting the neural network build one layer based on their distances from an initial input node. For example, a collection of nodes with the distances of n from the initial input node builds an n layer.

[0039] The neural network described in the present invention may include a deep neural network (DNN) having a plurality of hidden layers as well as input and output layers.

[0040] The network function includes a plurality of convolutional layers and a plurality of fully connected layers. The plurality of convolutional layers serve to perform the abstraction of the image to extract features, and the plurality of fully connected layers serve to predict the output probabilities of the detected objects.

[0041] FIG. 1 is a flowchart showing a labeling method according to the present invention.

[0042] A labeling method 100 according to an embodiment of the present invention may be performed by a personal computer, a work station, a server computer, and the like, which have operation ability, or by a separate device therefor.

[0043] Further, the labeling method 100 may be performed in one or more operation devices. For example, at least one or more steps of the labeling method 100 according to an embodiment of the present invention are performed in a client device, and other steps in a server device. In this case, the client device and the server device are connected to each other by a network and transmit and receive the results of operations to and from each other. Otherwise, the labeling method 100 according to the present invention may be performed through distributed computing.

[0044] At step S110, a medical image is acquired by a computing device. In this case, the medical image is a radiographic image, a computed tomography image, or the like for at least an area of a patient's body, but of course, the medical image may not be limited thereto.

[0045] For example, the computing device receives the patient's medical image from an external computing device connected thereto by means of wired and wireless communication, another device (e.g., medical imaging device) interworking therewith, or an external storage medium.

[0046] At step S120, at least one subject disease is set by the computing device according to user inputs. In this case, the subject disease is an object on which labeling is performed by the user, so that a lesion region is labeled on the medical image.

[0047] For example, the computing device provides a given user interface for the selection of the subject disease, and through the user interface, at least one of a plurality of diseases is selected as the subject disease by the user.

[0048] After the medical image has been received, at step S130, the computing device detects a region of interest corresponding to the at least one subject disease through a learned network function and displays the region of interest on the medical image. In this case, the region of interest represents a region on which features or specific anatomical features the lesion region of a specific disease has are included.

[0049] The network function has pre-learning for detection of the region of interest and/or lesion region through learning data (for example, medial images on which the regions of interest or lesion regions are detected by a specialist or other network functions).

[0050] In the embodiment of the present invention, the network function detects the region of interest from the medical image to the form of a boundary box and displays the detected region of interest on the medical image.

[0051] At step S140, the computing device labels the lesion region corresponding to the subject disease on the medical image according to the detected region of interest and the user inputs to the region of interest.

[0052] For example, the computing device corrects the outer boundary line of the region of interest or limits a given region within the region of interest to the lesion region according to user inputs, thereby completing the labeling of the lesion region corresponding to the subject disease on the medical image.

[0053] The step S140 will be explained in more detail later with reference to FIGS. 2 and 3.

[0054] At step S150, the labeled region of interest and/or medical image are stored in a memory of the computing device or in a database server connected to the computing device through communication.

[0055] In the embodiment of the present invention, the computing device crops and sortedly stores the region of interest on which the lesion region is labeled from the medical image or stores the entire medical image on which the lesion region is labeled, according to user inputs. In the embodiment of the present invention, while the region of interest and/or the medical image are being stored, image equalization, brightness adjustment, size control, blur application, sharpening application, conversion of black-and-white image to color image, RGB value designation, and the like may be performed by the computing device according to user inputs.

[0056] The stored region of interest and/or medical image may be used as learning data for a second network function adapted to read lesions of the subject disease.

[0057] FIG. 2 is a flowchart showing an embodiment of step S140 of FIG. 1.

[0058] Referring to FIG. 2, the step S140 of FIG. 1 includes steps S210 to S240.

[0059] At step S210, the computing device receives a first user input for selecting a given region in the region of interest.

[0060] For example, the first user input is an input for the selection of the given region in the region of interest through a mouse cursor. In specific, the user checks the region of interest displayed on the medical image and directly selects the given region within the region of interest determined as the lesion region through the mouse curser.

[0061] At step S220, the computing device sets a reference pixel value for the given region selected at the step S210. In this case, the pixel value represents the discrete value for color, brightness, or other properties a given pixel has. In specific, the R, G, and B color values of the given pixel are set with integer values within a given range (e.g., 0 to 360), and the brightness value of the given pixel is set with an integer value within a given range (e.g., 0 to 100).

[0062] At the step S220, accordingly, the color value or brightness value of the given region selected by the user input is set as the reference pixel value by the computing device.

[0063] At step S230, a region within the range of given pixel values from the reference pixel value with respect to the given region selected by the user input is detected as a lesion estimation region by the computing device, and next, the lesion estimation region is displayed on the region of interest and/or the medical image by the computing device.

[0064] The region corresponding to the lesion on the medical image may have similar properties of colors and brightness to other regions, and accordingly, the reference pixel value is set at a point determined as the lesion region by the user. Next, if the region within the range of given pixel values from the corresponding point is detected, it is likely that the detected region corresponds to the lesion region. Accordingly, the detected region is detected as the lesion estimation region and displayed on the region of interest and/or the medical image by the computing device.

[0065] For example, if the reference pixel value is set to 100, the region with pixel values in the range of +/−2 from the reference pixel value 100, that is, the region with the pixel values 98 to 102 is detected as the lesion estimation region.

[0066] In the embodiment of the present invention, the outer line of the lesion estimation region is displayed emphasizedly with a given color or thickness.

[0067] In the embodiment of the present invention, a plurality of lesion estimation regions are detectable by the computing device. In specific, the pixel value regions within different ranges with respect to the reference pixel value are detected as the lesion estimation regions. For example, if the reference pixel value is set to 100, the region with the pixel values in the range of +/−2 from the reference pixel value 100, that is, the region with the pixel values 98 to 102 is detected as a first lesion estimation region, and the region with the pixel values in the range of +/−4 from the reference pixel value 100, that is, the region with the pixel values 96 to 104 is detected as a second lesion estimation region. The detected lesion estimation regions are emphasized on their outer line and then displayed on the region of interest and/or the medical image. Accordingly, the outer lines of the plurality of lesion estimation regions are displayed on the region of interest and/or the medical image to the form of contour lines.

[0068] At step S240, the lesion estimation region is corrected by the computing device according to a second user input to the lesion estimation region to thus label the lesion region on the region of interest and/or the medical image.

[0069] For example, if a portion of the outer line of the lesion estimation region is dragged by the user through the mouse cursor, the portion is moved, enlarged, reduced, or deformed by the computing device so that the lesion estimation region is corrected to label the lesion region on the region of interest and/or the medical image. The labeling information of the lesion region includes the information of a disease, a progression degree, and the like corresponding to the lesion region.

[0070] FIG. 3 is a flowchart showing another embodiment of step S140 of FIG. 1.

[0071] Referring to FIG. 3, the step S140 of FIG. 1 includes steps S310 to S360. The steps S320 to S350 of FIG. 3 are the same as the steps S210 to S240 of FIG. 2, and for the brevity of the description, differences between the two embodiments will be explained.

[0072] At step S310, a resolution for at least one of detected regions of interest is increased by the computing device. For example, if one of the regions of interest displayed on the medical image is selected by the user, the resolution for the selected region is increased by the application of a subpixel method.

[0073] Next, the steps S320 to S350 are performed for the region of interest whose resolution is increased. Accordingly, the user performs the labeling for the lesion region more accurately and easily.

[0074] If the labeling for the region of interest whose resolution is increased is completed, at step S360, the resolution of the region of interest is returned to its original resolution equal to the resolution of the medical image.

[0075] After that, as mentioned above, the medical image on which the lesion region is labeled is stored in the memory or the external database server and used as the learning data for the second network function adapted to read the lesion of the subject disease.

[0076] FIGS. 4 and 5 are exemplary views showing an example of operations of network functions in the labeling method according to the present invention.

[0077] Referring to FIGS. 4 and 5, the medical image 10 inputted to the network function 410 is a dental image with the teeth of a patient, and the subject disease is a dental disease such as dental caries, and the like. Further, the dental image is a cephalometric radiographic image, a panoramic radiographic image, or the like, but it may not be limited thereto.

[0078] As mentioned above, the network function 410 has pre-learning for the detection of the region of interest 20, 21 and/or the lesion region through learning data (e.g., the medical images from which the region of interest 20, 21 and/or the lesion region are detected by the specialist or other network functions). Further, the network function 410 is used to pre-display the region of interest 20, 21 or the lesion estimation region to assist the labeling, and accordingly, the neural network that places more emphasis on a detection speed than accuracy the network function used for real lesion reading has may be adopted as the network function 410.

[0079] If the medical image 10 is inputted to the network function 410 by the computing device, the network function 410 detects the region of interest 20, 21 in which the lesion region such as dental caries is included from the medical image 10, and the detected region of interest 20, 21 is displayed on the medical image 10 by the computing device. In this case, the region of interest 20, 21 is the same as the lesion estimation region, and the detected region of interest is corrected through given methods by the user, thereby performing the labeling of the lesion region on the medical image 10.

[0080] FIGS. 6 and 7 are exemplary views showing another example of operations of network functions in the labeling method according to the present invention.

[0081] Referring to FIGS. 6 and 7, the computing device additionally detects the lesion estimation region 30 according to the first user input to the detected region of interest through the network function, and the detected lesion estimation region is corrected according to the second user input, thereby performing the labeling of the lesion region.

[0082] As shown in FIGS. 6 and 7, for example, if the computing device detects the region of interest 20 to a square box shape through the network function and displays the detected region of interest 20 on the medical image, the region of interest 20 is read directly by the user, and next, a given point (or region) determined as a lesion is selected through the mouse cursor.

[0083] Next, the computing device sets the reference pixel value according to the pixel value of the given point selected by the user, and the region having pixel values in a given range of the selected point of the user from the reference pixel value is detected as the lesion estimation region 30. In this case, the outer line of the detected lesion estimation region 30 is displayed emphasizedly with a given color or thickness.

[0084] Next, if the outer line of the lesion estimation region 30 is dragged by the user through the mouse cursor, the outer line is corrected by the computing device so that the corrected lesion estimation region 30 is labeled on the region of interest 20 and/or the medical image 10.

[0085] FIG. 8 is a schematic block diagram showing the computing device for performing the labeling method according to the present invention.

[0086] A communication unit 810 receives input data (medical image, etc.) for labeling. The communication unit 810 includes wired and wireless communication units. If the communication unit 810 includes the wired communication unit, the communication unit 810 includes one or more components for performing communication with a local region network (LAN), a wide region network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and a combination thereof. Further, if the communication unit 810 includes the wireless communication unit, the communication unit 810 transmits and receives data or signals wirelessly by using cellular communication, wireless LAN (e.g., Wi-Fi), and the like. According to the present invention, the communication unit 810 transmits and receives data or signals to and from an external device or external server under the control of a processor 840 as will be discussed later.

[0087] An input unit 820 receives various user commands through external control. To do this, the input unit 820 includes one or more input devices or is connected to the input devices. For example, the input unit 820 is connected to an interface for various inputs such as a keypad, a mouse, and the like and receives the user commands from the interface. To do this, the input unit 820 includes an interface such as a USB port, a Thunderbolt interface, and the like. Further, the input unit 820 includes various input devices such as a touch screen, a button, and the like or is connected to the input devices to receive the user commands from the outside.

[0088] A memory 830 stores programs and/or program commands for operating the processor 840 and temporarily or permanently stores data inputted and outputted. The memory 830 includes at least one storage medium of a flash memory, a hard disc, a multimedia card micro storage medium, a card type memory (e.g., SD or XD memory), random access memory (RAM), a static RAM (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disc, and an optical disc.

[0089] Further, the memory 830 stores various network functions and algorithms, while storing various data, programs (with one or more instructions), applications, software, commands, and codes for operating and controlling the computing device 800 according to the present invention.

[0090] The processor 840 controls all of the operations of the computing device 800. The processor 840 executes one or more programs stored in the memory 830. The processor 840 represents a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor through which the method, according to the technical ideas of the present invention, is performed.

[0091] According to the embodiment of the present invention, the processor 840 acquires a medical image, receives the medical image, detects regions of interest corresponding to at least one subject disease through a learned network function, displays the regions of interest on the medical image, and labels a lesion region corresponding to the subject disease on the medical image according to the regions of interest and user inputs to the regions of interest.

[0092] The labeling method, according to the embodiments of the present invention, may be implemented in the form of a program instruction that can be performed through various computing means, and the labeling method may be recorded in a computer readable recording medium. The computer-readable medium may include a program command, a data file, a data structure, and the like, independently or in combination. The program instruction recorded in the recording medium is specially designed and constructed for the present invention, but it may be well known to and may be used by those skilled in the art of computer software. The computer-readable recording medium may include a magnetic medium such as a hard disc, a floppy disc, and a magnetic tape, an optical recording medium such as a compact disc read-only memory (CD-ROM) and a digital versatile disc (DVD), a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a read-only memory (ROM), a random access memory (RAM), and a flash memory. Further, the program command may include a machine language code generated by a compiler and a high-level language code executable by a computer through an interpreter and the like.

[0093] Further, the labeling method, according to the disclosed embodiments of the present invention, is included in a computer program product. The computer program product as a product may be traded between a seller and a buyer.

[0094] The computer program product may include an S/W program and a computer-readable storage medium in which the S/W program is stored. For example, the computer program product may include an S/W program type product (e.g., downloadable app) electronically distributed through a manufacturing company of an electronic device or electronic market (e.g., Google play store, an app store, etc.). To do such electronic distribution, at least a portion of the S/W program may be stored in the storage medium or temporarily produced. In this case, the storage medium may be a storage medium of a server of the manufacturing company, a server of the electronic market, or a broadcast server for temporarily storing the S/W program.

[0095] The computer program product may include a storage medium of a server or a storage medium of a client device in a system composed of the server and the client device. If a third device (e.g., smartphone) connected to the server or client device exists, the computer program product may include a storage medium of the third device. Otherwise, the computer program product may include an S/W program itself transmitted from the server to the client device or the third device or from the third device to the client device.

[0096] In this case, one of the server, the client device, and the third device executes the computer program product to perform the method according to the embodiments of the present invention. Further, two or more devices of the server, the client device and the third device execute the computer program product to distributedly perform the method according to the embodiments of the present invention.

[0097] For example, the server (e.g., a cloud server or artificial intelligence server) executes the computer program product stored therein and controls the client device connected thereto to perform the method according to the embodiments of the present invention.

[0098] While the present invention has been described with reference to the particular illustrative embodiments, it is not to be restricted by the embodiments but only by the appended claims. It is to be appreciated that those skilled in the art can change or modify the embodiments without departing from the scope and spirit of the present invention.