Smart image processing method and device using same

[0001] The present application is a continuation of International Patent Application No. PCT/KR2020/018858, filed on Dec. 22, 2020, which is based upon and claims the benefit of priority to Korean Patent Application No. 10-2020-0180493 filed on Dec. 22, 2020. The disclosures of the above-listed applications are hereby incorporated by reference herein in their entirety.

[0002] The inventive concept relates to a smart image processing method and a smart image processing device using the same. More specifically, the inventive concept relates to a smart image processing method for selectively converting and storing personal sensitive information in a captured image and a smart image processing device using the same.

[0003] With the issue of invasion of personal privacy, it is prohibited to install security cameras in spaces where sensitive personal information may be exposed, such as changing rooms and toilets in Korean dry sauna, bathhouses, and gyms and capture the spaces. Due to this, there is no means to check whether a crime or an emergency has occurred in the corresponding space, and cases of abuse are increasing. As a method for solving this problem, Korean Patent Laid-Open Publication No. 10-2007-0113661 discloses a method of detecting a person face area in a captured image and simply mosaicizing, binarizing, and black-and-whitening other areas than the face area. However, according to the above prior art, the following problems may occur.

[0004] There is a possibility of restoring the original image, and privacy of individuals is not perfectly protected with only black and white, binary, and mosaic.

[0005] By transforming the entire background area except for the face, there is a risk of losing necessary information as well.

[0006] Embodiments of the inventive concept provide a smart image processing method and a smart image processing device using the same capable of solving the above problems.

[0007] According to an embodiment, a smart image processing device includes an image acquisition unit that acquires an image for a predetermined area through a camera, an image conversion unit that performs conversion from a body image of a person in the image into an image of the person wearing outer clothing when the person is detected in the acquired image, and a storage unit that stores the converted image.

[0008] In this case, the image conversion unit may obtain skeleton data based on joints or movement points of the person by applying a skeleton modeling technique to the person detected in the acquired image, and perform conversion from the body image of the person into an image of the person wearing clothes based on the skeleton data.

[0009] Further, the image conversion unit may detect an image of one or more persons located in the predetermined area in the image based on a first deep learning algorithm, and perform the conversion only on a person determined not to be wearing outer clothing in the person image.

[0010] Further, the image conversion unit may determine that at least one of a naked person and a person wearing only underwear is not wearing outer clothing.

[0011] Further, the image conversion unit may perform conversion from the body image of the person into may process of the person wearing outer clothing based on a second deep learning algorithm different from the first deep learning algorithm by using a skeleton modeling technique for the naked person and the person wearing only underwear.

[0012] Further, the outer clothing includes clothes capable of covering at least one of all parts of the body image except a face, disliked parts that cause discomfort to others when exposed of an upper body of the body image except the face, and disliked parts of a lower body of the body image except the face.

[0013] According to an embodiment, a smart image processing method to be performed by a device includes acquiring, by an image acquisition unit of the device, an image for a predetermined area, performing, by an image conversion unit of the device, conversion from a body image of a person in the image into an image of the person wearing outer clothing when the person is detected in the acquired image, and storing, by a storage unit of the device, the converted image.

[0020] Advantages and features of the inventive concept and methods for achieving them will be apparent with reference to embodiments described below in detail in conjunction with the accompanying drawings. However, the inventive concept is not limited to the embodiments disclosed below, but can be implemented in various forms, and these embodiments are to make the disclosure of the inventive concept complete, and are provided so that this disclosure will be thorough and complete and will fully convey the scope of the invention to those of ordinary skill in the art, which is to be defined only by the scope of the claims.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the inventive concept. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms “comprises” and/or “comprising” are intended to specify the presence of stated elements, but do not preclude the presence or addition of elements Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although “first”, “second”, and the like are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Thus, a first element discussed below could be termed a second element without departing from the teachings of the inventive concept.

[0022] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms such as those defined in commonly used dictionaries, will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0023] Hereinafter, embodiments of the inventive concept will be described with reference to the accompanying drawings.

[0024] The inventive concept provides a method of processing a captured image through a deep learning algorithm.

[0025] Before the description, the meaning of the terms used herein will be briefly described. However, it should be noted that the description of terms is intended to help the understanding of the present specification and is not used in the sense of limiting the technical spirit of the inventive concept unless explicitly described as limiting the inventive concept.

[0026] First, a deep learning algorithm is one of machine learning algorithms and refers to a modeling technique developed from an artificial neural network modeled after a person neural network. An artificial neural network may be configured in a multi-layered structure as shown in FIG. 1.

[0027] FIG. 1 is a diagram briefly illustrating the basic concept of an artificial neural network.

[0028] As shown in FIG. 1, an artificial neural network (ANN) may have a layered structure including an input layer, an output layer, and at least one middle layer (or hidden layer) between the input layer and the output layer. The deep learning algorithm may derive highly reliable results through learning that optimizes weights of activation functions between layers based on such a multi-layered structure.

[0029] Deep learning algorithms applicable to the inventive concept may include a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), and the like.

[0030] The DNN is basically characterized by improving learning results by increasing the number of middle layers (or hidden layers) in the existing ANN model. For example, the DNN is characterized by performing a learning process using two or more middle layers. Accordingly, a computer may derive an optimal output value by repeating a process of generating classification labels, distorting space, and classifying data by the computer itself.

[0031] The CNN is characterized by having a structure of extracting features of data and figuring out a pattern of the features, unlike existing techniques in which a learning process is performed by extracting knowledge from data. The CNN may be performed through a convolution process and a pooling process. In other words, the CNN may include an algorithm complexly composed of a convolution layer and a pooling layer. Here, a process of extracting features of data (so-called a convolution process) is performed in the convolution layer. The convolution process is a process of examining adjacent components of each component in data to identify characteristics and deriving the identified characteristics as one sheet as a single compression process to effectively reduce the number of parameters. In the pooling layer, a process of reducing the size of the layer that has undergone the convolution process (so-called a pooling process) is performed. The pooling process may reduce the size of data, cancel noise, and provide consistent features in fine parts. For example, the CNN may be used in various fields such as information extraction, sentence classification, and face recognition.

[0032] The RNN is a type of artificial neural network specialized in learning repetitive and sequential data and is characterized by having a circular structure therein. The RNN uses the circular structure to apply weights to past learning contents and reflect them to current learning, enabling a connection between current learning and past learning, and has the characteristic of being dependent on time. The RNN is an algorithm that solves the limitations in existing learning of continuous, iterative, and sequential data, and may be used for recognizing a speech waveform or identifying front and rear components of text.

[0033] However, these are only examples of specific deep learning techniques applicable to the inventive concept, and other deep learning techniques may be applied to the inventive concept according to embodiments.

[0034] An image described below may include any one of a still image such as a photograph, a video, and each still image corresponding to each frame constituting the video.

[0035] FIG. 2 is a schematic diagram illustrating a smart image processing method according to the inventive concept.

[0036] Referring to FIG. 2, a smart image processing method according to the inventive concept may include an image acquisition step (S210), an image conversion step (S220), and a storage step (S230).

[0037] For convenience of description below, it is assumed that the smart image processing method according to the inventive concept is performed by a smart image processing device. In this case, according to embodiments, the smart image processing device may be included in a security camera system as a module included in a security camera (e.g., a closed-circuit television (CCTV)) or, conversely, may include a security camera system.

[0038] In S210, the smart image processing device may acquire an image of a predetermined area. Here, the predetermined area may include an area captured by the smart image processing device.

[0039] In S220, the smart image processing device may determine whether a person is detected in the acquired image, and when the person is detected in the acquired image, perform conversion from a body image of the person (video or still image) in the image into an image (video or still image) of the person wearing outer clothing.

[0040] The outer clothing may cover at least one of all parts of the body image except the face, disliked parts of the upper body in the body image other than the face (parts that cause discomfort to others when exposed), and disliked parts of the face and the lower body in the body image.

[0041] The range of the outer clothing may vary widely. For example, outer clothing is a garment designed to cover all or part of the body except the face, as described above, and may include a single garment such as a Barbary coat or a dress, or costumes for well-known characters such as Spider-Man, or may include garments such as a typical long coat, long padding and a sweater. Of course, the outer clothing according to the inventive concept is not limited thereto, and may include clothes for covering all or some of parts of the body image except for the face as described above.

[0042] More specifically, the smart image processing device according to an embodiment of the inventive concept may perform image conversion processing as described below.

[0043] For example, the smart image processing device may perform conversion from the body image of the person into an image (video or still image) of the person wearing clothes by applying a skeleton modeling technique to the person detected in the image acquired through S210.

[0044] In this case, the body image of the person and the image of the person wearing clothes may be a video or each of still images of frames constituting the video.

[0045] FIG. 3 is a diagram illustrating an example of extracting skeleton data from a person detected in an image acquired according to the inventive concept.

[0046] Referring to FIG. 3, the smart image processing device of the inventive concept may obtain skeleton (model) data considering a person's joints or movement points by applying a skeleton modeling technique to a person detected in an image. Through the skeleton (model) data, the smart image processing device may more thoroughly figure out the posture of a corresponding person. Through this, it is possible to convert the body image of the person into an image of the person wearing clothes (e.g., a clothing image, etc.) (video or still image).

[0047] FIGS. 4A to 4C are diagrams showing examples of image conversion phases according to the inventive concept.

[0048] First, as shown in FIG. 4A, the smart image processing device may acquire an image of a predetermined area.

[0049] Subsequently, as shown in FIG. 4B, the smart image processing device may determine whether a person is detected in the acquired image, and when the person is detected in the acquired image, apply a skeleton modeling technique to the person. Through this, the smart image processing device may obtain skeleton (model) data of the person as shown in FIG. 4B.

[0050] Thereafter, as shown in FIG. 4C, the smart image processing device may convert a body image of a person into an image (video or still image) of the person wearing clothes based on the skeleton (model) data.

[0051] To this end, the smart image processing device may perform step S220 as described below.

[0052] More specifically, the smart image processing device may perform (i) a person image detection step of detecting a person image for one or more persons in the image acquired through step S210, (ii) a person image conversion step of performing conversion from the person image detected in the image into an image of a person wearing outer clothing.

[0053] As an example applicable to the inventive concept, the smart image processing device may perform the person image detection operation and the person image conversion operation through a single deep learning algorithm. In other words, the smart image processing device may provide the image acquired through step S210 to the single deep learning algorithm as input information, detect an image of a person through the single deep learning algorithm, and perform conversion from an image of the detected person into an image of a person wearing outer clothing.

[0054] As another example applicable to the inventive concept, the smart image processing device may perform the person image detection step and the person image conversion step through different deep learning algorithms. In other words, the smart image processing device may detect a person image by providing the image acquired through step S210 as input information to a first deep learning algorithm, and perform conversion into the detected person image into an image of a person wearing outer clothing by providing the detected person image as input information to a second deep learning algorithm.

[0055] Through the above-described method, the smart image processing device according to the inventive concept may perform conversion from a body image of all persons in the acquired image into an image of all persons wearing clothes.

[0056] As another example, the smart image processing device according to the inventive concept may perform conversion only on a body image of a person satisfying a predetermined condition among persons in the acquired image.

[0057] More specifically, the smart image processing device may perform image conversion operation according to step S220 only on a person determined not to be wearing outer clothing among one or more persons detected in the image. In this case, the person determined not to be wearing outer clothing may include at least one of a person determined to be naked and a person determined to be wearing only underwear.

[0058] In particular, as in the above-described embodiment, the operation of detecting a person who satisfies a predetermined condition in the smart image processing device may include detecting a person determined to be naked and/or a person determined to be wearing only underwear such that there is no false-negative as much as possible.

[0059] In step S230, the smart image processing device may store the image converted through step S220. In other words, the smart image processing device may store the image converted as shown in FIG. 4C in a separate storage device.

[0060] Additionally, the smart image processing device according to the inventive concept may display an image of a predetermined area captured by a separate camera device on a separate display device (e.g., a monitor, a mobile phone terminal, a smart phone, or the like). In this case, the smart video image processing device may perform control to display the converted image by providing (transmitting) the image converted through step S220 to the separate display device.

[0061] FIGS. 5 to 8 are schematic diagrams illustrating a smart image processing device according to the inventive concept.

[0062] Referring to FIGS. 5 to 8, the smart image processing device according to the inventive concept may have various configurations according to embodiments.

[0063] Basically, a smart image processing device 500 may include an image acquisition unit 510, an image conversion unit 520, and a storage unit 530 as shown in FIG. 5. Each component may operate as described below.

[0064] The image acquisition unit 510 may acquire an image of a predetermined area. For example, the image acquisition unit 510 may acquire the image from a separate camera device that captures an entire area including the predetermined area. Alternatively, as another example, the image acquisition unit 510 may acquire an image of the predetermined area by directly capturing the entire area including the predetermined area.

[0065] When a person is detected in the image acquired through the image acquisition unit 510, the image conversion unit 520 may perform conversion from a body image of the person in the image into an image of the person wearing outer clothing. As an example, the image conversion unit 520 may perform conversion from the body image of the person into an image of the person wearing clothes by applying a skeleton modeling technique to the person detected in the acquired image.

[0066] In the above example, the image conversion unit 520 may be configured in one phase as shown in FIG. 5 through a deep learning algorithm trained to enable the above-described image conversion with one deep learning model.

[0067] As an additional example, the image conversion unit 520 may have an object selection function. Through this, the image conversion unit 520 may select an object for image conversion and perform image conversion only on the object.

[0068] The storage unit 530 may store the image converted through the image conversion unit 520.

[0069] As another example, a smart image processing device 600 may include an image acquisition unit 610, an image conversion unit 620, and a storage unit 630 as shown in FIG. 6, and the image conversion unit 620 may include a conversion target selection unit 622 and an target-specific image conversion unit 624. Each component may operate as described below.

[0070] The image acquisition unit 610 may operate in the same manner as the image acquisition unit 510 described above.

[0071] When a person is detected in the image acquired through the image acquisition unit 610, the image conversion unit 620 may perform conversion from a body image into an image of a person wearing outer clothing only for a person determined not to be wearing outer clothing among one or more persons detected in the image.

[0072] To this end, the image conversion unit 620 may include the conversion target selection unit 622 that detects a person determined not to be wearing outer clothing among one or more persons detected in the image and the target-specific image conversion unit 624 that performs conversion from the body image of a person selected by the conversion target selection unit 622 into the image of a person wearing outer clothing.

[0073] In the inventive concept, the person determined not wearing outer clothing may include at least one of a person determined naked and a person determined to be wearing only underwear.

[0074] As described above, the image conversion unit 620 may include the conversion target selection unit 622 and the target-specific image conversion unit 624, and may be configured in two phases as shown in FIG. 6.

[0075] The storage unit 630 may operate in the same manner as the storage unit 530 described above.

[0076] As another example, a smart image processing device 700 may include an image acquisition unit 710, an image conversion unit 720, and a storage unit 730 as shown in FIG. 7, and the image conversion unit 720 may include a skeleton extraction unit 722 and an overlap unit 724. Each component may operate as described below.

[0077] The image acquisition unit 710 may operate in the same manner as the image acquisition unit 510 described above.

[0078] When a person is detected in the image acquired through the image acquisition unit 710, the image conversion unit 720 may perform conversion from the body image of the person into an image of a person wearing clothes by applying a skeleton modeling technique to the person detected in the image. To this end, the image conversion unit 720 may include the skeleton extraction unit 722 for deriving skeleton (model) data by applying a skeleton modeling technique to one or more people detected in the image and the overlap unit 724 for performing conversion from the body image of the person into an image of the person wearing clothes.

[0079] As such, the image conversion unit 720 may be configured in two phases as shown in FIG. 7 by including the skeleton extraction unit 722 and the overlap unit 724.

[0080] As an additional example, the above-described skeleton extraction unit 722 may have a target selection function. Through this, the skeleton extraction unit 722 may select a target from which skeleton (model) data is to be extracted among the acquired images, and extract skeleton (model) data only for the target.

[0081] The storage unit 730 may operate in the same manner as the storage unit 530 described above.

[0082] As another example, a smart image processing device 800 includes an image acquisition unit 810, an image conversion unit 820, and a storage unit 830, as shown in FIG. 8, and the image conversion unit 820 may include a conversion target selection unit 822, a skeleton extraction unit 824, and an overlap unit 826. Each component may operate as described below.

[0083] The image acquisition unit 810 may operate in the same manner as the image acquisition unit 510 described above.

[0084] When a person is detected in the image acquired through the image acquisition unit 810, the image conversion unit 820 may detect a person determined not to be wearing outer clothing among one or more persons detected in the image and perform conversion from a body image into an image of a person wearing outer clothing by applying the skeleton modeling technique only to the detected person. To this end, the image conversion unit 820 may include the conversion target selection unit 822 that detects a person determined not to be wearing outer clothing among one or more persons detected in the image, the skeleton extraction unit 824 that derives skeleton (model) data by applying the skeleton modeling technique to only the detected person, and the overlap unit 826 that perform conversion from a body image of the detected person into an image of the person wearing clothes based on the skeleton (model) data.

[0085] As described above, the image conversion unit 820 may be configured in three phases as shown in FIG. 8 by including the conversion target selection unit 822, the skeleton extraction unit 824, and the overlap unit 826.

[0086] The storage unit 830 may operate in the same manner as the storage unit 530 described above.

[0087] In addition, the smart image processing devices 500, 600, 700, and 800 according to the inventive concept may perform various smart image processing methods described above. To this end, the smart image processing devices 500, 600, 700, and 800 may control each component of the smart image processing devices 500, 600, 700, and 800 according to the various smart image processing methods.

[0088] Additionally, the computer program according to the inventive concept may be stored in a computer readable recording medium to execute various smart video image processing methods described above in combination with a computer.

[0089] The above-described program may include codes coded in a computer language, such as C, C++, JAVA, or a machine language, which are readable by a processor (CPU) of the computer through a device interface of the computer such that the computer reads the program and execute the methods implemented by the program. The codes may include functional codes associated with a function defining functions necessary to execute the methods or the like, and include control codes associated with an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, the codes may further include memory reference codes indicating at which location (address number) of the computer's internal or external memory, additional information or media required for the computer's processor to execute the functions can be referenced. In addition, when the processor of the computer needs to communicate with any other computer or server located remotely to execute the above functions, codes may further include communication-related codes for how to communicate with any other remote computer or server using a communication module of the computer, and what information or media to transmit/receive during communication.

[0090] The steps of a method or algorithm described in connection with the embodiments of the inventive concept may be implemented directly in hardware, in a software module executed by hardware, or in a combination thereof. The software module may reside in a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or in a computer readable recording medium that is well known in the art.

[0091] Although embodiments of the inventive concept have been described above with reference to the accompanying drawings, it is understood that those skilled in the art to which the inventive concept pertains may implement the inventive concept in other specific forms without changing the technical spirit or essential features thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

[0092] According to the inventive concept, a security camera according to the inventive concept can be installed even in a space where personal sensitive information is able to be exposed to capture the space. Through this, an individual's privacy can be protected not to be exposed and at the same time, the occurrence of a crime or dangerous situation in a corresponding area can be minimized.

[0093] However, effects of the inventive concept may not be limited to the above-described effects. Although not described herein, other effects of the inventive concept can be clearly understood by those skilled in the art from the following description.

[0094] While the inventive concept has been described with reference to embodiments, it will be apparent to those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the inventive concept. Therefore, it should be understood that the above embodiments are not limiting, but illustrative.