Method for segmenting sign language into morphemes, method for predicting morpheme positions, and method for augmenting data

[0001] This application is a U.S. National Stage Application of International Application No. PCT/KR2022/008665, filed on Jun. 20, 2022, which claims the benefit under 35 USC 119(a) and 365(b) of Korean Patent Application No. 10-2021-0084889, filed on Jun. 29, 2021, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes.

[0002] The disclosure relates to artificial intelligence (AI) technology, and more particularly, to a method for segmenting sign language in a sign language video into morphemes by using an AI model, and predicting frames in which segmented morphemes are positioned.

[0003] Sign languages refers to language that the deaf and dumb use to communicate with each other. With the rapid development of artificial intelligence (AI) technology, it is possible to automatically recognize sign language from a sign language video.

[0004] In order to create a sign language translation model of high performance, a huge amount of sign language data is required. A learning-based model that translates sign languages into language may require much data, but there is a limit to resources for constructing data and much time and cost are required. Sign language that a sign language translation model can translates into language is limited due to limited sign language data, and is very dependent on constructed data.

[0005] There is a related-art method for predicting enumerated words corresponding to sign language morphemes in a sign language video, but this method may not predict in which section each morpheme is generated in a sign language video. This may degrade accuracy of sign language translation.

[0006] The disclosure has been developed in order to address the above-discussed deficiencies of the prior art, and an object of the disclosure is to provide a method and a system for recognizing morphemes of a sign language video on a frame basis based on a skeleton model, segmenting sign language into morphemes and predicting positions of morphemes, and augmenting sign language data for training by combining morphemes acquired as a result of predicting.

[0007] According to an embodiment of the disclosure to achieve the above-described object, a sign language analysis system may include: a recognition unit configured to recognize key points of a speaker from a sign language video; and a prediction unit configured to input the recognized key points into an AI model, and to segment sign language into morphemes and to predict position information of the segmented morphemes.

[0008] The recognition unit may recognize key points of a speaker from a sign language video on a frame basis, and the prediction unit may predict position information of morphemes on a frame basis.

[0009] The position information of the morphemes may be information on frames in which morphemes are positioned among frames constituting the sign language video.

[0010] The key points may include joint information of body and hand of the speaker and facial expression information of the speaker.

[0011] The AI model may receive key points on a frame basis, may predict which morpheme a current frame corresponds to, and may output a result of predicting.

[0012] According to an embodiment of the disclosure, the sign language analysis system may further include a training unit configured to recognize key points of a speaker from a sign language video for training, and to train the AI model by using the recognized key points and morpheme information of the sign language video for training.

[0013] According to an embodiment of the disclosure, the sign language analysis system may further include an augmentation unit configured to augment training data by combining morphemes outputted from the prediction unit.

[0014] In addition, the augmentation unit may augment the training data by changing an order of the morphemes.

[0015] In addition, the augmentation unit may augment the training data by deleting some of the morphemes.

[0016] According to another embodiment of the disclosure, a sign language analysis method may include: recognizing key points of a speaker from a sign language video; and inputting the recognized key points into an AI model, and segmenting sign language into morphemes and predicting position information of the segmented morphemes.

[0017] According to still another embodiment of the disclosure, a sign language analysis system may include: a training unit configured to recognize key points of a speaker from a sign language video for training, and to train an AI model by using the recognized key points and morpheme information of the sign language video for training; and a prediction unit configured to recognize key points of a speaker from a sign language video, to input the recognized key points into the AI model which is trained by the training unit, to segment sign language into morphemes, and to predict position information of the segmented morphemes.

[0018] According to yet another embodiment of the disclosure, a sign language analysis method may include: recognizing key points of a speaker from a sign language video for training, and training an AI model by using the recognized key points and morpheme information of the sign language video for training; and recognizing key points of a speaker from a sign language video, inputting the recognized key points into the AI model which is trained by the training step, segmenting sign language into morphemes, and predicting position information of the segmented morphemes.

[0019] According to embodiments of the disclosure as described above, by recognizing morphemes of a sign language video on a frame basis based on a skeleton model and thereby segmenting sign language into morphemes and predicting positions of morphemes, it is possible to lay the foundations for accurate sign language translation.

[0020] In addition, according to embodiments of the disclosure, by augmenting sign language data for training by variously combining predicted morphemes, training of a sign language recognition model can be reinforced and accuracy of prediction can be further enhanced.

[0031] Hereinafter, the disclosure will be described in more detail with reference to the drawings.

[0032] Embodiments of the disclosure propose a method for segmenting sign language in a sign language video into morphemes by using an AI model, predicting frames in which segmented morphemes are positioned, and augmenting training data of the AI model.

[0033] Sign language has different meanings according to a position and a motion of body and hand of a speaker, a shape of a finger, a facial expression. In an embodiment of the disclosure, sign language may be recognized based on joint information and a facial expression of a speaker.

[0034] Sign language differs from the language grammatical system, but has morphemes constituting one sign language sentence. In an embodiment of the disclosure, sign language may be segmented into morphemes and positions of morphemes may be predicted on a frame basis.

[0035] Furthermore, in an embodiment of the disclosure, morphemes which are acquired as a result of predicting may be variously combined and various combinations of morphemes may be used for augmenting training data.

[0036] FIG. 1 is a block diagram of an AI-based sign language analysis system according to an embodiment of the disclosure. A sign language analysis system according to an embodiment of the disclosure may include a training unit 100 and an inference unit 200.

[0037] The training unit 100 may be configured to train an AI model for analyzing sign language, and may include a recognition unit 110 and a model generation unit 120.

[0038] The recognition unit 100 may recognize joint information of body and hand of a speaker who uses sign language and facial expression information of the speaker from a sign language video for training on a frame basis.

[0039] The model generation unit 120 may train an AI model that segments sign language into morphemes based on information recognized by the recognition unit 110, and predicts position information of the segmented morphemes on a frame basis.

[0040] The inference unit 200 may analyze sign language in a sign language video to infer by using the AI model trained by the model generation unit 120, and may augment training data based on analyzed sign language data. The inference unit 200 may include a recognition unit 210 and a prediction/augmentation unit 220.

[0041] The recognition unit 210 may recognize joint information of body and hand of a speaker and facial expression information of the speaker from a sign language video to infer on a frame basis.

[0042] The prediction/augmentation unit 220 may segment sign language into morphemes by using the AI model trained by the model generation unit 120, based on information recognized by the recognition unit 210, and may predict position information of the segmented morphemes on a frame basis.

[0043] In addition, the prediction/augmentation unit 220 may augment training data by variously combining morphemes acquired as a result of predicting.

[0044] FIG. 2 is a block diagram of the recognition unit 110 provided in the training unit 100 shown in FIG. 1. As shown in FIG. 2, the recognition unit 110 may include a sign language database 111 for training, and a key point extraction module 112.

[0045] The sign language database 111 for training is a database in which sign language videos for training are stored and morpheme position information on videos are stored with annotations (labels).

[0046] The key point extraction module 112 may receive an input of a sign language video for training, which is stored in the sign language database 111 for training, and may extract joint information of body and hand of a speaker and facial expression information of the speaker as key points.

[0047] By extracting key points, all unnecessary elements that may exist in image frames except for a motion or a facial expression of the speaker may be removed. FIG. 3 illustrates a result of extracting key points. As shown in FIG. 3, information of positions of key points may be acquired from the body, hand, face of a speaker. In FIG. 3, positions expressed by circles are key points, and lines are just drawn for easy explanation of a shape and do not correspond to key points.

[0048] FIG. 4 is a block diagram of the model generation unit 120 provided in the training unit 100 shown in FIG. 1. As shown in FIG. 4, the model generation unit 120 may include a data normalization module 121 and a frame-based morpheme recognition network 122.

[0049] The data normalization module 121 may normalize key points that are extracted by the key point extraction module 112. Persons may have different body types and different positions in an image. Accordingly, data may be normalized and normalized key points may be used as input data to the frame-based morpheme recognition network 122.

[0050] The frame-based morpheme recognition network 122 is an AI model that segments sign language into morphemes by using normalized key points as an input, and predicts positions information of the segmented morphemes on a frame basis.

[0051] FIG. 5 illustrates a detailed configuration of the frame-based morpheme recognition network 122. As shown in FIG. 5, the frame-based morphemes recognition network 122 is a network that receives key points normalized on a frame basis and recognizes morphemes on a frame basis, that is, recognizes which morpheme a current frame corresponds to.

[0052] FIG. 6 illustrates a result of recognizing morpheme positions on a frame basis by the frame-based morpheme recognition network 122. As shown in FIG. 6, the frame-based morpheme recognition network 122 recognizes in which frames morphemes constituting sign language are positioned, that is, which morpheme each frame constituting a video indicates, and provides a result of recognizing.

[0053] The frame-based morpheme recognition network 122 is trained so as to minimize a loss in the result of predicting and morpheme position information on a corresponding sign language video, which is stored in the sign language database 111 for training.

[0054] FIG. 7 is a block diagram of the recognition 210 provided in the inference unit 200 shown in FIG. 1. As shown in FIG. 7, the recognition 210 may include a new sign language image acquisition module 211 and a key point extraction module 212.

[0055] The sign language acquisition module 211 may acquire a new sign language video, that is, a sign language video to infer, and may input the new sign language video to the key point extraction module 212.

[0056] The key point extraction module 212 may extract joint information of body and hand of a speaker and facial expression information of the speaker from the inputted sign language video as key points. A detailed function of the key point extraction module 212 is the same as a detailed function of the key point extraction module 112 of the training unit 100.

[0057] FIG. 8 is a block diagram of the prediction/augmentation unit 220 shown in FIG. 1. As shown in FIG. 8, the prediction/augmentation unit 220 may include a data normalization module 221, a frame-based morpheme recognition network 222, and a morpheme combination data augmentation module 223.

[0058] The data normalization module 221 may normalize key points which are extracted by the key point extraction module 212. The normalized key points may be used as input data of the frame-based morpheme recognition network 222.

[0059] The frame-based morpheme recognition network 222 is the frame-based morpheme recognition network 122 that is trained by the model generation unit 120. The frame-based morpheme recognition network 222 may segment sign language into morphemes by using key points which are normalized by the data normalization module 221 as an input, and may predict position information of the segmented morphemes on a frame basis.

[0060] The morpheme combination data augmentation module 223 may augment training data by variously combining morphemes which are acquired as a result of predicting by the frame-based morpheme recognition network 222.

[0061] FIG. 9 illustrates an example of augmenting training data by the morpheme combination data augmentation module 223. When a sentence, “Where is the bank?”, is expressed as sign language, the sign language may be comprised of sign language morphemes [“bank”, “where”, “place” ].

[0062] The sign language database 111 for training may have a sign language video that is expressed in the above order, and, if the frame-based morpheme recognition network 222 has been trained with this training data, a sign language video made in order of [“bank”, “where”, “place” ] may be properly inferred.

[0063] However, when sign language is spoken, the order of morphemes may be changed to [“bank”, “place”, “where” ] according to a person who does sign language, and sometimes, a person may express [“bank”, “place” ] by omitting a morpheme. In order to constitute a robust sign language recognition model, there is a need for a meaningful morpheme combination that does not exist in the sign language database 111 for training.

[0064] Accordingly, the morpheme combination data augmentation module 223 may augment data by combining morpheme-based sign language video frames from the result of recognizing a meaning and a position of a morpheme in each frame.

[0065] FIG. 9 illustrates an example of augmenting a sign language video made in order of [“bank”, “where”, “place” ] to a sign language video made in order of [“bank”, “place”, “where” ], a sign language video made in order of [“bank”, “place” ], and a sign language video made in order of [“bank”, “where” ].

[0066] The augmented sign language videos may be stored in the sign language database 111 for training with annotations indicating position information of the morphemens, and may be used as training data.

[0067] FIG. 10 is a view illustrating a hardware structure for implementing the AI-based sign language analysis system shown in FIG. 1.

[0068] The system according to an embodiment of the disclosure may be implemented by a computing system that is established by including a communication unit 310, an output unit 320, a processor 330, an input unit 330, and a storage unit 350.

[0069] The communication unit 310 is a communication means for communicating with an external device and accessing an external network. The output unit 320 is a display for displaying a result of executing by the processor 330, and the input unit 330 is a user input means for delivering a user command to the processor 330.

[0070] The processor 330 is configured to perform functions of the AI-based sign language analysis system shown in FIG. 1, and includes a plurality of graphics processing units (GPUs) and a central processing unit (CPU).

[0071] The storage unit 350 provides a storage space necessary for operations and functions of the processor 330.

[0072] Up to now, a method for segmenting sign language into morphemes, a method for predicting positions of morphemes, and a method for augmenting data have been described in detail with reference to preferred embodiments.

[0073] In the above-described embodiments, by recognizing morphemes of a sign language video on a frame basis based on a skeleton model and thereby segmenting sign language into morphemes and predicting positions of morphemes, it is possible to lay the foundations for accurate sign language translation.

[0074] In addition, by augmenting sign language data for training by variously combining predicted morphemes, training of a sign language recognition model can be reinforced and accuracy of prediction can be further enhanced.

[0075] The technical concept of the disclosure may be applied to a computer-readable recording medium which records a computer program for performing the functions of the apparatus and the method according to the present embodiments. In addition, the technical idea according to various embodiments of the disclosure may be implemented in the form of a computer readable code recorded on the computer-readable recording medium. The computer-readable recording medium may be any data storage device that can be read by a computer and can store data. For example, the computer-readable recording medium may be a read only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, or the like. A computer readable code or program that is stored in the computer readable recording medium may be transmitted via a network connected between computers.

[0076] In addition, while preferred embodiments of the disclosure have been illustrated and described, the disclosure is not limited to the above-described specific embodiments. Various changes can be made by a person skilled in the art without departing from the scope of the disclosure claimed in claims, and also, changed embodiments should not be understood as being separate from the technical idea or prospect of the disclosure.