Development platform for facilitating the optimization of natural-language-understanding systems

[0001] The device and method disclosed in this document relates to natural language understanding systems and, more particularly, to a development platform for facilitating the optimization of performance of natural language understanding systems.

[0002] Unless otherwise indicated herein, the materials described in this section are not admitted to be the prior art by inclusion in this section.

[0003] Intelligent conversation systems (also referred to as dialog systems) have become increasingly popular in consumer markets especially in recent years. Text or voice-based conversation systems generally include a natural language understanding component that receives a input sentence from a user and is responsible for identifying the user's intent and the relevant slots or entities in the input sentence. The natural language understanding component is a critical part of an intelligent conversation systems that partially determines whether the system can accurately understand the user's input and thus provide a satisfactory response to the user.

[0004] Given the vast need of developing conversation systems, especially those task-specific conversation systems that support business activities, various development platforms, such as RASA and Dialogflow, have been introduced that help developers develop the natural language understanding component, optionally together with the whole conversation system. The existence of such development platforms has boosted the number of conversation systems under development and in usage.

[0005] However, the currently available development platforms for conversation systems only provide very limited functions for developing the natural language understanding component. Particularly, these development platforms primarily allow the developer to input training sentences, optionally with slot/intent information, to train a natural language understanding model using either a selected algorithm or a default algorithm. Although this simplifies development of a conversation systems, it is only suitable for small-scale applications having low complexity. In real applications, especially when a business expands, developers often need to expand the conversation system to cover an increased number of use cases and must provide more and more training data to train the natural language understanding model to handle the increased number of use cases.

[0006] As a result of the periodic expansion in the scope of use cases, the conversation system will often perform increasingly worse over time. Particularly, as the conversation system is expanded and as more and more training data is provided, the complexity of the slot/intent information increases and ambiguities in the training data emerge because the ever-expanding training data becomes disorganized. Moreover, as the conversation system is expanded, the natural language understanding algorithm used to perform intent detection and slot filling often remains the same, regardless of whether the chosen algorithm is the best option in light of the expanded scope of the conversation system (note that the best algorithm varies for data with different characteristics).

[0007] What is needed is a new development platform for natural language understanding systems that provides tools to the developer for refining intent and slot definitions, improving data organization, and supporting the selection and analysis of the best algorithm from a pool of state-of-the-art natural language understanding algorithms. It would be advantageous if the platform would not only support the development of a natural language understanding component but also make optimization of the component convenient, leading to efficient development of conversation systems with improved performance.

[0008] A method for optimizing performance of a natural language understanding model is disclosed. The method comprises receiving, with a processor, a plurality of training sentences, each training sentence being labeled with a respective intent from a plurality of intents, at least one portion of at least some of the plurality of training sentences being labeled with a respective slot type from a plurality of slot types. The method further comprises determining, with the processor, a recommended modification to the plurality of training sentences, the recommended modification including at least one of (i) relabeling and (ii) deleting a training sentence in the plurality of training sentences. The method further comprises generating, with the processor, a modified plurality of training sentences by performing the recommended modification to the plurality of training sentences in response to a user input received via a user interface. The method further comprises training, with the processor, a first natural language understanding model using the modified plurality of training sentences.

[0009] A non-transitory computer-readable medium that stores a computer program for optimizing performance of a natural language understanding model is disclosed. The computer program has program instructions that, when executed by a processor, cause the processor to receive a plurality of training sentences, each training sentence being labeled with a respective intent from a plurality of intents, at least one portion of at least some of the plurality of training sentences being labeled with a respective slot type from a plurality of slot types. The computer program has further program instructions that, when executed by the processor, cause the processor to determine a recommended modification to the plurality of training sentences, the recommended modification including at least one of (i) relabeling and (ii) deleting a training sentence in the plurality of training sentences. The computer program has further program instructions that, when executed by the processor, cause the processor to generate a modified plurality of training sentences by performing the recommended modification to the plurality of training sentences in response to a user input received via a user interface. The computer program has further program instructions that, when executed by the processor, cause the processor to train a first natural language understanding model using the modified plurality of training sentences.

[0021] For the purposes of promoting an understanding of the principles of the disclosure, reference will now be made to the embodiments illustrated in the drawings and described in the following written specification. It is understood that no limitation to the scope of the disclosure is thereby intended. It is further understood that the present disclosure includes any alterations and modifications to the illustrated embodiments and includes further applications of the principles of the disclosure as would normally occur to one skilled in the art which this disclosure pertains.

System Overview

[0022] FIG. 1 shows an exemplary workflow 10 of a development platform for developing a natural language understanding model and optimizing performance thereof. In some embodiments, the workflow 10 is just one aspect of a comprehensive development platform for developing a task-specific conversation system that utilizes the natural language understanding model. The exemplary workflow 10 incorporates a main interface 20 and an optimization interface 30, both of which can be embodied as one or more graphical user interfaces. The main interface 20 is the primary interface of the workflow 10, whereas the optimization interface 30 is an interface for optimizing performance of the natural language understanding model.

[0023] Using the main interface 20, a user (e.g., a developer) first uploads or otherwise inputs a set of training data that will be used for training the natural language understanding model (block 22). The training data is in the form of a plurality of training sentences, which are each labeled with an intent and possibly with one of more slots. In one example, the main interface 20 has an “Input Training Data” button that, when selected by the user, provides a data-uploading interface that enables the user to upload training sentences (either by inputting the training sentences one by one or by uploading a file that contains all of the training sentences) together with the corresponding ground-truth intent labels and slot labels. Additionally, the user may also upload relevant slot-value lists (also referred to herein as “entity lists”) for certain slots or slot types.

[0024] After the training data is uploaded via the data-uploading interface, the user can return to the main interface 20 and proceed with development of the natural language understanding model. The user may optionally perform one or more optimizations (block 24), prior to beginning model training (block 26). In one example, the main interface 20 has an “Optimization” button that, when selected by the user, provides the optimization interface 30. Within the optimization interface 30, the user can utilize a variety of optimization tools including: an “Entity Refinement” tool with which the user can refine the slot-value lists and slot labels of the training sentences (block 32); an “Intent Refinement” tool with which the user can refine the intent labels of the training sentences (block 34); a “Training Sentence Reorganization” tool with which the user can adjust the organization of the training sentences (block 36); and an “Algorithm Comparison” tool with which the user can identify and select the most suitable training algorithm for training the natural language understanding model. As discussed in greater detail below, each of the optimization tools provides graphical user interfaces for simple step-by-step optimization of the natural language understanding model.

[0025] After performing optimizations using the optimization tools of the optimization interface 30 (or if the optimizations were skipped), the user can return to the main interface 20 and proceed with development of the natural language understanding model. In one example, the main interface 20 has an “Begin Model Training” button that, when selected by the user, begins training the natural language understanding model (block 26). If optimizations were performed, then the natural language understanding model is trained using the training data with the optimized intent/slot labels, optimized slot-value lists, and the selected training algorithm. However, if no optimization was performed, then the natural language understanding model is trained using the original intent/slot labels, the original slot-value lists, and a default training algorithm.

[0026] After training the natural language understanding model, the user can return to the main interface 20 and proceed with testing the natural language understanding model. In one example, the main interface 20 has a “Testing/Inference” button that, when selected by the user, provides a testing interface via which the user can conduct single-sentence inference or batch testing, in a conventional manner.

Exemplary Hardware Embodiment

[0027] FIG. 2 shows an exemplary embodiment of the development platform 100 or other computing device that can be used to develop and train natural language understanding model(s) 122 for performing intent detection and slot filling within an intelligent conversation system or dialog system. The development platform 100 comprises a processor 110, a memory 120, a display screen 130, a user interface 140, and at least one network communications module 150. It will be appreciated that the illustrated embodiment of the development platform 100 is only one exemplary embodiment is merely representative of any of various manners or configurations of a server, a desktop computer, a laptop computer, or any other computing devices that are operative in the manner set forth herein. In some embodiments, the development platform 100 is in communication with a database 102, which may be hosted by another device or which is stored in the memory 120 of the development platform 100 itself.

[0028] The processor 110 is configured to execute instructions to operate the development platform 100 to enable the features, functionality, characteristics and/or the like as described herein. To this end, the processor 110 is operably connected to the memory 120, the display screen 130, and the network communications module 150. The processor 110 generally comprises one or more processors which may operate in parallel or otherwise in concert with one another. It will be recognized by those of ordinary skill in the art that a “processor” includes any hardware system, hardware mechanism or hardware component that processes data, signals or other information. Accordingly, the processor 110 may include a system with a central processing unit, graphics processing units, multiple processing units, dedicated circuitry for achieving functionality, programmable logic, or other processing systems.

[0029] The memory 120 is configured to store data and program instructions that, when executed by the processor 110, enable the development platform 100 to perform various operations described herein. The memory 120 may be of any type of device capable of storing information accessible by the processor 110, such as a memory card, ROM, RAM, hard drives, discs, flash memory, or any of various other computer-readable medium serving as data storage devices, as will be recognized by those of ordinary skill in the art.

[0030] The display screen 130 may comprise any of various known types of displays, such as LCD or OLED screens, configured to display graphical user interfaces. The user interface 140 may include a variety of interfaces for operating the development platform 100, such as buttons, switches, a keyboard or other keypad, speakers, and a microphone. Alternatively, or in addition, the display screen 130 may comprise a touch screen configured to receive touch inputs from a user.

[0031] The network communications module 150 may comprise one or more transceivers, modems, processors, memories, oscillators, antennas, or other hardware conventionally included in a communications module to enable communications with various other devices. Particularly, the network communications module 150 generally includes a Wi-Fi module configured to enable communication with a Wi-Fi network and/or Wi-Fi router (not shown) configured to enable communication with various other devices. Additionally, the network communications module 150 may include a Bluetooth® module (not shown), as well as one or more cellular modems configured to communicate with wireless telephony networks.

[0032] In at least some embodiments, the memory 120 stores program instructions of the natural language understanding model(s) 122 that, once the training is performed, are configured to perform intent detection and slot filling within an intelligent conversation system or dialog system. In at least some embodiments, the database 102 stores a plurality of training sentences 160, which include natural language sentences (i.e., text) which are each labeled with a respective intent label and portions of which are labeled with respective slot labels. Additionally, in at least some embodiments, the database 102 further stores a plurality of slot-value lists 170 (also referred to as “entity lists”), each consisting of a plurality of possible values for a corresponding slot or slot type that exists in at least some of the training sentences.

Methods for Optimizing a Natural Language Understanding Model

[0033] A variety of methods and processes are described below for operating the development platform 100 or other computing device to develop and train the train natural language understanding model(s) 122 for performing intent detection and slot filling within an intelligent conversation system or dialog system. In these descriptions, statements that a method, processor, and/or system is performing some task or function refers to a controller or processor (e.g., the processor 110 of the development platform 100) executing programmed instructions stored in non-transitory computer readable storage media (e.g., the memory 120 of the development platform 100) operatively connected to the controller or processor to manipulate data or to operate one or more components in the development platform 100 or of the database 102 to perform the task or function. Additionally, the steps of the methods may be performed in any feasible chronological order, regardless of the order shown in the figures or the order in which the steps are described.

[0034] FIG. 3 shows a flow diagram for a method 200 for a developing a natural language understanding model for intent detection and slot filling. The method 200 advantageously enables the user to easily refine intent and slot definitions, improve data organization, and select the of the best natural language understanding algorithm from a pool of state-of-the-art natural language understanding algorithms. Thus, the method 200 not only supports the development of a natural language understanding model but also makes optimization of the component convenient, leading to efficient development of conversation systems with improved performance.

[0035] The method 200 begins with receiving a plurality of training sentences, each training sentence being labeled with a respective intent, at least one portion of at least some training sentences being labeled with a respective slot type (block 210). Particularly, the processor 110 receives and/or the database 102 stores a plurality of training sentences (i.e., the training sentences 160). Each training sentence comprises a natural language text string. Each training sentence is labeled with a respective intent label from a plurality of intent labels. Additionally, for some or all training sentences, at least one portion of the training sentence (e.g., a word or sequence of words within the sentence) is labeled with a respective slot type label. In an exemplary embodiment, the processor 110 operates the display 130 to display a graphic user interface (e.g., a data-uploading interface) that enables the user to upload training sentences (either by inputting the training sentences one by one or by uploading a file that contains all the training sentences) together with the corresponding ground-truth intent labels and slot type labels.

[0036] It will be appreciated by those of ordinary skill in the art that an “intent” refers to a goal, objective, task, or similar which is supported by a natural language understanding system. For example, a simple conversation system provided by a website for an airline or travel agent might support the following intents: a “Book Ticket” intent that enables a user to book an airline ticket and a “Check Flight Status” intent that enables to user to check the status of a flight. For each intent supported by the natural language understanding system, a multiplicity of training sentences are provided that are labeled with that intent.

[0037] Additionally, it will be appreciated by those of ordinary skill in the art that a “slot” refers portions of a sentence that includes critical information for fulfilling the intent of the sentence. The training sentences provided for each intent may include sentences having a variety of different sentence patterns and including a variety of exemplary critical information for fulfilling the intent. For example, the “Book Ticket” intent might include some training sentences having the sentence pattern “I want to book {NumberOfTicket} tickets to {Destination} in {TripDate}” and some training sentences having the sentence pattern “I want to get {NumberOfTicket} {TripDate} ticket to {Destination}.” As can be seen, the two exemplary sentence structures have the same three slots {NumberOfTicket}, {Destination}, and {TripDate} that include the critical information required to are fulfill the intent of booking a ticket, but with a slightly different sentence structure. It should be appreciated that the training sentences themselves include actual exemplary text values for each labeled slot (e.g., “I want to book four tickets to Paris in June.”).

[0038] Each slot (also referred to as an “entity”) labeled in a sentence has a slot type (e.g., “NumberOfTicket”, “Destination”, and “TripDate”) and a slot value (e.g., “four”, “Paris”, and “June”). Using various machine learning algorithms, a natural language understanding model can be trained purely based on labeled sentences and will be able to detect slot(s) (e.g., detect the portion “Paris” in the sentence “I want to book four tickets to Paris in June” as a slot with slot type “Destination” and slot value “Paris”) based on sentence context during inference/testing. However, in many cases, leveraging pre-defined slot-value lists (e.g., a list of possible destination cities for the slot type “Destination”) for some or all slot types in both training and testing/inference of natural language understanding model 122 with certain machine learning methods can be beneficial for performance improvement. Leveraging pre-defined slot-value list(s) is a common choice for the development of domain-specific natural language understands.

[0039] To these ends, in at least some embodiments, the processor 110 also receives and/or the database 102 also stores a plurality of slot-value lists (i.e., the slot-value lists 170). Each slot-value list defines a plurality of acceptable values for a respective slot type. For example, the {Destination} slot type may utilize a slot-value list that provides possible values for a flight destination (e.g., Paris, New York, London, etc.). As an example, the conversation system provided by an airline or travel agent might further support a “Purchase In-Flight Drink” intent that requires critical information in a {Drink_to_Order} slot type. A slot-value list may be useful to provides possible values for the {Drink_to_Order} slot type (e.g., soda, beer, water, etc.).

[0040] It should be appreciated that, in some cases, a slot-value list may comprise an exhaustive list of all acceptable values for a particular slot type. However, in other cases, a slot value list may comprise a non-exhaustive list of acceptable values for a particular slot type, which is automatically extensible by the natural language understanding model(s) 122. Finally, it should be appreciated that not all slot types necessarily require a slot-value list. Particularly, slot types whose corresponding slot values are dates, times, or numbers generally do not require a pre-defined slot-value list for the natural language understanding model(s) 122 to accurately perform slot filling.

[0041] Finally, it should be appreciated that at least some intents do not require any slots. For example, a “Confirmation” intent might include training sentences such as “Yes.” or “Sure, of course.” In intents such as this, there isn't any additional critical information needed aside from simply the intent itself.

[0042] With continued reference to FIG. 3, the method 200 continues with determining a recommended modification to the plurality of training sentences, the recommended modification including at least one of (i) relabeling and (ii) deleting a training sentence in the plurality of training sentences (block 220). Particularly, the processor 110 is configured to determine or identify at least one recommended modification the plurality of training sentences 160. In some cases, the recommended modification may include relabeling an intent of one or more training sentences. In some cases, the recommended modification may include relabeling a slot, in particular a slot type, of a portion of one or more training sentences. In some cases, the recommended modification may include deleting one or more training sentences from the plurality of training sentences 160.

[0043] In an exemplary embodiment, in response to a corresponding user selection, the processor 110 operates the display 130 to display a graphic user interface (e.g., the optimization interface 30) that enables the user to interact with one or more optimization tools to identify and implement recommended modifications to the plurality of training sentences 160. Such optimizations tools for identifying and implementing recommended modifications to the plurality of training sentences 160 include the “Entity Refinement” tool, the “Intent Refinement” tool, and the “Training Sentence Reorganization” tool, introduced above and discussed in greater detail below.

[0044] FIG. 4 shows a method 300 for refining slot type labels in a plurality of training sentences for training a natural language understanding model. The method 300 corresponds generally to the “Entity Refinement” tool introduced above and advantageously helps the user identify potential problems and improve the definitions of slots or slot types (also referred to as “entities”). For example, in real-life development of an intelligent conversation system, a typical scenario is that after an initial launch of the intelligent conversation system, the scope of the intelligent conversation system grows over time to support new intents having new slots or slot types. The expansion in scope generally occurs continuously or periodically, possibly by different developers (e.g., different people may participate in the development of the intelligent conversation system over several years). This often leads to suboptimal definition of slot types and/or their corresponding slot-value lists, which will lead to poor performance of the natural language understanding model(s) 122 for the intents utilizing those slot types.

[0045] The method 300 begins with comparing the plurality of values listed in each slot-value list (block 310). Particularly, in response to a corresponding user selection via the graphical user interface, the processor 110 compares and determines a similarity between each combination of two slot-value lists. The similarity between two slot-value lists can be quantified in a variety of ways. In one embodiment, for each combination of two slot-value lists, the processor 110 quantifies the similarity by calculating an overlap ratio between the values listed in the two slot-value lists (i.e., a ratio of values that exist in both slot-value lists). The numerator of the ratio is the number of values that exist in both slot-value lists. The denominator of the overlap ratio may correspond to a total number of unique values between the two slot-value lists or to a total number of values in one of the slot-value lists. Of course, in some embodiments, multiple different overlap ratios formed with different denominators can be utilized for comparison.

[0046] In another embodiment, for each combination of two slot-value lists, the processor 110 quantifies the similarity by calculating a similarity of training sentences that are labeled with a slot type corresponding to one of the two slot-value lists. Particularly, for each sentence labeled with one of the corresponding slot types, the processor 110 determines a sentence context vector using a natural language semantics model (e.g., a neural network). The processor 110 calculates a similarity between any two sentences as a distance between the corresponding sentence context vectors. The sentence context similarity between two slot-value lists can be calculated using a variety of different statistical methods based on the corresponding sentence context vectors (e.g., average distance, etc.). In one embodiment, the processor 110 clusters the sentence context vectors using a clustering algorithm and calculates a further overlap ratio as a ratio of sentence contexts (clusters) represented the training sentences corresponding to both of the slot-value lists.

[0047] The method 300 continues with identifying a group of slot-value lists that are similar to one another (block 320). Particularly, the processor 110 identifies at least two slot-value lists that are similar to one another based on the comparison discussed above, in particular based on the quantified similarities between the slot-value lists. In one embodiment, the processor 110 identifies a pair or group of slot-value lists as being similar to one another if the overlap ratio between the pair of slot-value lists or each pair in the group of slot-value lists exceeds a predetermined threshold. In one embodiment, the processor 110 identifies a pair or group of slot-value lists as being similar to one another if the sentence context similarity (e.g., an average distance between sentence context vectors) of the pair of slot-value lists or each pair in the group pair of slot-value lists falls below a predetermined threshold (or exceeds a predetermined threshold, if a larger value indicates greater similarity).

[0048] In further embodiments, both the overlap ratio and the sentence context similarity can be used. Particularly, in one embodiment, the processor 110 identifies a pair or group of slot-value lists as being similar to one another if the overlap ratios exceed a first predetermined threshold and the sentence context similarities falls below a second predetermined threshold (or exceeds the second predetermined threshold, if a larger value indicates greater similarity). In some embodiments, the processor 110 combines, as a weighted sum, (1) the overlap ratio of the values in each slot-value lists and (2) the overlap ratio of the sentence contexts represented the training sentences corresponding to both of the slot-value lists. The weighted sum may consist of a linear summation or the like. The processor 110 identifies a pair or group of slot-value lists as being similar to one another if the weight sum(s) exceed a predetermined threshold

[0049] The method 300 continues with merging the identified group of slot-value lists into a single slot-value list corresponding to a single slot type in response to a user input (block 330). Particularly, after one or more pairs/groups of similar slot-value lists are identified, the processor 110 operates the display 130 to display a graphic user interface that displays the identified pairs/groups of similar slot-value lists to the user. FIG. 5 shows an exemplary graphical user interface 400 for recommending mergers of similar slot-value lists. In the graphical user interface 400, the available slot-value lists are displayed in a top section 410 (described as “entity lists” in the illustration). Additionally, the identified pairs/groups of similar slot-value lists are displayed to the user in a middle section 420 (e.g., by encircling the names of the entity lists with a dashed circle). Text entry fields are provided via in a lower section 430, via which the user can provide a name for a new merged slot-value list.

[0050] In response to the user confirming any of the recommended mergers (e.g., by entering a new name in the corresponding text entry field), the processor 110 is configured to combine the identified pairs/groups of similar slot-value lists into a single new slot-value list, which is stored with the slot-value lists 170. Additionally, the processor 110 relabels each portion of each training sentence in the plurality of training sentences 160 that is labeled with a slot type corresponding to one of the identified pairs/groups of similar slot-value lists with a new slot type label corresponding to the new slot-value list (i.e., with the slot type name entered by the user in the corresponding text entry field).

[0051] After performing the mergers, the processor 110 repeats the processes of blocks 310 and 320 to recalculate the similarity between the updated slot-value lists and to provide further recommendations of pairs/groups of slot-value lists that are similar to one another and which could be merged. In the graphical user interface 400, the processor 110 updates the available slot-value lists that are displayed in the top section 410 and updates the identified pairs/groups of similar slot-value lists that are displayed to the user in the middle section 420. The process can continue until the user clicks a “Done” button 440 in the graphical user interface 400 or until there are no more pairs/groups of similar slot-value lists detected. The system will then return to the optimization interface 30, via which the user can continue the next step of optimization, with the updated slot-value lists and the modified training sentences with the updated slot type labels.

[0052] FIG. 6 shows a method 500 for refining intent labels in a plurality of training sentences for training a natural language understanding model. The method 500 corresponds generally to the “Intent Refinement” tool introduced above and advantageously helps the user identify potential problems and improve the definitions of the various intents supported by the intelligent conversation system. Particularly, similar to the slot definitions, intent definitions face the same problem that confusion often increases when the application scope extends. Moreover, by way the refinement of the slot-value lists and slot type labels in the training sentences 160, additionally ambiguities with respect to the intent definitions may be revealed. For example, when the slot-value lists and slot type labels are merged, it may be revealed that the training sentences for two different intents are actually more similar to one another than they appeared to be prior to the refinement of the slot-value lists and slot type labels. As with the slot-value lists, suboptimal and confusing definition of the various intents supported by the conversation system will often lead to poor performance of the natural language understanding model(s) 122 in recognizing the intents.

[0053] The method 500 begins with comparing the training sentences that are labeled with each the at least two intents with one another (block 510). Particularly, in response to a corresponding user selection via the graphical user interface, the processor 110 compares and determines a similarity between the training sentences corresponding to each combination of two intents (e.g., a similarity between of training sentences labeled with a first intent and training sentences labeled with a second intent). The similarity between the training sentences corresponding to two different intents can be quantified in a variety of ways.

[0054] In some embodiments, the processor 110 is configured to compare and determine a similarity between the sentence patterns of the training sentences corresponding to each combination of two intents. First, for each training sentence that labeled with one of the two intents, the processor 110 determines a sentence pattern of the training sentence by substituting the slot values (i.e., the text in the sentence that is labeled with a slot type) with the slot type label itself (i.e., a name of the slot type). For example, the training sentence “I'd like to order a beer.”, in which the text “beer” is labeled with the slot type “Drink_to_Order” would be converted into the sentences pattern “I'd like to order a {Drink_to_Order}.” The similarity between the sentence patterns of the training sentences corresponding to two different intents can be quantified in a variety of ways.

[0055] In one embodiment, for each combination of two intents, the processor 110 quantifies the similarity by calculating an overlap ratio between the sentence patterns that exist in the training sentences labeled with each of the two intents (i.e., a ratio of sentence patterns that exist in the training sentences for both intents). The numerator of the ratio is the number of sentence patterns that exist in the training sentences for both intents. The denominator of the overlap ratio may correspond to a total number of unique sentence patterns between the two intents or to a total number of sentence patterns for one of the two intents. Of course, in some embodiments, multiple different overlap ratios formed with different denominators can be utilized for comparison.

[0056] It should be appreciated that, for a sentence pattern to exist in the training sentences for both intents or, in other words for two sentences patterns to be the “same,” the sentences patterns don't necessarily need to be exactly the same, verbatim. Instead, the processor 110 can determine two sentence patterns as being the “same” if they are within a threshold similarity of one another. For example, in one embodiment, the processor 110 determines a sentence context vector for each sentence pattern using a natural language semantics model (e.g., a neural network). The processor 110 then determines that two sentence patterns are the “same” if their respective sentence context vectors are within a threshold distance from one another.

[0057] The method 500 continues with identifying a group of intents that are similar to one another (block 520). Particularly, the processor 110 identifies a pair of intents or a group of intents that are similar to one another based on the comparison discussed above, in particular based on the quantified similarities between the training sentences labeled with each intent. In one embodiment, the processor 110 identifies a pair or group of intents as being similar to one another if the overlap ratio between the sentence patterns in the training sentences of the pair of intents or each pair in the group of intents exceeds a predetermined threshold.

[0058] The method 500 continues with merging the identified group of intents into a single intent in response to a user input (block 530). Particularly, after one or more pairs/groups of similar intents are identified, the processor 110 operates the display 130 to display a graphic user interface that displays the identified pairs/groups of intents to the user. FIG. 7 shows an exemplary graphical user interface 600 for recommending mergers of similar intents. In the graphical user interface 600, the available intents are displayed in a top section 610. In one embodiment, in response to the user clicking one of the available intents clicking each listed intent, the processor 110 displays a popup in the graphical user interface 600 that displays a definition of that intent as well as a certain amount of training sentences. Additionally, the identified pairs/groups of similar intents are displayed to the user in a middle section 620 (e.g., by encircling the names of the intents with a dashed circle). Text entry fields are provided via in a lower section 630, via which the user can provide a name for a new merged intent.

[0059] In response to the user confirming any of the recommended mergers (e.g., by entering a new name in the corresponding text entry field), the processor 110 is configured to combine the identified pairs/groups of similar intents into a single new intent by relabeling each training sentence in the plurality of training sentences 160 that is labeled with an intent of the identified pairs/groups of intents with a new intent label (i.e., with the name entered by the user in the corresponding text entry field).

[0060] After performing the mergers, the processor 110 repeats the processes of blocks 510 and 520 to recalculate the similarity between the training sentences with the updated intent labels and to provide further recommendations of pairs/groups of intents that are similar to one another and which could be merged. In the graphical user interface 600, the processor 110 updates the available intents that are displayed in the top section 610 and updates the identified pairs/groups of similar intents that are displayed to the user in the middle section 620. The process can continue until the user clicks a “Done” button 640 in the graphical user interface 600 or until there are no more pairs/groups of similar intents detected. The system will then return to the optimization interface 30, via which the user can continue the next step of optimization, with the modified training sentences having the updated intent labels.

[0061] FIG. 8 shows a method 700 for reorganizing training sentences for training a natural language understanding model. The method 700 corresponds generally to the “Training Sentence Reorganization” tool introduced above and advantageously helps the user identify potential problems and improve the organization of the various training sentences 160. Particularly, for modern natural language understanding system development, the training sentences are typically collected through cloud sourcing and are input by multiple cloud sourcing workers. However, since different people may have a different understanding of the same data input request and since some requests could be confusing by nature, the same training sentence or the same sentence pattern may occur in the training sentences corresponding to multiple different intents. This can potentially cause problems with the natural language understanding model(s) 122 trained using the training sentences. This problem is especially serious for conversation systems that are designed to support a large number of different intents, or which have inappropriate intent definitions.

[0062] The method 700 begins with identifying a sentence pattern corresponding to training sentences labeled with at least two different intents (block 710). Particularly, in response to a corresponding user selection via the graphical user interface, the processor 110 identifies one or more sentence patterns corresponding to training sentences in the plurality of training sentences 160 that are labeled with at least two different intents. These sentence patterns are considered “confusing” sentence patterns because they give conflicting information to the natural language understanding model(s) 122 in training regarding the learning of intent detection.

[0063] In at least some embodiments, for each training sentence in the plurality of training sentences 160, the processor 110 determines a sentence pattern of the training sentence by substituting the slot values (i.e., the text in the sentence that is labeled with a slot type) with the slot type label itself (i.e., a name of the slot type). Next, the processor 110 compares sentence patterns to identify the one or multiple sentence patterns corresponding to training sentences that are labeled with at least two different intents. As an example, the sentence pattern “I'd like to travel to {Destination}” might appear in the training sentences for an “Activity Recommendation” intent that provides recommended activities for a travel destination, as well as a “Hotel Recommendation” intent that provides recommended hotels for a travel destination.

[0064] As similarly noted above, it should be appreciated that, for a sentence pattern to exist in the training sentences for two different intents or, in other words for two sentences patterns to be the “same,” the sentences patterns don't necessarily need to be exactly the same, verbatim. Instead, the processor 110 can determine two sentence patterns as being the “same” if they are within a threshold similarity of one another. For example, in one embodiment, the processor 110 determines a sentence context vector for each sentence pattern using a natural language semantics model (e.g., a neural network). The processor 110 then determines that two sentence patterns are the “same” if their respective sentence context vectors are within a threshold distance from one another.

[0065] The method 700 continues with deleting at least some of the training sentences corresponding to the identified sentence pattern in response to a user input (block 720). Particularly, after one or more sentence patterns are identified as corresponding to training sentences labeled with at least two different intents, the processor 110 operates the display 130 to display a graphic user interface that displays the identified confusing sentence patterns to the user. FIG. 9 shows an exemplary graphical user interface 800 for recommending reorganization of training sentences having confusing sentence patterns. In the graphical user interface 800, the confusing sentence patterns are displayed in a left-hand section 810. Additionally, a confusion level (i.e., the number of intents for which the respective sentence pattern appears in the training sentences) is listed for each confusing sentence pattern in a middle section 820. Selection boxes and/or drop down menus are provided in a right-hand section 830, which list the different intents associated with each confusing sentence pattern. The user can select one or more associated intents for which the training sentences having the confusing sentence pattern should be deleted (or, alternatively, select the intent for which the training sentences should be kept). It some cases, when a sentence pattern is considered too confusing, the user could make selections to delete all training sentences having the confusing sentence pattern.

[0066] In response to the user confirming the recommended reorganization of the training sentences for an intent (e.g., by entering a new name in the corresponding text entry field), the processor 110 is configured to delete at least some of the training sentences having the confusing sentence pattern, while keeping other training sentences having the confusing sentence pattern. In one example, the processor 110 deletes the training sentences having the confusing sentence pattern and which are labeled with a first intent, while keeping those labeled with second intent, based on the user inputs. In another example, the processor 110 deletes all training sentences having the confusing sentence pattern and which are labeled with any intent other than a selected intent, based on the user inputs. In a further example, when a sentence pattern is considered too confusing, the processor 110 deletes all training sentences having the confusing sentence pattern, regardless of the intent label, based on the user inputs.

[0067] In some embodiments, to make the graphical user interface 800 easier to use, a ranking feature can be activated by selecting a ranking button 840, which allows the user to rank and/or sort the confusing sentence patterns in the graphical user interface 800 by either the confusion level (i.e., the number of intents) or by alphabetical order. Particularly, in response to a user input (e.g., pressing the ranking button 840), the processor 110 is configured to rank the multiple confusing sentence patterns by a respective total number of different intents that include training sentences corresponding to each respective confusing sentence pattern. In response to a further user input (e.g., pressing the ranking button 840 again), the processor 110 is configured to sort the multiple confusing sentence patterns alphabetically (e.g., by text of the sentence pattern).

[0068] After the user has performed any desired reorganization of the training sentences, the user can click a “Done” button 850. The system will then return to the optimization interface 30, via which the user can continue with any additional optimizations.

[0069] Returning to FIG. 3, the method 200 continues with a step of generating a modified plurality of training sentences by performing the recommended modification to the plurality of training sentences in response to a user input received via a user interface (block 230). Particularly, as already discussed in greater detail above with respect to the methods 300, 500, and 700, the processor 110 is configured to perform the selected and/or recommended modifications to the training sentences (i.e., relabeling or deleting training sentences) in response to user inputs confirming that the modifications should be performed. In practice, the user will often make many different modifications in an iterative manner, which are performed immediately after user confirmation. Thus, it should be understood that the processes of blocks 220 and 230 of the method 200 are performed simultaneously, alternatingly, and/or iteratively, until the user is satisfied with the modifications to the plurality of training sentences 160 and/or the plurality slot-value lists 170.

[0070] With continued reference to FIG. 3, the method 200 continues with enabling the user to compare performances of a plurality of different algorithms for training the natural language understanding model (block 240). Particularly, in response to a corresponding user selection via the graphical user interface, the processor 110 operates the display 130 to display a graphic user interface enables the user to selectively train and test different algorithms for the natural language understanding model. A wide variety of natural language understanding algorithm can be supported by the development platform, such as Joint BERT (with CRF), Joint BERT (without CRF), Stack Propagation+BERT, Bi-Model, Attention Encoder-Decoder, or InterNLU. To these ends, in response to corresponding user selections via the graphical user interface, the processor 110 trains a plurality of instances of the natural language understanding model(s) 122 using a plurality of different algorithms. Additionally, the processor 110 determines a performance of each of the plurality of instances of the natural language understanding model(s) 122.

[0071] FIG. 10 shows an exemplary graphical user interface 900 for enabling a user to compare performances of a plurality of different algorithms for training a natural language understanding model. The graphical user interface 900 corresponds generally to the “Algorithm Comparison” tool introduced above. A data preparation menu is provided in the top section 910 of the graphical user interface 900. The data preparation menu enables the user to, for the purpose of algorithm evaluation, prepare a training dataset. The graphical user interface 900 provides two ways for the user prepare the training dataset for algorithm comparison. First, the user can either use the “Load Testing Data” button 912 to upload a separate testing dataset, or use the “Temporary Data Split” button 914 to temporarily split the previously provided/generated training data (i.e., the training sentences 160) into a multiple data subsets: a subset of the modified training sentences 160 to be used for training, a subset of the modified training sentences 160 to be used for validation, and a subset of the modified training sentences 160 to be used for testing. These subsets of the modified training sentences 160 are temporarily generated and used for algorithm comparison in this interface only. Particularly, it should be appreciated that, once the user exits graphical user interface 900, the previously provided/generated training data remains unchanged. The benefit of this temporary data split is to allow the user to reserve testing data for the final evaluation (i.e., making the tuning procedure blind to the testing data).

[0072] An algorithm comparison interface is provided in a lower section 920 of the graphical user interface. Particularly, once the data is prepared, the user can train an instance of the natural language understanding model(s) 122 with each supported algorithm, by clicking a corresponding “Start Training” button in a training interface section 930. Additionally, the training interface section 930 enables the user to change the algorithm parameters that are used for each supported algorithm by clicking a corresponding “Parameter Setting” button in the training interface section 930 (otherwise the default values will be used). If the plurality of training sentences 160 were temporarily split, the processor 110 trains each respective instance of the natural language understanding model(s) 122 using the corresponding algorithm and using the training subset of the plurality of training sentences 160.

[0073] Once the training is done, the time spent on the training procedure is displayed on the interface. Additionally, the user can then click a corresponding “Start Testing” button in a testing interface section 940 to conduct inference on the testing dataset and once done, calculate the statistics. Particularly, if the plurality of training sentences 160 were temporarily split, the processor 110 determines a performance of each respective instance of the natural language understanding model(s) 122 using the validation and/or testing subset of the plurality of training sentences 160 (or otherwise using a separately provided batch of testing data). The resulting overall accuracy (i.e., the ratio of sentences that both the intent detection and slot filling results are correct) is displayed in the testing interface section 940, while the detailed results such as slot F1 can be shown by clicking the “Show details” button in the testing interface section 940.

[0074] Once the algorithms are evaluated, the performance of the algorithms can be easily compared by the user using graphical user interface 900. The user can thus select one algorithm based on the achieved performances as the algorithm to use in the final training of the natural language understanding model(s) 122 for the conversation system being developed. Particularly, the processor 110 receives a user input (e.g., a selection of a check box on the right hand side of the graphical user interface 900) indicating a selected algorithm to use in the final training of the natural language understanding model(s) 122, as well as parameter settings for the selected algorithm. After the user has selected an algorithm, the system will then return to the main interface 20 via which the user can proceed with the final training of the natural language understanding model(s) 122.

[0075] Returning to FIG. 3, the method 200 continues with training a natural language understanding model using the modified plurality of training sentences (block 250). Particularly, in response to a corresponding user selection via the graphical user interface, the processor 110 trains a final instance of the natural language understanding model(s) 122 using the selected algorithm with the selected parameters and using all of the modified plurality of training sentences 160. If no parameters were selected, then the processor 110 trains the final instance of the natural language understanding model(s) 122 using the selected algorithm with default parameters. If no algorithm was selected, then the processor 110 trains the final instance of the natural language understanding model(s) 122 using a default algorithm and default parameters. If no modifications were made to the training sentences 160, then the processor 110 trains the final instance of the natural language understanding model(s) 122 using the originally provided training sentences 160.

[0076] After training the natural language understanding model, the user can return to the main interface 20 and proceed with testing the natural language understanding model. As discussed above, the main interface 20 has a “Testing/Inference” button that, when selected by the user, provides a testing interface via which the user can conduct single-sentence inference or batch testing, in a conventional manner.

[0077] Embodiments within the scope of the disclosure may also include non-transitory computer-readable storage media or machine-readable medium for carrying or having computer-executable instructions (also referred to as program instructions) or data structures stored thereon. Such non-transitory computer-readable storage media or machine-readable medium may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such non-transitory computer-readable storage media or machine-readable medium can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures. Combinations of the above should also be included within the scope of the non-transitory computer-readable storage media or machine-readable medium.

[0078] Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, objects, components, and data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.

[0079] While the disclosure has been illustrated and described in detail in the drawings and foregoing description, the same should be considered as illustrative and not restrictive in character. It is understood that only the preferred embodiments have been presented and that all changes, modifications and further applications that come within the spirit of the disclosure are desired to be protected.