Method and system for context-specific generative model answers from context-specific queries

[0001] This application is a non-provisional application of, and thereby claims benefit to, U.S. Patent Application Ser. No. 63/616,477 filed on Dec. 29, 2023. U.S. Patent Application Ser. No. 63/616,477 is incorporated herein by reference in its entirety.

[0002] Chatbots, powered by generative machine learning models such as large language models, are commonly used to answer online questions posed by users. The user submits a query via a graphical user interface. The query is sent to the chatbot, which provides a response to the user's question.

[0003] Chatbots, and generative models generally, have limitations. For example, while chatbots might be useful for answering general questions (e.g., what are the IRS rules for filing as married-filing jointly versus married-filing separately), chatbots are unable to answer satisfactorily specific questions (e.g., which filing status will save me more money, married-filing jointly versus married-filing separately).

[0004] One or more embodiments provide for a method. The method includes receiving a context-specific query from a user device of a user. The context-specific query is specific to the user. The method also includes creating a computer-readable data structure for storing data. The computer-readable data structure is specific to the context-specific query. The method also includes determining a feature related to the context-specific query. The method also includes retrieving a user value for the feature. The user value is retrieved from a data repository storing user-specific data that includes the user value. The method also includes modifying the computer-readable data structure to generate a modified computer-readable data structure by adding the feature and the user value for the feature to the computer-readable data structure. The method also includes applying a generative model to the modified computer-readable data structure to generate an output. The output includes a context-specific answer to the context-specific query. The method also includes returning the output.

[0005] One or more embodiments also provide for a system. The system includes a computer processor and a data repository in communication with the computer processor. The data repository stores a context-specific query received from a user device of a user. The context-specific query is specific to the user. The data repository also stores a computer-readable data structure for storing data. The computer-readable data structure is specific to the context-specific query. The data repository also stores a feature related to the context-specific query. The data repository also stores a user value for the feature. The data repository also stores user-specific data that includes the user value. The data repository also stores a modified computer-readable data structure modified with the feature and the user value for the feature. The data repository also stores an output. The output includes a context-specific answer to the context-specific query. The system also includes a server controller in communication with the computer processor. The server controller is also programmed to receive the context-specific query. The server controller is also programmed to create the computer-readable data structure. The server controller is also programmed to determine the feature related to the context-specific query. The server controller is also programmed to retrieve the user value for the feature. The user value is retrieved from the user-specific data. The server controller is also programmed to modify the computer-readable data structure to generate the modified computer-readable data structure. The server controller is also programmed to return the output. The system also includes a generative model in communication with the computer processor and programmed to generate the output.

[0006] One or more embodiments also provide for a non-transitory computer readable storage medium storing program code which, when executed by a computer processor, performs a computer-implemented method. The computer-implemented method also includes receiving a context-specific query from a user device of a user. The context-specific query is specific to the user. The computer-implemented method also includes creating a computer-readable data structure for storing data. The computer-readable data structure is specific to the context-specific query. The computer-implemented method also includes determining a feature related to the context-specific query. The computer-implemented method also includes retrieving a user value for the feature. The user value is retrieved from a data repository storing user-specific data that includes the user value. The computer-implemented method also includes modifying the computer-readable data structure to generate a modified computer-readable data structure by adding the feature and the user value for the feature to the computer-readable data structure. The computer-implemented method also includes applying a generative model to the modified computer-readable data structure to generate an output. The output includes a context-specific answer to the context-specific query. The computer-implemented method also includes returning the output.

[0007] Other aspects of one or more embodiments will be apparent from the following description and the appended claims.

[0015] Like elements in the various figures are denoted by like reference numerals for consistency.

[0016] One or more embodiments are directed to systems and methods for generating context-specific generative model answers from context-specific queries, in accordance with one or more embodiments. In brief summary, the system receives a context-specific query. The system may retrieve general information, related to the query, from a content knowledge database. The system then may determine the relevant features to use in obtaining context-specific information. The relevant features also may be used by a generative model in formulating a context-specific answer. Generation of the relevant features may include determining both the number of features to be used and the type of features to be used because, as explained further below, using too many or too few features may result in an unsatisfactory answer output by the generative model.

[0017] The system also generates computer-readable data structure, which is to be used for the specific query received by the system. The computer-readable data structure stores the features as keys and stores data related to the features as values. The computer-readable data structure stores the features and values as key-value pairs.

[0018] Once the relevant features are determined, and the computer-readable data structure generated, the system retrieves context-specific information (i.e. information specific to the query) from one or more context-specific databases, possibly together with the general information. The gathered information, whether context-specific or general, is related to the features in the computer-readable data structure. Thus, the computer-readable data structure stores values for the determined features, such that information desirable for returning a context-specific answer to the query is now feasible.

[0019] A generative model is then applied to the computer-readable data structure. The generative model may be a large language model (e.g. CHATGPT®), but also may be other types of generative models, as explained further below. Because the input to the generative model contains context-specific information, the output of the generative model may be a context-specific answer to the query. See FIG. 4 for a specific example of the above-described procedure.

[0020] In this manner, one or more embodiments provide a technical approach to addressing a technical challenge. Specifically, current generative models do not perform well (e.g. do not generate answers deemed by a user to be satisfactory) when applied to context-specific queries. One or more embodiments described herein address this technical challenge using the techniques described above, as elaborated further below. Specifically, as mentioned above, one or more embodiments may permit a machine learning model to generate a satisfactory (e.g. from the perspective of a user) context-specific output from a context-specific input.

[0021] The example above is a summary. Additional embodiments, extensions, and variations of the above-described procedure are presented below. Thus, the summary described above does not necessarily limit other embodiments.

[0022] Attention is now turned to the figures. FIG. 1 shows a computing system for context-specific generative model answers from context-specific queries, in accordance with one or more embodiments. The system shown in FIG. 1 includes a data repository (100). The data repository (100) is a type of storage unit or device (e.g., a file system, database, data structure, or any other storage mechanism) for storing data. The data repository (100) may include multiple different, potentially heterogeneous, storage units and/or devices.

[0023] The data repository (100) stores a context-specific query (102). The context-specific query (102) is a query submitted to the system shown in FIG. 1. The context-specific query (102) is submitted by a user. The term “user” refers to a person, an organization, a user device, or an automated system that submits a query to the system shown in FIG. 1.

[0024] The term “context-specific” means that the answer to the context-specific query is not present in a content knowledge base (110) defined below. The term “context-specific” also means that the answer to the context-specific query applies to fewer users than the answer to a broader query (a broader query is a query that is applicable to more users than a context-specific query).

[0025] As an example, a term “context-specific” is a query that that is particular, or specific, to the user. In another example, a “context-specific query” cannot be answered satisfactorily, as determined by the user, by general information or rules contained within a content knowledge base (110) (defined below). In still another example, a “context-specific” query may apply general rules or information to user-specific data in order to generate an output (i.e., an answer) that is specific to the user. A “context-specific” query may cause a system (e.g., the system of FIG. 1) to draw upon two sources of information, one general and one user specific, to generate an automated answer (i.e. output) that is satisfactory from the point of view of the user.

[0026] However, because the system does not necessarily know from which sources of information to draw, or necessarily know how many features to use when formulating an input to a generative model (122) (defined below), querying and retrieving data from both sources of information to answer the query is often not practical. For example, the context-specific information may be applied to general principles or rules from the general information in order to generate an answer perceived by the user to be satisfactory. The procedures for addressing such a technical challenge are described with respect to FIG. 2.

[0027] The data repository (100) also stores an output (104). The output (104) is the result generated by the generative model (122) (defined below). The output (104) may be described as a context-specific answer to the context-specific query. The term “context-specific” is defined above for the context-specific query (102). The term “context-specific,” in the context of the output (104), means that the output (104) applies to fewer users than an output to a broader query (defined above). Likewise, the context-specific answer is not present in the content knowledge base (110).

[0028] The data repository (100) also stores a computer-readable data structure (106). The computer-readable data structure (106) is a computer-readable structure which stores computer-readable data. The computer-readable data structure (106) may be, for example, a relational database, a graph database, a matrix or other table, etc. In an embodiment, the computer-readable data structure (106) is a structured language text file, such as a JAVASCRIPT® object notation (JSON) file.

[0029] The computer-readable data structure (106) may be structured to contain key-value pairs. A key is a type of data. The value that corresponds to the key is the entry value for the key.

[0030] In one or more embodiments, the key may be a feature (108). The feature (108) is an instance of a category of information (i.e., a specific instance of a key). For example, a feature may be “annual gross income,” for a key that could represent many different types of income information. The value, in this case, is the corresponding value of the feature. Thus, for example, the value for the feature “annual gross income” (i.e., instance of the key) may be “$50,000.”

[0031] The computer-readable data structure (106) also may be referred to as a “data model.” However, the term “computer-readable data structure” is used instead of the term “data model” in order to avoid confusion with other uses of the word “model,” such as machine learning models including the generative model (122) (defined below).

[0032] In an embodiment, the computer-readable data structure (106) may be generated for each instance of the context-specific query (102). Thus, the computer-readable data structure (106) may be generated once per submitted instance of the context-specific query (102).

[0033] The data repository (100) also stores a content knowledge base (110). The content knowledge base (110) is a computer-readable data source that stores information about a subject to which the context-specific query (102) may pertain. For example, the context-specific query (102) may pertain to tax questions, then the content knowledge base (110) may be a computer-readable data source that stores information about taxes and the preparation of taxes (e.g., laws, rules, best practices, procedures, etc.) The content knowledge base (110) may contain general information or rules but may not contain user-specific information. In other words, the content knowledge base (110) may not store data specific to the user that submits the context-specific query (102).

[0034] The data repository (100) also stores a context-specific database (112). The context-specific database (112) is also a computer-readable data source that stores, but the context-specific database (112) stores data or information that is specific to the user that submitted the context-specific query (102). Thus, continuing the above example, the context-specific database (112) may store one or more of the following instances of information specific to the user: user name or identifier, a user profile, gross income, deductibles, family information, demographic information, etc.

[0035] The data repository (100) also may store a user value (114). The user value (114) is a value for the feature (108) that is specific to the user who submitted the context-specific query (102). Thus, the user value (114) is also specific to the context-specific query (102). The user value (114) may be a value retrieved from the context-specific database (112).

[0036] The system shown in FIG. 1 also may include other components. For example, the system shown in FIG. 1 also may include a server (116). The server (116) is one or more computing systems, possibly operating in a distributed computing environment. The distributed computing systems and corresponding components of FIG. 1 may be operated by different vendors or business entities. For example, the generative model (122) may be a third party generative model accessed by the server (116). An example of the server (116) may be the computing system (500) shown in FIG. 5A.

[0037] The server (116) includes a processor (118). The processor (118) is one or more hardware or virtual processors which may execute one or more controllers, software applications, algorithms, or models as described herein. The processor (118) may be the computer processor(s) (502) in FIG. 5A.

[0038] The server (116) may host a server controller (120). The server controller (120) is software or application specific hardware that, when executed by the processor, performs one or more operations described with respect to the method of FIG. 2 or the example shown in FIG. 4. The server (116) may execute the operations called-for by the data flows described with respect to FIG. 3A, FIG. 3B, and FIG. 3C. The server controller (120) may coordinate operations of the generative model (122) and the contextualizer (124), both defined below.

[0039] The server (116) also may host a generative model (122). The generative model (122) is a type of machine learning model that processes text. The generative model takes text as input and transforms the input into an output. For example, the generative model may summarize (the output) a large corpus of text (the input). The generative model also may encode text into a computer data structure (e.g., a vector) and also may encode the semantic meaning of that text. The functions of the generative model (122) are described with respect to FIG. 2.

[0040] More generally, the generative model (122) is a statistical model of a joint probability distribution given an observable variable and a target variable. Examples of generative models include probabilistic context-free grammar models, Boltzmann machines, a generative adversarial network, a hidden Markov model, and a large language models, such as CHATGPT®. In several embodiments, the generative model (122) is a large language model.

[0041] The server (116) also may host a contextualizer (124). The contextualizer (124) is software or application-specific hardware that takes, as input, the context-specific query (102) and generates, as output, one or more features predicted to be related to the context-specific query (102). In other words, the contextualizer (124) is used to determine the types of features to be used in the computer-readable data structure (106) and the number of features to be used in the computer-readable data structure (106).

[0042] The contextualizer (124) may be the generative model (122) but may be a different generative model, including a large language model. The contextualizer (124) may be a type of classification machine learning model, such as a multi-label classification model.

[0043] The system shown in FIG. 1 also may include one or more of the user devices (126). The user devices (126) may include one or more user input devices, such as the user input device (128). The user input devices are keyboards, mice, microphones, cameras, etc. with which a user may provide input to the user devices (126).

[0044] The user devices (126) may include one or more display devices, such as the display device (130). The display devices are monitors, televisions, touchscreens, etc. which may display information to a user.

[0045] While FIG. 1 shows a configuration of components, other configurations may be used without departing from the scope of one or more embodiments. For example, various components may be combined to create a single component. As another example, the functionality performed by a single component may be performed by two or more components.

[0046] FIG. 2 shows a flowchart of a method for context-specific generative model answers from context-specific queries, in accordance with one or more embodiments. The method of FIG. 2 may be implemented using the system of FIG. 1 or the data flows shown in FIG. 3A, FIG. 3B, and FIG. 3C. The method of FIG. 2 also may be executed using the computing system and network environment shown in FIG. 5A and FIG. 5B.

[0047] Step 200 includes receiving a context-specific query from a user device of a user. The context-specific query is specific to the user. The context-specific query may be received from a remote user device via a network. For example, the context-specific query may be received in the form of a user chatbot into which a user provides input. The context-specific query may be received in a variety of different manners, including email, private message, text, etc.

[0048] An answer to the context-specific query is absent from the content knowledge base (i.e., using a direct query to perform a direct retrieval of the requested information is not possible or practical in this instance). For example, the context-specific query may be a request to determine how much tax is owed by the particular user in a particular situation, which requires the application of rules to context-specific data related to the user. In this example, using a direct query to determine how much tax is owed by the particular user is not possible, and thus the answer to the example query is absent from the content knowledge base.

[0049] Step 202 includes creating a computer-readable data structure for storing data. The computer-readable data structure is specific to the context-specific query. The computer-readable data structure may be built by defining a number of features and establishing logical data storage for values for the features. In an embodiment, the computer-readable data structure may be built by establishing a structured data file including key-value pairs. The features form the keys and values are the specific data values for the features. In an embodiment, the computer-readable data structure may be a JSON file.

[0050] Step 204 includes determining a feature related to the context-specific query. The number of features used can affect the operation of the computer-readable data structure. For example, if too few features and corresponding values are available in the computer-readable data structure, then the generative model may output an unsatisfactory output due to the lack of data available to the generative model. However, if too many features and corresponding values are available in the computer-readable data structure, then the generative model may generate an output that does not make sense or is unsatisfactory to the user (an event known as “hallucinating” in the art of machine learning, though the generative model is not sentient and thus cannot hallucinate in the normal sense of the term).

[0051] The terms “too few” and “too many” are defined by the context of the output of the generative model. The output of the generative model is deemed unsatisfactory to a user when the generative model uses a lower threshold number of features as input. In this case, there are “too few” features. The output of the generative model also is deemed unsatisfactory to a user when the generative model uses an upper threshold number of features as input. In this case, there are “too many” features.

[0052] Determining the feature may include selecting a pre-determined number of features based on a type of the context-specific query. For example, if the context-specific query is determined to be a tax question related to a particular section of the tax code, then a selected number of pre-identified features may be included in the computer-readable data structure. However, while the feature or features may be pre-determined, or even manually determined by a computer technician, there is utility is automatically determining which features to include in the computer-readable data structure and also to automatically determine how many features to include in the computer-readable data structure.

[0053] Automatic determinization of the number of features to use or the type of features to use may proceed according to a number of different techniques. In an embodiment, a contextualizer may be applied to the context-specific query to return one or more features predicted to be related to the context-specific query. The returned features are included in the computer-readable data structure, and the values for those features may then be retrieved from the context-specific database, the content knowledge base, or a combination thereof.

[0054] The contextualizer also may determine a number of features to be used. In other words, the contextualizer (e.g., a multilabel classification model) may predict not only the type of features to be used, but how many features should be used. In this manner, the features most relevant to the query are determined automatically. Additionally, in this manner, the number of features used may be predicted to be between the lower threshold and the upper threshold of the number of features to be used. Accordingly, hallucination of the generative model may be avoided.

[0055] However, other techniques may be used to determine the number of features to be used. For example, based on the context-specific query, a large language model may be applied to the content knowledge base to return the number of features. The context-specific query and the content knowledge base may serve as input to the large language model, and a prompt to the large language model may command the large language model to determine the number of features to be used.

[0056] Step 206 includes retrieving a user value for the feature. The user value is a user-specific value for the feature. The user value is retrieved from a data repository storing user-specific data that includes the user value. However, the user value may be retrieved from other databases or may be supplied by the user at the request of the automated system performing the method of FIG. 2.

[0057] Step 208 includes modifying the computer-readable data structure to generate a modified computer-readable data structure by adding the feature and the user value for the feature to the computer-readable data structure. A processor may perform the modification of the computer-readable data structure, which is then stored in a data repository or some other form of memory. Modifying the computer-readable data structure also may include adding multiple features and multiple values to the computer-readable data structure. Note that if a feature is determined as belonging in the computer-readable data structure for the specific context-specific query in question, but a corresponding value for the feature is not available or does not exist, then the value of the feature may be set to null (e.g., zero, or empty).

[0058] For example, the computer-readable data structure may be a structured language file having key-value pairs. In this case, modifying the computer-readable data structure includes adding the feature as a key of one of the key-value pairs and adding the user value as a value of one of the key-value pairs.

[0059] Step 210 includes applying a generative model to the modified computer-readable data structure to generate an output having a context-specific answer to the context-specific query. Applying the generative model includes providing the modified computer-readable data structure as input to the generative model when a processor executes the generative model. The output is the result of the execution of the generative model.

[0060] In an example, the server controller transmits an application programming interface (API) call to the generative model using the API of the generative model. The API call includes the modified computer-readable data structure. The generative model responds to the API call with output in accordance with a predefined communication protocol.

[0061] The execution of the generative model may include multiple different types of input. For example, information from the content knowledge base (e.g. general information, rules, etc.) may be added as input to the generative model. If the generative model takes the form of a large language model, then the generative model may also include a prompt as input. The prompt instructs the large language model with parameters that adjust the execution of the large language model. An example of a prompt is shown in FIG. 3D.

[0062] Step 212 includes returning the output. The output may be returned by a number of different techniques. For example, the output may be returned by storing the output in a data repository. The output may be returned by presenting the output to a user. The output may be returned by providing the output to some other automated process to further process the output to generate a secondary result. For example, the output estimating a tax liability for a specific user may be provided to a tax program for further analysis and the generation of a secondary result that is of greater interest to the user. The secondary result may then be presented to the user.

[0063] The method of FIG. 2 may be varied by adding more steps, by varying the order of steps, or by removing certain steps. For example, in an embodiment, the method of FIG. 2 also may include applying, prior to applying the generative model to the modified computer-readable data structure, a search algorithm to a content knowledge base to retrieve information related to the context-specific query. The search algorithm searches the content knowledge base based on the context-specific query. The information in the context knowledge base then may be used as one or more of the features or values in the corresponding computer-readable data structure. Alternatively, the information in the context knowledge base may be supplied in addition to the computer-readable data structure.

[0064] In either case, applying the generative model at step 210 also may include applying the generative model to a combination of the information and the computer-readable data structure to generate the output. The combination may be adding the information from the context knowledge base in the form of features and values in the computer-readable data structure, or may be adding the information from the context knowledge base in addition to the computer-readable data structure. The selection of which form of combination to use may be made based on the type of the generative model. For example, a large language model may take as input the separated information from the computer-readable data structure and the content knowledge base.

[0065] The method of FIG. 2 also may include a user feedback process. For example, a feedback prompt may be displayed to the user. For example, the user may be prompted to either confirm or deny that the returned output satisfied the user. However, some other user feedback could be used, such as a scale system or a prompt for the user to provide feedback in the form of text.

[0066] Then, a feedback value is received from the user device. For example, the user may transmit a confirmation or a denial that the returned output satisfied the user. Alternatively, the user feedback could be received as a number on a scale system or the text feedback generated by the user.

[0067] Then, based on the feedback value, the number of features to return as the feature at step 204 may be modified. For example, if the user is not satisfied, then more or fewer features may be added at step 204 when a new context-specific query, of a similar type to the context-specific query that the user submitted, is received. Similarly, the identity of features may be varied based on the user feedback.

[0068] In an embodiment, the user value may be protected information. Protected information is information which is deemed sensitive to the user (e.g., social security number, birth date, income information, etc.). In some cases, sensitive information may not be directly available when performing the method of FIG. 2 in order to protect the security of the sensitive information. Nevertheless, sensitive information may still be used in some instances.

[0069] For example, assume the user value is protected information (e.g., a social security number). In this case, the method also may include abstracting the user value to generate an abstracted user value used as the abstracted user value. Abstracting the information may include determining a bin having a range of values within which the user value falls, and returning the bin as the abstracted user value. However, other abstraction techniques may be used. For example, the protected information may be accessible by the method of FIG. 2, but the protected information may be blocked from presentation to anyone other than the user.

[0070] In still another variation, determining whether to use the method of FIG. 2 when a context-specific query is received may be useful. For example, the method may include applying, before creating the computer-readable data structure, a classification model to the context-specific query. The output of the classification model generates a numerical prediction of whether a general answer will answer the context-specific query. If the numerical prediction fails to satisfy a threshold (i.e., a general answer will answer the context-specific query), then the general answer may be returned to the user without engaging the method of FIG. 2. Thus, creating the computer-readable data structure is performed responsive to the numerical prediction satisfying a threshold value.

[0071] While the various steps in the flowchart of FIG. 2 are presented and described sequentially, at least some of the steps may be executed in different orders, may be combined or omitted, and at least some of the steps may be executed in parallel. Furthermore, the steps may be performed actively or passively.

[0072] FIG. 3A and FIG. 3B show examples of data flows for context-specific generative model answers from context-specific queries, in accordance with one or more embodiments. Attention is first turned to the data flow of FIG. 3A. The data flow of FIG. 3A is similar to the method of FIG. 2.

[0073] A user-specific query (300) is received. Then a knowledge retrieval process (302) is executed. The knowledge retrieval process (302) retrieves general knowledge from a content knowledge base (304). However, the general knowledge is relevant to the user-specific query (300). For example, if the user-specific query (300) relates to the differences between filing taxes as married-filing jointly versus married-filing separately, then the rules for the two forms of tax filings may be retrieved from the content knowledge base (304). Thus, the knowledge retrieved is general (i.e., not specific to the user), but the knowledge retrieved is still relevant to the user-specific query (300).

[0074] Then, a context identification process (306) is executed. The context identification process (306) identifies the features to be used in a computer-readable data structure. The context identification process (306) also may generate a prompt (in the case that a generative model is a large language model), or some other query to be submitted to the generative model.

[0075] Then, a feature value retrieval process (308) is executed. The feature value retrieval process (308) may retrieve information from a context-specific database (310). Thus, the feature value retrieval process (308) retrieves context-specific data that is specific to the user. In other words, the user's personal information is retrieved from the context-specific database (310), and the personal information may be applied to the tax rules retrieved from the content knowledge base (304)).

[0076] Then, a computer-readable data structure (312) is generated during the feature value retrieval process (308). The computer-readable data structure (312) includes features retrieved during one or both of the knowledge retrieval process (302) or the feature value retrieval process (308). The computer-readable data structure (312) also includes values for the features (e.g., certain rules or facts retrieved from the content knowledge base (304) or user-specific values retrieved from the context-specific database (310)).

[0077] The computer-readable data structure (312) is then provided to a generative model (314). A processor executes the generative model (314), which outputs a context-specific answer (316). See the description for FIG. 2 for further details regarding the operation of the generative model (314).

[0078] FIG. 3B shows an alternative data flow for generating a context-specific answer to a user-specific query (320). In the data flow of FIG. 3, receipt of the user-specific query (320) triggers a feature value retrieval process (322), which retrieves user-specific values from a content-specific database (324). Concurrently, receipt of the user-specific query (320) triggers a knowledge retrieval process (326), which retrieves general information from a content knowledge database (328). The general information is still relevant to the user-specific query (320), as described above with respect to FIG. 3A.

[0079] Thus, potentially relevant information, both context-specific and general but relevant to the user-specific query (320), is gathered. Then, a context identification process (330) is executed. The context identification process (330) identifies the features to be used in a computer-readable data structure. The context identification process (330) also may generate a prompt (in the case that a generative model is a large language model), or some other query to be submitted to the generative model. The context identification process (330) also generates a computer-readable data structure (332), including features determined for use in the computer-readable data structure (332) and values for the features. The context identification process (330) also may generate a prompt (334).

[0080] The computer-readable data structure (332) and the prompt (334) are provided as input to a large language model (336). The large language model (336) is executed, and outputs a context-specific answer (338).

[0081] FIG. 3C shows an example of a data flow for automatically selecting one or more features for use with a computer-readable data structure, in accordance with one or more embodiments. Thus, FIG. 3C shows a data flow for performing at least part of step 204 of FIG. 2.

[0082] An input generation process (340) generates a set of inputs to be provided to a large language model (354). The inputs generated include a context-specific query (342) received from the user, query-related knowledge (344), a feature superset (346), and a prompt (348).

[0083] The context-specific query (342) may be the same query that is received from the user. However, if desirable, the query received from the user may be pre-processed prior to adding the context-specific query (342) to the input to the large language model (354). For example, certain terms may be replaced, spelling or grammar errors identified and corrected, etc.

[0084] The query-related knowledge (344) is retrieved from a content knowledge base (350). As with FIG. 3A and FIG. 3B, the query-related knowledge (344) is relevant to the context-specific query (342), but nevertheless is still considered general knowledge which does not directly answer the context-specific query (342).

[0085] The feature superset (346) may be generated from a user schema (352). The feature superset (346) may be a set of features which may be available for use as relevant features. In other words, the relevant features (356) are output by the data flow of FIG. 3C is taken from the feature superset (346). The user schema (352) may define various aspects of the users (e.g., an individual, a corporation, demographic information, etc.) that may be used to generate the feature superset (346). For example, a greater set of all possible features may be available, but the user schema (352) may be used to narrow the number of appropriate features to the feature superset (346).

[0086] The prompt (348) is a set of instructions which place parameters, limits, and conditions on how the large language model (354) is to process the input generated by the input generation process (340). An example of the prompt (348) is shown in FIG. 3E.

[0087] The combination of the context-specific query (342), the query-related knowledge (344), the feature superset (346), and the prompt (348) are provided as input to the large language model (354). The output of the large language model (354) is the relevant features (356). In other words, the large language model (354) predicts which of the features in the feature superset (346) are to be the relevant features (356).

[0088] Later, returning to FIG. 2 for example, the relevant features (356) are used as the features to be included in the computer-readable data structure. Similarly, the relevant features (356) may be used as the features to be used in the computer-readable data structure (312) of FIG. 3A or the computer-readable data structure (332) of FIG. 3B.

[0089] FIG. 3D and FIG. 3E show examples of prompts for use in one or more of the data flows described with respect to FIG. 2, FIG. 3A, FIG. 3B, and FIG. 3C, in accordance with one or more embodiments. FIG. 3D shows a prompt (360) for use in predicting a context-specific answer. For example, the prompt (360) may be the prompt (334) of FIG. 3B, or may be used as a prompt when the generative model (314) of FIG. 3A is a large language model. Likewise, the prompt (360) may be a prompt used when the generative model of FIG. 2, step 210, is a large language model. The prompt (360) provides instructions to the large language model that limit how the model should process the other forms of input (i.e., the computer-readable data structure and possibly the context-specific query).

[0090] FIG. 3E shows a prompt (370) for use in predicting which features, and how many features, to be included in a computer-readable data structure, such as the computer-readable data structure (106) of FIG. 1, the creating of the computer-readable data structure at step 202 of FIG. 2, the computer-readable data structure (312) of FIG. 3A, and the computer-readable data structure (332) of FIG. 3B. Thus, the prompt (370) may be the prompt (348) of FIG. 3C. Like the prompt (360), the prompt (370) provides instructions to the large language model that limit how the model should process the other forms of input (e.g., the context-specific query (342), the query-related knowledge (344), and the feature superset (346) of FIG. 3C).

[0091] FIG. 4 shows an example of a customer service chatbot for generating context-specific generative model answers from context-specific queries, in accordance with one or more embodiments. Thus, FIG. 4 is a use-case example. The example of FIG. 4 does not necessarily limit other examples and embodiments described herein.

[0092] In the example of FIG. 4, a user named Alicia is using a user input device (450) while preparing her taxes. Alicia has a subscription to a tax preparation software service hosted by a server (452). The server (452) also hosts a chatbot which provides help to Alicia as she uses the tax preparation software. When Alicia wants help, she selects a “help” widget on a display device of the user input device (450). In response, a chatbot window opens on the display device. Alicia types her query (454) into the chatbot window. The query (454) is, “What is the difference in my tax bill if I file as married-filing jointly or if I file as married-filing separately?” The query (454) is submitted to the server (452).

[0093] The server (452) processes the query (454) and determines, using a large language model, that the query (454) is a context-specific query. In other words, the large language model predicts that Alicia does not want a general answer from general information, but wants a specific answer to her particular tax situation. Thus, the server (452) performs a method similar to that shown in FIG. 2 or the data flow of FIG. 3A.

[0094] At step 456 the server (452) retrieves information from a knowledge base. In this case, the information is internal revenue service (IRS) rules. The information is relevant to the query (454). For example, the retrieved information may be the rules and laws that apply to the filing status of a tax payer (i.e., married-filing jointly and married-filing separately).

[0095] At step 458 the server (452) creates a computer-readable data structure, which includes key-value pairs. This process may be performed concurrently with retrieving information from a knowledge base at step 456. The computer-readable data structure is initially not populated with keys and values, but is ready to receive the features that will be used as the keys and the corresponding values of the features.

[0096] At step 460 the server (452) determines the feature or features to be used in the computer-readable data structure. In this example, the features are “taxable income,” “marital status,” “filing status,” and a four other features from a user profile associated with Alicia.

[0097] At step 462 the server (452) retrieves values for the features determined at step 460. The server (452) may retrieve, from a data repository (464), financial data provided by or stored for Alicia. Additionally, values for features from the general knowledge base (retrieved at step 456) also may be added to the computer-readable data structure.

[0098] At step 466, the server (452) modifies the computer-readable data structure to include the features selected at step 460 and the values retrieved at step 462 (and possibly information from the knowledge base at step 456. In this example, context-specific information is included in the modified computer-readable data structure. Specifically, Alicia's particular annual income ($50,000 per year), Alicia's martial status (married), and her other profile information values associated with the determined features are all added to the modified computer-readable data structure at step 466.

[0099] The computer-readable data structure from step 466, a prompt (470) pre-generated by a computer technician, and possibly information from the knowledge base retrieved at step 456 are provided in combination as input to a large language model (468). The server (452) executes the large language model (468). In response, the large language model (468) generates an output (472), which is a context-specific answer. In particular, the context-specific answer is, “You will pay $250 more in taxes if your filing status is married-filing separately, compared to if your filing status is married-filing jointly.”

[0100] Thus, one or more embodiments provide Alicia with a specific answer to her context-specific query, not just general rules or principles that Alicia herself would have to apply to arrive at the answer she desired. Because the answer is specific to Alicia's situation, Alicia is satisfied with the output of the chatbot. Alicia decides to select “married-filing jointly” as her filing status and continues preparing her taxes using the tax preparation software service hosted on the server (452).

[0101] One or more embodiments may be implemented on a computing system specifically designed to achieve an improved technological result. When implemented in a computing system, the features and elements of the disclosure provide a significant technological advancement over computing systems that do not implement the features and elements of the disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware may be improved by including the features and elements described in the disclosure.

[0102] For example, FIG. 5A and FIG. 5B shows an example of a computing system useful for generating context-specific generative model answers from context-specific queries, in accordance with one or more embodiments. In FIG. 5A, the computing system (500) may include one or more computer processor(s) (502), non-persistent storage device(s) (504), persistent storage device(s) (506), a communication interface (508) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functionalities that implement the features and elements of the disclosure. The computer processor(s) (502) may be an integrated circuit for processing instructions. The computer processor(s) (502) may be one or more cores or micro-cores of a processor. The computer processor(s) (502) includes one or more processors. The computer processor(s) (502) may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), combinations thereof, etc.

[0103] The input device(s) (510) may include a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The input device(s) (510) may receive inputs from a user that are responsive to data and messages presented by the output device(s) (512). The inputs may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (500) in accordance with one or more embodiments. The communication interface (508) may include an integrated circuit for connecting the computing system (500) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) or to another device, such as another computing device, and combinations thereof.

[0104] Further, the output device(s) (512) may include a display device, a printer, external storage, or any other output device. One or more of the output devices may be the same or different from the input device(s) (510). The input and output device(s) may be locally or remotely connected to the computer processor(s) (502). Many different types of computing systems exist, and the aforementioned input and output device(s) may take other forms. The output device(s) (512) may display data and messages that are transmitted and received by the computing system (500). The data and messages may include text, audio, video, etc., and include the data and messages described above in the other figures of the disclosure.

[0105] Software instructions in the form of computer readable program code to perform embodiments may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable storage medium such as a solid state drive (SSD), compact disk (CD), digital video disk (DVD), storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code that, when executed by the computer processor(s) (502), is configured to perform one or more embodiments, which may include transmitting, receiving, presenting, and displaying data and messages described in the other figures of the disclosure.

[0106] The computing system (500) in FIG. 5A may be connected to or be a part of a network. For example, as shown in FIG. 5B, the network (520) may include multiple nodes (e.g., node X (522), node Y (524)). Each node may correspond to a computing system, such as the computing system shown in FIG. 5A, or a group of nodes combined may correspond to the computing system shown in FIG. 5A. By way of an example, embodiments may be implemented on a node of a distributed system that is connected to other nodes. By way of another example, embodiments may be implemented on a distributed computing system having multiple nodes, where each portion may be located on a different node within the distributed computing system. Further, one or more elements of the aforementioned computing system (500) may be located at a remote location and connected to the other elements over a network.

[0107] The nodes (e.g., node X (522), node Y (524)) in the network (520) may be configured to provide services for a client device (526), including receiving requests and transmitting responses to the client device (526). For example, the nodes may be part of a cloud computing system. The client device (526) may be a computing system, such as the computing system shown in FIG. 5A. Further, the client device (526) may include or perform all or a portion of one or more embodiments.

[0108] The computing system of FIG. 5A may include functionality to present data (including raw data, processed data, and combinations thereof) such as results of comparisons and other processing. For example, presenting data may be accomplished through various presenting methods. Specifically, data may be presented by being displayed in a user interface, transmitted to a different computing system, and stored. The user interface may include a graphical user interface (GUI) that displays information on a display device. The GUI may include various GUI widgets that organize what data is shown as well as how data is presented to a user. Furthermore, the GUI may present data directly to the user, e.g., data presented as actual data values through text, or rendered by the computing device into a visual representation of the data, such as through visualizing a data model.

[0109] As used herein, the term “connected to” contemplates multiple meanings. A connection may be direct or indirect (e.g., through another component or network). A connection may be wired or wireless. A connection may be a temporary, permanent, or semi-permanent communication channel between two entities.

[0110] The various descriptions of the figures may be combined and may include or be included within the features described in the other figures of the application. The various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, or altered as shown in the figures. Accordingly, the scope of the present disclosure should not be considered limited to the specific arrangements shown in the figures.

[0111] In the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before”, “after”, “single”, and other such terminology. Rather, ordinal numbers distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0112] Further, unless expressly stated otherwise, the conjunction “or” is an inclusive “or” and, as such, automatically includes the conjunction “and,” unless expressly stated otherwise. Further, items joined by the conjunction “or” may include any combination of the items with any number of each item, unless expressly stated otherwise.

[0113] In the above description, numerous specific details are set forth in order to provide a more thorough understanding of the disclosure. However, it will be apparent to one of ordinary skill in the art that the technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. Further, other embodiments not explicitly described above can be devised which do not depart from the scope of the claims as disclosed herein. Accordingly, the scope should be limited only by the attached claims.