EXPLAINABLE AND EFFICIENT TEXT SUMMARIZATION

[0001] This application is a continuation of U.S. application Ser. No. 18/374,676, filed on Sep. 29, 2023, which claims priority to U.S. Provisional Application No. 63/522,470, filed on Jun. 22, 2023. The entire disclosures of the above-referenced applications are incorporated herein by reference.

[0002] The present invention relates to Artificial Intelligence (AI) and machine learning, and, in particular, to a method, system and computer-readable medium for explainable and efficient text summarization.

[0003] Large Language Models (LLMs), such as ChatGPT, exhibit strong performance on many Natural Language Processing (NLP) tasks, including text summarization. Within the generated summary, however, such generative LLMs can generate information that could be false. In particular, the generated text can contain factually incorrect information, referred to also as hallucinations of fact, that at the same time appears to be stated with confidence, thereby resulting in a lack of trust and reliability of LLM systems, in addition to making their use dangerous in a number of higher risk scenarios. Moreover, the use of such LLM systems is inefficient, in terms of computational resources and compute time.

[0004] In an embodiment, the present invention provides a computer-implemented, machine learning method for generating explainable text summaries. A subset of sentences are extracted from at least one input document as an extractive summary. Context is added to the extracted sentences to generate a prompt. A fluent summary is generated by using the prompt as input to a generative language model. Source information for a sentence from the fluent summary is determined by mapping the sentence from the fluent summary to at least one sentence in the extractive summary and the at least one sentence from the extractive summary to at least one sentence from the at least one input document. A transparent summary view is generated showing the sentence from the fluent summary along with the source information from the extractive summary and the at least one input document for display on a user interface. The method has applications including, but not limited to medical AI, public safety and other machine learning applications for reliable and explainable document summarization.

[0013] Embodiments of the invention provide to make the use of LLMs more trustworthy, secure, reliable and efficient, in terms of required computation resources and/or compute time, for summarization by modifying the input in a transparent and explainable manner. At the same time, embodiments of the present invention provide to significantly reduce the costs of LLM by reducing the size of input documents are before they are given to the LLM, therefore shortening the input lengths and thus enabling to reduce the computational load, in terms of required computational resources and/or compute time. This, in turn, enables to free up computational resources for other tasks, such as other incoming queries, and allows for the processing of an increased amount of queries in a secure, reliable and trustworthy manner.

[0014] According to existing technology, LLM systems suffer from the technical deficiency of hallucinating facts and can generate wrong information in the summary (i.e., factually incorrect information with respect to the information contained within the original document). This is especially dangerous for high risk applications and use cases. For example, if a doctor makes a decision for a patient based on a summary that was obtained automatically from an LLM, and the summary contains false information, then the doctor could make the wrong decision based on the false information and risk the patient's life. For example, an LLM could generate a summary that contains factually incorrect information that the patient doesn't smoke when the original document states that the patient does smoke. In such a case, the doctor might not realize that the cause of a symptom might be smoking. Embodiments of the present invention provide to improve such LLM systems by mitigating hallucinations of fact and improving the factual accuracy of summaries, thereby also improving the reliability and trustworthiness of the LLM systems.

[0015] Also, according to existing technology, querying an LLM system is expensive in terms of computational resources and compute time because it is highly computationally intensive. For example, in the case of a call center, where the call center employee needs to summarize the last call, the cost of the employee for creating this one summary in terms of salary might be lower than querying an LLM system like ChatGPT to do the same task. The computational cost of the querying depends on the input length given to the LLM system. By definition, in summarization, the LLM is provided with very long (or even multiple) documents. This is particularly useful for reducing the reading time and cognitive burden on a human, which is very high when humans need to read large amounts of text. Embodiments of the present invention enable to reduce this cognitive load with the use of improved AI technology. At the same time, embodiments of the present invention enable to significantly reduce the cost of using LLM systems, by reducing the computational load, thereby enabling to conserve computational resources and/or compute time. In particular, embodiments of the present invention provide to reduce the input length of the LLM query, and enable to even make it as short as possible.

[0016] Embodiments of the present invention provide solutions that address both of these shortcomings of existing technology jointly, and provide to reduce the input length in a secure and reliable manner before asking the LLM to generate a fluent summary.

[0017] In a first aspect, the present invention provides a computer-implemented, machine learning method for generating explainable text summaries includes extracting a subset of sentences from at least one input document as an extractive summary and adding context to the extracted sentences to generate a prompt. A fluent summary is generated by using the prompt as input to a generative language model. Source information for a sentence from the fluent summary is determined by mapping the sentence from the fluent summary to at least one sentence in the extractive summary and the at least one sentence from the extractive summary to at least one sentence from the at least one input document. A transparent summary view is generated showing the sentence from the fluent summary along with the source information from the extractive summary and the at least one input document for display on a user interface.

[0018] In a second aspect, the present invention provides the method according to the first aspect, wherein mapping the sentence from the fluent summary to the at least one sentence in the extractive summary and/or mapping the at least one sentence from the extractive summary to the at least one sentence from the at least one input document is performed using a natural language inference model that predicts for the mapping whether a respective one of the sentences is entailed by another one of the sentences.

[0019] In a third aspect, the present invention provides the method according to the first or second aspect, wherein mapping the sentence from the fluent summary to the at least one sentence in the extractive summary is performed by embedding the sentence from the fluent summary and each respective one of the extracted sentences as a numerical vector using a sentence embedding model, and selecting a number k of the extracted sentences that are nearest neighbors to the sentence from the fluent summary as evidence in the extractive summary.

[0020] In a fourth aspect, the present invention provides the method according to any of the first to third aspects, wherein mapping the at least one sentence from the extractive summary to the at least one sentence from the at least one input document is performed by embedding each respective one of the at least one sentence from the at least one input document as a numerical vector using a sentence embedding model, and selecting, as evidence in the input documents, a number k of the at least one sentence from the at least one input document that are nearest neighbors to the number k of the extracted sentences that are in the evidence in the extractive summary.

[0021] In a fifth aspect, the present invention provides the method according to any of the first to fourth aspects, further comprising removing meaningless words and phrases from the extracted sentences prior to generating the prompt.

[0022] In a sixth aspect, the present invention provides the method according to any of the first to fifth aspects, wherein the meaningless words and phrases are determined by comparing the extracted sentences to a database containing words and phrases that have been previously classified as meaningless.

[0023] In a seventh aspect, the present invention provides the method according to any of the first to sixth aspects, further comprising determining the subset of sentences using a neural network that receives the at least one input document and outputs an informativeness score for each sentence contained in the at least one input document.

[0024] In an eighth aspect, the present invention provides the method according to any of the first to seventh aspects, wherein adding the context to the extracted sentences includes resolving ambiguities in individual ones of the extracted sentences by performing co-reference resolution and entity linking based on the at least input document.

[0025] In a ninth aspect, the present invention provides the method according to any of the first to eighth aspects, further comprising checking whether one or more of the extracted sentences is a duplicate by semantically comparing embeddings of the extracted sentences using a similarity threshold, and excluding the one or more of the extracted sentences from the prompt based on a determination that the one or more of the extracted sentences is within the similarity threshold to another one of the extracted sentences.

[0026] In a tenth aspect, the present invention provides the method according to any of the first to ninth aspects, wherein the prompt comprises a list of the extracted sentences and, for respective ones of the extracted sentences having the added context, the added context is concatenated to the respective extracted sentence, and wherein the prompt further comprises an instruction to the generative language model to summarize, paraphrase or re-write the extracted sentences, which is output as the fluent summary.

[0027] In an eleventh aspect, the present invention provides the method according to any of the first to tenth aspects, wherein the transparent summary view highlights on the user interface the sentence from the fluent summary as well as the source information including the at least one sentence from the extractive summary and the at least one sentence from the at least one input document.

[0028] In a twelfth aspect, the present invention provides the method according to any of the first to eleventh aspects, wherein the at least one input document includes patient data, and wherein the transparent summary view is used to support decision-making in a medical Artificial Intelligence (AI) or automated healthcare use case.

[0029] In a thirteenth aspect, the present invention provides the method according to any of the first to twelfth aspects, wherein the at least one input document includes a criminal investigation report, and wherein the transparent summary view is used to support decision-making in a public safety use case and/or to activate a forensic tool.

[0030] In a fourteenth aspect, the present invention provides a computer system for generating text summaries comprising one or more processors which, alone or in combination, are configured to perform a machine learning method for generating explainable text summaries according to any of the first to thirteenth aspects.

[0031] In a fifteenth aspect, the present invention provides a tangible, non-transitory computer-readable medium for generating explainable text summaries containing instructions which, upon being executed by one or more hardware processors, provide for execution of a machine learning method according to any of the first to thirteenth aspects.

[0032] FIG. 1 schematically illustrates a method and overall system architecture 100 for generating text summaries in accordance with an embodiment of the present invention. At least one document is taken as (a) input 102. This is then passed to the (1) extractive summarizer 104. Next, the (1) extractive summarizer 104 selects a subset of sentences from the (a) input 102. Then, the (3) preprocessor 108 adds context to the extracted sentences and removes meaningless words and phrases to generate the prompt for the abstractive summarizer as a (c) preprocessed summary 110. The (2) abstractive summarizer 112 (which also could be an LLM) then takes the (c) preprocessed summary 110 as input and generates (d) fluent summary 114 as output. Finally, the (4) explainer 120 receives three different summaries ((b) extractive summary 106, (c) preprocessed summary 110, (d) fluent summary 114) for the (a) input 102 and generates a transparent summary view for another AI system and/or a user.

[0033] Thus, in total, the system according to an embodiment of the present invention creates three different summaries ((b) extractive summary 106, (c) preprocessed summary 110, (d) fluent summary 114) for the (a) input 102. Then, the (e) transparent summary view 125 links these four texts ((a) input 102, (b) extractive summary 106, (c) preprocessed summary 110, (d) fluent summary 114) together in a transparent and, therefore, secure, reliable and trustworthy manner. In Table 1 below, the advantages of the different summaries and input are provided:

TABLE 1
		(b)	(c)	(d)
	(a)	Extractive	Preprocessed	Fluent
Topic	Input	Summary	Summary	Summary
Low	−	=	=	+
cognitive
burden for
humans
Low Time	−	=	=	+
requirement
for humans
Safe	+	+	−	−
Fluent	+	−	−	+
Short input	−	+	+	n/a
for lowering
LLM cost

[0034] The (1) extractive summarizer 104 is configured to extract informative textual units contained in the (a) input 102 so that the (2) abstractive summarizer 112 receives a reduced number of tokens while maintaining the same level of informativeness in the final (d) fluent summary 114. Moreover, giving only the extracted subset as input to the (2) abstractive summarizer 112 contributes to a more factually consistent summary with respect to the documents of (a) input 102.

[0035] The (1) extractive summarizer 104 receives the documents of (a) input 102 and outputs the (b) extractive summary 106, which is a part of the data contained in the documents of (a) input 102. For example, it is a collection of sentences, phrases, words, and subwords contained in documents of (a) input 102. The order between these units (e.g., sentences) can be defined to read them as natural documents.

[0036] One implementation of the (1) extractive summarizer 104, for example, could be a script that retrieves the first k sentences of each document in documents of (a) input 102, where k is a parameter of the (1) extractive summarizer 104. Another implementation is to use a neural network that receives the documents of (a) input 102 and outputs an informativeness score for each sentence in the documents of (a) input 102. If the informativeness score for a sentence is greater than a threshold m, the sentence is included in the (b) extractive summary 106. Here, m is a parameter of the (1) extractive summarizer 104. A neural network can be trained using the training data (D, L), where the label L represents the informativeness for each sentence s in document D of (a) input 102. Thus, given a document D consisting of text, the trained neural network tries to predict the label L.

[0037] FIG. 2 schematically illustrates an overall architecture of a preprocessor 200, and steps performed by the preprocessor 200, in accordance with an embodiment of the present invention. The preprocessor 200 is configured to generate a prompt for the abstractive summarizer (e.g., an LLM system such as ChatGPT) such that (1) it reduces the number of tokens of the prompt to save the computational cost and inference time, and (2) adds context to extracted sentences in order to mitigate hallucinations of fact in the final summary. Instead of using the full original text as a prompt to the abstractive summarizer, the preprocessor 200 takes a (b) extractive summary 206 as an input (which is essentially a list of sentences), and it then returns a final prompt that is given to the abstractive summarizer in order to generate an abstractive and fluent summarization.

[0038] The preprocessor 200 takes the (b) extractive summary 206 as an input. The (b) extractive summary 206 consists of a list of sentences, which were selected and ranked by the extractive summarization module (see FIG. 1). These sentences are the original sentences from the original (a) input 202. Then, each sentence is investigated individually according to an embodiment of the present invention. The individual parts of the preprocessor according to an embodiment of the present invention are marked with numbers in the figure as follows:

• • [0039] (1) A module according to an embodiment of the present invention pops a sentence 214 from the list of sentences 212 in the extractive summary and adds it to a database of previous sentences. This is done for each sentence in the extractive summary.
  • [0040] (2) The contextualizer 216 receives the document(s) of (a) input 202 and a sentence 214 from the (b) extractive summary 206, and generates an additional context for a sentence 218 so that the LLM can correctly understand the meaning of the sentence stand-alone. For example, the context could help resolving ambiguities with personal pronouns (e.g., “He” refers to “John F. Kennedy”). This component also converts the sentence 214 to the contextualized sentence 220, which can be interpreted without any context. A method and system architecture 300 of the contextualizer 216 according to an embodiment of the present invention is shown in FIG. 3 and includes parts marked with symbols in the figure as follows:
    • [0041] (α) The context comes from a co-reference resolution module 304 and entity linking 306, each of which are pretrained, connected to a knowledge graph (KG) database 308. These steps were done “offline” on the entire original document (or on the entire collection of documents if the input consists of multiple documents). Their context is used to resolve ambiguities, because many of the selected sentences from the summary will contain ambiguous terms. Such ambiguous terms could, for example, be personal pronouns, or names of places or people for which it is not clear what are they referring to (e.g., “John F. Kennedy” can refer both to the former late President of the US and to the airport in New York). For the co-reference resolution part, it is explained to what string this pronoun is referring to (e.g., in “He lived in Washington” the “He” refers to “John F. Kennedy”) using a pretrained coreference resolution model. For the entity linking 306, the description of the entity is taken from a knowledge graph from the KG database 308 and added as an additional context (e.g., in “John F. Kennedy was crowded today”, “John F. Kennedy” is an airport in New York). The entity linker 306 is pretrained to link to the knowledge graph for adding the additional context. All personal pronouns can be considered ambiguous, as well as nouns and entity names. Other ambiguous terms can be determined by looking into a database of strings. If one string has multiple meanings or entries (e.g., “bank” can be an institution, a building or a side of a river), then it can be considered to be an ambiguous term. Once the entity is resolved with respect to the reference knowledge graph, then the entity description can simply be retrieved from the knowledge graph (as this is information that most knowledge graphs have today).
    • [0042] (β) The context retriever 312 receives a sentence from the extractive summary 310 and the previously generated context. Then, it retrieves and outputs the context for the sentence 318 (e.g., retrieves a co-reference resolution phrase for a pronoun in a sentence). This step is done “online” only on the sentences from the extractive summary.
    • [0043] (γ) Finally, the sentence rewriter 314 receives the documents of (a) input 302, extracted sentence in the extracted summary 310 and context for the sentence 318, and converts them to the “contextualized” sentence 320 (contextualized sentence 220 in FIG. 2). The sentence rewriter 314 can be implemented by concatenating the context to the original sentence (e.g., in “He lived in Washington” the “He” refers to “John F. Kennedy”). Another implementation is to use a neural network which is trained to paraphrase a sentence. For example, this implementation can use a neural network that receives (s1, s2) pairs as input data, where s1 is an original sentence and s2 is the paraphrased sentence and is trained such that, given an input sentence s1, the model generates s2 such that it is different in terms of words being used, but has the same meaning as s1.
  • [0044] (3) The contextualized sentence 220 is passed to the dropping module according to an embodiment of the present invention. The dropping module drops words or phrases that are considered to be meaningless 222, in particular, words or phrases where if one is to remove them, it will not change the meaning of the original sentence, in order to produce a reduced sentence 226. The module could be, for example, a trained model or neural network that is trained to drop such meaningless words. Likewise, the module could, for example, be informed by an external database of words and phrases 224. In particular, whenever the module encounters certain words or phrases that are marked by the database 224 as meaningless (e.g., “still”, “on the other hand”, etc.), the module drops them. The database 224 can contain a list of words and phrases, including a predefined dictionary, that can be dropped, including conjunct adverbials (still, on the other hand, however, to sum up, in other words, etc.), articles (e.g., the, a) and irrelevant conjuncts (e.g., “The iphone costs $1000, but it's not worth it). Advantageously, embodiments of the present invention are agnostic with respect to the source of these words or phrases: they can come from a database, a recommender system, etc. The reduced sentence 226 is also added to the database of previous sentences 230.
  • [0045] (4) Next, a module according to an embodiment of the present invention checks if the current sentence is a semantic duplicate 228 with a previously processed sentence and, if so, filters out the sentence 226. This is achieved according to an embodiment of the present invention by using sentence embeddings to semantically compare two sentences. If the similarity is below a certain threshold, then the sentences are considered to be equivalent. If they are equivalent, then the currently considered sentence is dropped. Otherwise, the procedure continues, and the sentence is added 232 to the database of previous sentences 230. It is also possible to apply a consistency filter to recognize whether there is a contradiction in an original sentence, and if so, the sentence or contradiction can be filtered out as well.
  • [0046] (5) For each sentence, an instruction prompt is generated 236 to provide a sentence level prompt 238. When the abstractive summarizer is tuned to follow instructions (e.g., ChatGPT), the prompt consists of: (a) an instruction for the abstractive summarizer, and (b) each preprocessed sentence, as well as important context for it. The instruction for the abstractive summarizer is static and can be any prompt suggesting the abstractive summarizer to summarize the given text. For example, such prompt could be: “Rewrite the following text in a more fluent manner. Each sentence will be written on a new line. Each sentence will be written in double columns and will contain context for disambiguation after the sentence, separated by a tab. Example: “He lived in Washington”→“He” refers to “John F. Kennedy””. When the abstractive summarizer is not tuned to follow instructions (e.g., Bidirectional Auto-Regressive Transformers (BART)), the prompt consists of each preprocessed sentence only and a concatenation of them is used as the input document to the model.
  • [0047] (6) The general instruction for the LLM together with each sentence and its additional context are concatenated to generate the final prompt 240 for generating the (c) preprocessed summary 210.

[0048] Referring again now to FIG. 1, the (2) abstractive summarizer 112 receives the reduced (c) preprocessed summary 110 and outputs the (d) fluent summary 114. The (d) fluent summary 114 is the information contained in the reduced (c) preprocessed summary 110 transformed into a fluent and coherent document that is easy for the user to read. The content of the (d) fluent summary 114, which includes entities (e.g., names of people, organizations, and places), claims, and facts, must be included in the content of the reduced (c) preprocessed summary 110.

[0049] One implementation is to use a neural network that receives the reduced (c) preprocessed summary 110 and outputs a paraphrase of the content it contains as the (d) fluent summary 114. For example, by fine-tuning a pre-trained language model with a summarization dataset, a summarizer can be obtained. Alternatively, a language model already trained on a generic task (e.g., ChatGPT) can be used. Given a reduced summary (c) and task instructions (e.g., “summarize the document”), it generates an abstract summary as the (d) fluent summary 114. The (2) abstractive summarizer 112 can be run via a web application processing interface (API) or via a local computing system.

[0050] The (4) explainer 120 is configured to trace the sentences in the final (d) fluent summary 114 back to the sentences in the original documents of (a) input 102 as well as in the (b) extractive summary 106. This component improves the explainability of the summarization system, since a user can easily confirm the source of a summary sentence and whether factually inconsistent information is included in a summary.

[0051] FIG. 4 schematically illustrates an overall architecture of an explainer 400 according to an embodiment of the present invention, and steps performed by the explainer 400, in accordance with an embodiment of the present invention. The explainer 400 receives the documents of (a) input 402, the (b) extractive summary 406, the (d) fluent summary 414, and (f) each sentence in the fluent summary 408. It outputs the (e) transparent summary view 425. The explainer 400 includes a first evidence retriever 430 with a (5) tracer for a sentence in fluent summary 410, and a second evidence retriever 440 with a (6) tracer for evidence in the extractive summary 416. The explainer 400 operates as follows:

• • [0052] (1) The (5) tracer for a sentence in fluent summary 410 receives (f) each sentence from the fluent summary 408 and the (b) extractive summary 406 and outputs the (g) evidence in the extractive summary 412, which evidence are textual units (e.g., sentences) contained in the (b) extractive summary 406 that are the source information of the sentence. FIG. 5 illustrates two possible implementations 500a, 500b of the (5) tracer for a sentence in fluent summary 410 as follows:
    • [0053] First implementation 500a: The (5) tracer for a sentence in fluent summary 410 is implemented with a natural language inference (NLI) model 510 that predicts whether the meaning of one text (hypothesis) is entailed by the meaning of another text (premise), for example using an existing NLI model that is so trained. For each sentence in the (b) extractive summary 506a, it is checked whether a sentence in the fluent summary 508a entails the sentence. Then, only the sentences that are predicted as “entailment” are filtered 512 and defined as the (g) evidence in the extractive summary 514a. Since in NLI, a task is determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”, NLI models are trained to determine entailment of the hypothesis. For example, considering the premise p=“A soccer game with multiple males playing”. Then, the following hypothesis is true: h=“Some men are playing a sport”. In this case, h is entailed by p.
    • [0054] Second implementation 500b: Alternatively or additionally, the (5) tracer for a sentence in fluent summary 410 can be implemented with a sentence embedding model (dense retriever), such as the SentenceBERT model, wherein each input sentence is represented as an n-dimensional vector. A sentence in the (f) fluent summary 508b and each sentence in the (b) extractive summary 506b are converted into numerical vectors that are embeddings 520 in a latent space. Then, the distance between the vector of the fluent summary sentence and the vector of an extractive summary sentence are computed in each case using a k-nearest neighbors (kNN) retriever 522. The sentences are ranked in decreasing order of distance and the top k closest sentences from the fluent summary sentence are retrieved and provided as the (g) evidence in the extractive summary 514b, where k is a parameter that can be learned or predetermined. The process is repeated also for each sentence in the (f) fluent summary 508b, providing embeddings of each sentence in the (f) fluent summery 508b and the (b) extractive summary 506b.
  • [0055] (2) The (6) tracer for evidence in the extractive summary 416 receives the (g) evidence in the extractive summary 412 and the documents in (a) input 402 and outputs the (h) evidence in the input documents 418, which evidence are textual units (e.g. sentences) contained in the documents of (a) input 402, which are the source information of the (g) evidence in the extractive summary 412. The (6) tracer for evidence in the extractive summary 416 can be implemented by the same methods used for the (5) tracer for a sentence in fluent summary 410. Thus, in the method for the (6) tracer for evidence in the extractive summary 416 relative to the method for the (5) tracer for a sentence in fluent summary 410, the sentences in the extractive summary that are output from the (5) tracer for a sentence in fluent summary 410 as (g) evidence in the extractive summary 412 takes the place of (f) each sentence in the fluent summary 402, and the (a) input documents 402 takes the place of the (b) extractive summary 406. Embeddings are therefore determined for each sentence in the (a) input documents 402.
  • [0056] (3) The (7) summary viewer 422 receives the (d) fluent summary 414, (f) a sentence in the fluent summary 408, the (b) extractive summary 406, the (g) evidence in the extractive summary 412, the documents of (a) input 402, and the (h) evidence in the input documents 418. The (7) summary viewer 422 outputs the (e) transparent summary view 425, which highlights the fluent summary sentence and the source of information in the extractive summary and input documents. FIG. 6 illustrates an example of the (e) transparent summary view 425 including the (d) fluent summary having each sentence in the fluent summary, the (b) extractive summary including evidence text in the extractive summary and the documents of the (a) input including the evidence text in the input documents. Thus, a user can select an individual sentence from the fluent summary, and the evidence for that sentence is highlighted automatically within the extractive summary and input documents in the E transparent summary view 425.

[0057] Embodiments of the present invention can be practically applied to effect further improvements in a number of technical fields, such as medical AI, automated healthcare, AI assisted drug or material development, resource allocation and forensics.

[0058] In an embodiment, the present invention can be applied for summarization of patient diagnostics. Here, a use case could be, given doctor transcripts (from one or multiple medical doctors), to summarize the documents along with the final diagnosis of a patient. The data source (input) includes at least one document of doctor notes (from at least one doctor) about a patient with certain symptoms. The document(s) contains information about diagnosis as well as certain methods for treating the particular patient. Application of the method according to an embodiment of the present invention provides to shorten the input document(s) and create three levels of summaries: an extractive summary, a simplified summary, and a fluent summary. These summaries contain information about a patient diagnosis as well as the particular methods of how to treat the particular patient. These summaries are automatically highlighted for the doctors to inspect for safety.

[0059] In another embodiment, the present invention can be applied for summarization for knowledge work, for example for AI assisted drug development, contact center or consulting support, cyberthreat intelligence (CTI, e.g., for summarizing CTI reports), etc. Here, a use case could be, given at least one input document, generate three summaries at different levels of safety vs. fluency and lowered human cognitive burden. The data source (input) includes at least one document. Application of the method according to an embodiment of the present invention provides to shorten the input document(s) and creates three levels of summaries: an extractive summary, a simplified summary, and a fluent summary. The relations are traced between the summaries and to the original document(s). The output is three different summaries and a user interface that traces and highlights how the different summaries relate to each other and the original document(s). Actions can be taken in an automated or semi-automated fashion based on the summaries, which are used to support decision making. Also, the s1, s2 highlighted pair of sentences can also be used for computing factuality score, thus automatically estimating the faithfulness of the generated sentence in the fluent summary. This provides further information for the user, allowing to further increase trust in the AI system.

[0060] In another embodiment, the present invention can be applied for patient history summaries for medical AI or automated healthcare. Here, a use case could be, given at least one patient report from the patient's history (e.g., from Electronic Health Records (EHR)), produce a safe summary and explainer interface to the doctor. The data source (input) includes at least one patient report. Application of the method according to an embodiment of the present invention provides to shorten the input document(s) and create three levels of summaries: an extractive summary, a simplified summary, and a fluent summary. The relations are traced between the summaries and to the original document(s). The output is three different summaries and a user interface that traces and highlights how the different summaries relate to each other and the original document(s) and which gives a final summary of the patient's history on which a diagnosis can be made. Based on the report and/or the diagnosis, potential drugs or treatments could be generated in an automated or semi-automated manner.

[0061] In another embodiment, the present invention can be applied for a citizen report summary. Here, a use case could be, given at least one citizen report, produce a safe summary and explainer interface to a government worker (e.g., citizen could be seeking employment in an employment agency and government worker could be the citizen's case worker). This saves the time of the government worker and allows them to draw insights that the might have been missed otherwise. The data source (input) includes at least one citizen report. Application of the method according to an embodiment of the present invention provides to shorten the input document(s) and create three levels of summaries: an extractive summary, a simplified summary, and a fluent summary. The relations are traced between the summaries and to the original document(s). The output is three different summaries and a user interface that traces and highlights how the different summaries relate to each other and the original document(s) and which gives a final summary of the citizen's report based on which the government worker makes a decision. Based on the report, predictions for employment opportunities could be generated in an automated or semi-automated manner.

[0062] In another embodiment, the present invention can be applied for patent summarization. Here, a use case could be, given at least one patent or patent application, produce a safe summary and explainer interface. This saves the time of the patent reader and allows them to draw insights that the might have missed otherwise. The data source (input) includes at least one patent or patent application. Application of the method according to an embodiment of the present invention provides to shorten the input document(s) and create three levels of summaries: an extractive summary, a simplified summary, and a fluent summary. The relations are traced between the summaries and to the original document(s). The output is three different summaries and a user interface that traces and highlights how the different summaries relate to each other and the original document(s) and which gives a final summary of the patent or patent application to the user to make a final decision (e.g. whether the patent or patent application is relevant with regards to another patent).

[0063] In another embodiment, the present invention can be applied for suspect report summarization for automated forensic tools or public safety. Here, a use case could be, given at least one suspect report, produce a safe summary and explainer interface to a police worker. This saves the time of the police worker and allows them to draw insights that the might have missed otherwise. The data source (input) includes at least one suspect report. Application of the method according to an embodiment of the present invention provides to shorten the input document(s) and create three levels of summaries: an extractive summary, a simplified summary, and a fluent summary. The relations are traced between the summaries and to the original document(s). The output is three different summaries and a user interface that traces and highlights how the different summaries relate to each other and the original document(s) and which gives a final summary of the suspect's report based on which the police worker makes a decision. Based on the report, forensic tools could be activated in an automated or semi-automated manner.

[0064] In an embodiment, the present invention provides a method for creating summaries in a transparent, explainable and cost saving manner, the method comprising the steps of:

• • [0065] 1) Choose a method for extractive summarization and implement it.
  • [0066] 2) The extractive summarizer takes at least one document as input documents (a) and selects a subset of sentences from them.
  • [0067] 3) The preprocessor adds context to the extracted sentences (b) and removes meaningless words and phrases to generate the prompt (c) for an abstractive summarizer (e.g., ChatGPT).
  • [0068] 4) The abstractive summarizer takes the prompts (c) as input and generates fluent summary (d).
  • [0069] 5) The explainer maps sentence from (d) to (b) and sentence from (b) to (a).
  • [0070] 6) A user interface highlights how (a), (b) and (d) relate to each other.

[0071] Embodiments of the present invention enable the following improvements over existing technology:

• • [0072] 1) Generating an abstractive summary (e.g., generated by an LLM) in a more secure, transparent and reliable manner by first employing an extractive summarizer and giving only the extracted subset as input to the abstractive summarizer (e.g., an LLM).
  • [0073] 2) Reducing costs and computational resources required to generate the abstractive summary by further reducing unnecessary words in the extractive summary.
  • [0074] 3) Increasing the efficiency, security, reliability and trustworthiness of a generative summarization by employing contextualizer (e.g., co-reference resolution), and helping to avoid hallucinations of fact. In contrast, feeding raw extracted text into an abstractive summarizer can cause hallucinations.
  • [0075] 4) Increasing the efficiency, security, reliability and trustworthiness of a generative summarization by employing an explainer, which can trace from sentences in the final summary to the sentences in the original input, as well as in the intermediate representations (which are outputs of steps 2) and 3) of the above method).

[0076] Current technology of LLM systems such as ChatGPT are unsafe and expensive to use, especially with longer input query length. Embodiments of the present invention increase the safety of summarization, which is especially crucial for high risk domains, such as medicine and healthcare. Embodiments of the present invention also provide to reduce the cost because the computational costs of LLM usage increase with the increased input length, and embodiments of the present invention can be used to first safely reduce the input length.

[0077] Zhang, Haopeng, et. al., “Extractive Summarization via ChatGPT for Faithful Summary Generation,” arXiv:2304.04193 (2023), which is hereby incorporated by reference herein describe a summarization method with LLM in which they propose to adopt an extract-then-abstract strategy to improve factuality of the summary. However, in contrast to embodiments of the present invention, the summarization method does not (1) consider adding context for extracted sentence to mitigate hallucination nor (2) to equip the system to traceback from the summary to original documents.

[0078] Norkute, Milda, et. al., “Towards Explainable AI: Assessing the Usefulness and Impact of Added Explainability Features in Legal Document Summarization,” CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Article No.: 53 Pages 1-7 (May 2021), which is hereby incorporated by reference herein describe a traceback system, which is to directly traceback from a summary sentence to a sentence in original documents. However, this system is not efficient for memory because it needs to retrain the similarity scores for every pair of summary sentence and original document sentence. In contrast, embodiments of the present invention retain only the similarity scores between summary sentence and extracted sentence from original documents, thereby providing for computational improvements. This feature reduces the usage of memory and is especially beneficial when the length of the original documents is longer.

[0079] Choi, Eunsol, et. al., “Decontextualization: Making Sentences Stand-Alone,” arXiv:2102.05169 (2021), which is hereby incorporated by reference herein, describe a technique for rewriting a sentence to be interpretable out of context, while preserving its meaning. However, the technique is not applied to extracted sentences in extract-then-abstract summarization system and does not mitigate hallucinations.

[0080] Referring to FIG. 7, a processing system 700 can include one or more processors 702, memory 704, one or more input/output devices 706, one or more sensors 708, one or more user interfaces 710, and one or more actuators 712. Processing system 700 can be representative of each computing system disclosed herein.

[0081] Processors 702 can include one or more distinct processors, each having one or more cores. Each of the distinct processors can have the same or different structure. Processors 702 can include one or more central processing units (CPUs), one or more graphics processing units (GPUs), circuitry (e.g., application specific integrated circuits (ASICs)), digital signal processors (DSPs), and the like. Processors 702 can be mounted to a common substrate or to multiple different substrates.

[0082] Processors 702 are configured to perform a certain function, method, or operation (e.g., are configured to provide for performance of a function, method, or operation) at least when one of the one or more of the distinct processors is capable of performing operations embodying the function, method, or operation. Processors 702 can perform operations embodying the function, method, or operation by, for example, executing code (e.g., interpreting scripts) stored on memory 704 and/or trafficking data through one or more ASICs. Processors 702, and thus processing system 700, can be configured to perform, automatically, any and all functions, methods, and operations disclosed herein. Therefore, processing system 700 can be configured to implement any of (e.g., all of) the protocols, devices, mechanisms, systems, and methods described herein.

[0083] For example, when the present disclosure states that a method or device performs task “X” (or that task “X” is performed), such a statement should be understood to disclose that processing system 700 can be configured to perform task “X”. Processing system 700 is configured to perform a function, method, or operation at least when processors 702 are configured to do the same.

[0084] Memory 704 can include volatile memory, non-volatile memory, and any other medium capable of storing data. Each of the volatile memory, non-volatile memory, and any other type of memory can include multiple different memory devices, located at multiple distinct locations and each having a different structure. Memory 704 can include remotely hosted (e.g., cloud) storage.

[0085] Examples of memory 704 include a non-transitory computer-readable media such as RAM, ROM, flash memory, EEPROM, any kind of optical storage disk such as a DVD, a Blu-Ray® disc, magnetic storage, holographic storage, a HDD, a SSD, any medium that can be used to store program code in the form of instructions or data structures, and the like. Any and all of the methods, functions, and operations described herein can be fully embodied in the form of tangible and/or non-transitory machine-readable code (e.g., interpretable scripts) saved in memory 704.

[0086] Input-output devices 706 can include any component for trafficking data such as ports, antennas (i.e., transceivers), printed conductive paths, and the like. Input-output devices 706 can enable wired communication via USB®, DisplayPort®, HDMI®, Ethernet, and the like. Input-output devices 706 can enable electronic, optical, magnetic, and holographic, communication with suitable memory 706. Input-output devices 706 can enable wireless communication via WiFi®, Bluetooth®, cellular (e.g., LTE®, CDMA®, GSM®, WiMax®, NFC®), GPS, and the like. Input-output devices 706 can include wired and/or wireless communication pathways.

[0087] Sensors 708 can capture physical measurements of environment and report the same to processors 702. User interface 710 can include displays, physical buttons, speakers, microphones, keyboards, and the like. Actuators 712 can enable processors 702 to control mechanical forces.

[0088] Processing system 700 can be distributed. For example, some components of processing system 700 can reside in a remote hosted network service (e.g., a cloud computing environment) while other components of processing system 700 can reside in a local computing system. Processing system 700 can have a modular design where certain modules include a plurality of the features/functions shown in FIG. 7. For example, I/O modules can include volatile memory and one or more processors. As another example, individual processor modules can include read-only-memory and/or local caches.

[0089] While subject matter of the present disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. Any statement made herein characterizing the invention is also to be considered illustrative or exemplary and not restrictive as the invention is defined by the claims. It will be understood that changes and modifications may be made, by those of ordinary skill in the art, within the scope of the following claims, which may include any combination of features from different embodiments described above.

[0090] The terms used in the claims should be construed to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the article “a” or “the” in introducing an element should not be interpreted as being exclusive of a plurality of elements. Likewise, the recitation of “or” should be interpreted as being inclusive, such that the recitation of “A or B” is not exclusive of “A and B,” unless it is clear from the context or the foregoing description that only one of A and B is intended. Further, the recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise. Moreover, the recitation of “A, B and/or C” or “at least one of A, B or C” should be interpreted as including any singular entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B and C.