DEVICE, DATASTRUCTURE AND COMPUTER IMPLEMENTED METHOD FOR DIGITAL CONTENT PROCESSING

[0001] The present application claims the benefit under 35 U.S.C. § 119 of Europe Patent Application No. EP 24 19 8905.2 filed on Sep. 6, 2024, which is expressly incorporated herein by reference in its entirety.

[0002] The present invention relates to a device, a datastructure, and a computer implemented method for digital content processing.

[0003] In machine learning workflows, understanding differences of two datasets is a crucial problem, for instance: (i) comparing synthetic and real data, (ii) comparing data on which a machine learning model predicts correctly versus incorrectly, (iii) or in understanding the domain shift from a model's training data to data observed after deployment. Ideally, the differences should be described in natural language such that they are interpretable and actionable.

[0004] Multi-modal foundation models are capable of processing data modalities such as digital images or audio signals and to express semantics, e.g., in natural language, for data analysis.

[0005] For instance, semantic or geometric properties of a datum, such as a digital image or audio signal, can be expressed in natural language. Large language models are capable of acting on natural language, in particular for performing operations on natural language such as, e.g., summarization.

[0006] According to an example embodiment of the present invention, a computer implemented method for digital content processing, comprises providing a first dataset, wherein the first dataset comprises elements, providing a second dataset, wherein the second dataset comprises elements, wherein a digital content of a respective element of the elements comprises a digital image, for example a video image, a radar image, a LiDAR image, an ultrasonic image, a motion image, or a thermal image, or wherein a content of a respective element of the elements comprise a digital audio signal, generating, in particular with a data-to-text model, a first set of descriptions, wherein the first set comprises an element-wise description of the elements of the first dataset, wherein the description of the respective element of the first dataset is determined depending on the content of the respective element of the first dataset, generating, in particular with the data-to-text model, a second set of descriptions, wherein the second set comprises an element-wise description of the elements of the second dataset, wherein the description of the respective element of the second dataset is determined depending on the content of the respective element of the second dataset, determining, in particular with a large language model, common concepts in the first dataset that are non-existent in the second dataset or less frequent in the second dataset than in the first dataset, determining, in particular with a text-data-similarity metric, for the elements of the first dataset a first plurality of text-data-similarities, wherein the first plurality comprises the element-wise and common concept-wise text-data-similarity of pairs of the content of one element of the first dataset and one common concept, determining, in particular with the text-data-similarity metric, for the elements of the second dataset a second plurality of text-data-similarities, wherein the second plurality comprises the element-wise and common concept-wise text-data-similarity of pairs of the content of one element of the second dataset and one common concept, determining for the first plurality common concept-wise the average text-data similarity that is associated with the respective common concept according to the first plurality, determining for the second plurality common concept-wise the average text-data similarity that is associated with the respective common concept according to the second plurality, associating the common concepts common concept-wise with a rank, wherein the rank is determined by the average text-data similarities associated with the common concepts according to the first plurality and by the average text-data similarities associated with the common concepts according to the second plurality, selecting at least one common concept depending on the ranks that are associated with the common concepts, and outputting the selected at least one common concept.

[0007] The common concepts are text, in particular a natural language text, that provides a hypothesis about the differences between the first dataset and the second dataset. The text-data-similarity compares the similarity of the content of a respective element, i.e., the digital image or audio signal, with the text of the respective common concept. The text-data-similarity quantifies to which extent the content of the element in the pair supports the evidence for the hypothesis provided by the common concept in the pair regarding difference between the first dataset and the second dataset.

[0008] The hypothesis can be used to detect anomalies in a technical system by computing differences between a set of recent measurement on which a model makes mistakes (the first dataset) to a reference correctly classified dataset (the second dataset). The first dataset can be seen as the anomalous or rare data, and the at least one common concept allows explaining the core properties of the difference, i.e., the anomaly.

[0009] According to an example embodiment of the present invention, determining the rank may comprise ranking a common concept that has a higher average text-data similarity in the first plurality higher than a common concept that has a lower text-data similarity according to the first plurality.

[0010] According to an example embodiment of the present invention, determining the rank may comprise ranking a common concept that has a lower average text-data similarity in the second plurality higher than a common concept that has a higher text-data similarity according to the second plurality.

[0011] According to an example embodiment of the present invention, the method may comprise capturing the content of the elements with a sensor, in particular capturing the digital image with a camera, capturing the video image with a camera, capturing the radar image with a radar sensor, capturing the LiDAR image with a LiDAR sensor, capturing the ultrasonic image with an ultrasound sensor, capturing the motion image with a motion sensor, or capturing the thermal image with a thermal image sensor, or capturing the audio signal with a microphone.

[0012] In particular for comparing synthetically generated data with real-world data, the content of the elements of the first dataset is synthetically generated content, and the content of the elements of the second dataset is content captured with a sensor in the real-world.

[0013] According to an example embodiment of the present invention, the method may comprise sending the at least one common concept to at least one technical system, in particular a test bench or a vehicle or a robot, for selecting captured content depending on the at least one common concept. The method interacts with the technical system for example in the following way: The technical system collects data on which a model produces undesired behavior, e.g. misclassifications, and data on which the model behaves normally. The method explains the differences. Based on this explanation, novel data can be collected from data collected by the technical system such that it covers the problematic condition better.

[0014] According to an example embodiment of the present invention, the method may comprise receiving the content of the elements of the first dataset and/or the second dataset from at least one technical system, in particular a test bench or a vehicle or a robot. For instance, the textual description in the at least one common concept can be sent to a fleet of vehicles that apply a CLIP-based retrieval filter to select appropriate data matching the textual description. Based on this collected data, the model can be retrained.

[0015] According to an example embodiment of the present invention, a device for digital content processing comprises at least one processor, at least one memory, wherein the at least one memory comprises instructions that are executable by the at least one processor and that, when executed by the at least one processor cause the device to execute the method of the present invention.

[0016] According to an example embodiment of the present invention, a computer program may be provided, wherein the computer program comprises computer readable instructions that, when executed by the computer, cause the computer to execute the method of the present invention.

[0017] According to an example embodiment of the present invention, a datastructure may be provided, wherein the datastructure comprises at least one data field for a first dataset, wherein the first dataset comprises elements, the datastructure comprises at least one data field for a second dataset, wherein the second dataset comprises elements, wherein a digital content of a respective element of the elements comprises a digital image, for example a video image, a radar image, a LiDAR image, an ultrasonic image, a motion image, or a thermal image, or wherein a content of a respective element of the elements comprise a digital audio signal, wherein the datastructure comprises at least one data field for a first set of descriptions, generated, in particular with a data-to-text model, wherein the first set comprises an element-wise description of the elements of the first dataset, wherein the description of the respective element of the first dataset is determined depending on the content of the respective element of the first dataset, wherein the datastructure comprises at least one data field for a second set of descriptions generated, in particular with a data-to-text model, wherein the second set comprises an element-wise description of the elements of the second dataset, wherein the description of the respective element of the second dataset is determined depending on the content of the respective element of the second dataset, wherein the datastructure comprises at least one data field for common concepts in the first dataset that are non-existent in the second dataset or less frequent in the second dataset than in the first dataset, in particular common concepts determined with a large language model, wherein the datastructure comprises at least one data field for a first plurality of text-data-similarities determined, in particular with a text-data-similarity metric, for the elements of the first dataset, wherein the first plurality comprises the element-wise and common concept-wise text-data-similarity of pairs of the content of one element of the first dataset and one common concept, wherein the datastructure comprises at least one data field for a second plurality of text-data-similarities determined, in particular with the text-data-similarity metric, for the elements of the second dataset, wherein the second plurality comprises the element-wise and common concept-wise text-data-similarity of pairs of the content of one element of the second dataset and one common concept, wherein the datastructure comprises at least one data field for the average text-data similarity that is associated with the respective common concept according to the first plurality determined for the first plurality common concept-wise, wherein the datastructure comprises at least one data field for the average text-data similarity that is associated with the respective common concept according to the second plurality determined for the second plurality common concept-wise, wherein the datastructure comprises at least one data field for ranks associated with the common concepts common concept-wise, wherein the rank is determined by the average text-data similarities associated with the common concepts according to the first plurality and by the average text-data similarities associated with the common concepts according to the second plurality, wherein the datastructure comprises at least one data field for at least one common concept selected depending on the ranks that are associated with the common concepts.

[0018] Further exemplary embodiments are derived from the following description and the figures.

[0022] FIG. 1 schematically depicts a device 100 for digital content processing. The device 100 comprises at least one processor 102 and at least one memory 104. The device 100 for example comprises an interface 106 to a technical system 110. The interface 106 is configured to receive digital content from the technical system. The interface 106 is configured to send at least one common concept to the technical system 110.

[0023] The technical system 110 is for example configured to select digital content depending on the at least one common concept and to send the selected digital content to the interface 106.

[0024] The technical system 110 may be a test bench or a vehicle or a robot.

[0025] The digital content comprises for example a digital image or a digital audio signal.

[0026] The digital image is for example a video image, a radar image, a LiDAR image, an ultrasonic image, a motion image, or a thermal image.

[0027] The technical system 110 is for example configured for capturing the content with a sensor 112. The device 100 may comprise the sensor 112 instead of the sensor 112 arranged in the technical system 110.

[0028] The sensor 112 comprises for example a camera for capturing the digital image or the video image. The sensor 112 comprises for example a radar sensor for capturing the radar image. The sensor 112 comprises for example a LiDAR sensor for capturing the LiDAR image. The sensor 112 comprises for example an ultrasound sensor for capturing the ultrasonic image. The sensor 112 comprises for example a motion sensor for capturing the motion image. The sensor 112 comprises for example a thermal image sensor for capturing the thermal image. The sensor 112 comprises for example a microphone for capturing the audio signal.

[0029] The at least one memory 104 comprises instructions that are executable by the at least one processor 102 and that, when executed by the at least one processor 102 cause the device 100 to execute a method for digital content processing.

[0030] FIG. 2 depicts a flowchart comprising steps of the method for digital content processing.

[0031] The method comprises a step 202.

[0032] The step 202 comprises providing a first dataset

$D^A=\left\{x_i^a\right\}.$

[0033] The first dataset comprises n elements

$x_i^a,$

i=1, . . . , n.

[0034] The elements

$x_i^a$

comprise digital content.

[0035] For evaluating real-world content, the content of the elements

$x_i^a$

is content captured in the real-world, e.g., by the sensor 112.

[0036] The real-world content may be received from the technical system 110 or the sensor 112.

[0037] For evaluating synthetically generated content, the content of the elements

$x_i^a$

is synthetically generated content. The synthetically generated content may be generated by a generative model.

[0038] The method comprises a step 204.

[0039] The step 204 comprises providing a second dataset

$D^B=\left\{x_i^b\right\}$

[0040] The second dataset comprises m elements

$x_i^b,$

i=1, . . . , m.

[0041] The elements

$x_i^b$

comprise digital content.

[0042] For evaluating real-world content, the content of the elements

$x_i^b$

is content captured in the real-world, e.g., by the sensor 112.

[0043] The real-world content may be received from the technical system 110 or the sensor 112.

[0044] For evaluating synthetically generated content, the content of the elements

$x_i^b$

is synthetically generated content. The synthetically generated content may be generated by the generative model.

[0045] The digital content of a respective element of the elements

$x_i^a,x_i^b$

comprises for example a respective digital image.

[0046] The digital image is for example a video image, a radar image, a LiDAR image, an ultrasonic image, a motion image, or a thermal image.

[0047] The method is not limited to processing digital content comprising a digital image. The digital content of a respective element of the elements

$x_i^a,x_i^b$

may comprise a digital audio signal.

[0048] According to an example, the elements

$x_i^a,x_i^b$

comprise the same modality or modalities, i.e., digital image, digital audio signal, or both: digital image and digital audio signal.

[0049] The method comprises a step 206.

[0050] The step 206 comprises generating a first set of descriptions

$C^A=\left\{c_1^a,\dots ,c_n^a\right\}.$

[0051] The first set of descriptions C^A is for example determined with a data-to-text model f. The data-to-text model f is for example BLIP2 (arXiv:2301.12597) or LLaVa (arXiv:2304.08485).

[0052] The first set C^A comprises an element-wise description

$c_i^a,$

in particular description

$c_i^a,$

of the elements

$x_i^a$

of the first dataset D^A. The description

$c_i^a$

of the respective element

$x_i^a$

of the first dataset D^A is determined depending on the content of the respective element

$x_i^a$

of the first dataset D^A:

$c_i^a=f\left(x_i^a\right)$

[0053] The method comprises a step 208.

[0054] The step 208 comprises generating a second set of descriptions

$C^B=\left\{c_1^b,\dots ,c_n^b\right\}.$

[0055] The second set of descriptions C^B is for example determined with the data-to-text model f.

[0056] The second set C^B comprises an element-wise description

$c_i^b,$

in particular text description

$c_i^b,$

of the elements

$x_i^b$

of the second dataset D^B. The description

$c_i^b$

of the respective element

$x_i^b$

of the second dataset D^B is determined depending on the content of the respective element

$x_i^b$

of the second dataset D^B:

$c_i^b=f\left(x_i^b\right)$

[0057] The method comprises a step 210.

[0058] The step 210 comprises determining common concepts in the first dataset D^A that are non-existent in the second dataset D^B or less frequent in the second dataset D^B than in the first dataset D^A.

[0059] The common concepts are for example determined with a large language model, e.g., Mistral-7B (arXiv:2310.06825).

[0060] For example, the following steps are repeated N times (j=0, . . . N−1):

• • [0061] a. Sample K descriptions from C^A and K descriptions from C^B uniform at random.
  • [0062] b. Construct a first text prompt based on the samples text descriptions and a prompt template. The prompt template can be for instance:

[0063] “Given descriptions for two sets of measurements D^A and D^B as follows:

$A:c_{i_1}^{a}A:c_{i_2}^a,\dots ,A:c_{i_K}^{a}B:c_{i_1}^{b}B:c_{i_2}^b,\dots ,B:c_{i_K}^b$

[0064] Please list common concepts in the descriptions of set D^A that are non-existent or rare in set D^B.”, where the respective descriptions

$c_{i_k}^a,c_{i_k}^b$

are inserted.

[0065] The method is not limited to this first text prompt. More or less sophisticated prompt templates are possible and compatible.

• • [0066] c: Provide the first text prompt to the large language model and record the answers of the large language model as H_j.

[0067] H_j can be interpreted as a list of L hypotheses h_j,l regarding the differences of the two sets of measurements:

$H_j=\left\{h_{j,1},\dots ,h_{j,L}\right\}$

[0068] The N lists H_{j,j=1, . . . ,N-1} may be used as common concepts in the first dataset D^A that are non-existent in the second dataset D^B or less frequent in the second dataset D^B than in the first dataset D^A.

[0069] The N lists H_{j,j=1, . . . ,N-1} may comprise redundancy.

[0070] To remove redundancy in the hypotheses, after N times repeating the steps a, b, c, the method may comprise generating a second text prompt as follows:

[0071] “The following bullet point list contains relevant concepts that are present in a sets of measurements D^A but not in D^B: {h_1,1, . . . ,h_N-1,L}. Above bullet point list is highly redundant and too fine-grained, and should be made more concise without losing diversity of covered concepts. Do not make bullet points longer or more detailed-better abstract several concepts into a more general one. Note that redundant entries might be stated slightly different—interpret redundancy as ‘semantically similar’ concepts. Shorten the list substantially by only keeping a single representative entry for groups of redundant entries. Do not remove any entries that are not well represented by another entry.”

[0072] The method is not limited to this second text prompt. More or less sophisticated prompts are possible and compatible.

[0073] Provide the second text prompt to the large language model and record the answers of the large language model as common concepts H={h₁, . . . , h_R} in the first dataset D^A that are non-existent in the second dataset D^B or less frequent in the second dataset D^B than in the first dataset D^A.

[0074] The method comprises a step 212.

[0075] The step 212 comprises determining for the elements

$x_i^a$

of the first dataset D^A a first plurality of text-data-similarities, wherein the first plurality comprises the element-wise and common concept-wise text-data-similarity

$s_{i,j}^a$

of pairs of the content of one element

$x_i^a$

of the first dataset and one common concept h_j.

[0076] The first plurality of text-data-similarities is for example determined with a text-data-similarity metric.

[0077] The common concept h_j and the content of the element

$x_i^a$

of a pair are for example mapped in particular with a Contrastive Language-Image Pre-Training (CLIP, arXiv:2103.00020) neural network to respective embeddings in a joint embedding space. The text-data-similarity

$s_{i,j}^a$

is for example a cosine similarity of the respective embeddings in the joint embedding space.

[0078] The method comprises a step 214.

[0079] The step 214 comprises determining for the elements

$x_i^b$

of the second dataset D^B a second plurality of text-data-similarities, wherein the second plurality comprises the element-wise and common concept-wise text-data-similarity

$s_{i,j}^b$

of pairs of the content of one element

$x_i^b$

or the second dataset D^B and one common concept h_j.

[0080] The second plurality of text-data-similarities is for example determined with the text-data-similarity metric.

[0081] The common concept h_j and the content of the element

$x_i^b$

of a pair are for example mapped in particular with the CLIP neural network to respective embeddings in the joint embedding space. The text-data-similarity

$s_{i,j}^b$

is for example a cosine similarity of the respective embeddings in the joint embedding space.

[0082] The method comprises a step 216.

[0083] The step 216 comprises determining for the first plurality common concept-wise the average text-data similarity that is associated with the respective common concept according to the first plurality.

[0084] The method comprises a step 218.

[0085] The step 218 comprises determining for the second plurality common concept-wise the average text-data similarity that is associated with the respective common concept according to the second plurality.

[0086] The method comprises a step 220.

[0087] The step 220 comprises associating the common concepts common concept-wise with a rank.

[0088] The rank is determined by the average text-data similarities associated with the common concepts according to the first plurality and by the average text-data similarities associated with the common concepts according to the second plurality.

[0089] Determining the rank may comprise ranking a common concept that has a higher average text-data similarity in the first plurality higher than a common concept that has a lower text-data similarity according to the first plurality

[0090] Determining the rank may comprise ranking a common concept that has a lower average text-data similarity in the second plurality higher than a common concept that has a higher text-data similarity according to the second plurality.

[0091] The rank is for example determined with a metric R that determines how well hypothesis h_j allows distinguishing measurements from first dataset D^A from those of the second dataset D^B, based upon the content of the elements

$x_i^a,x_i^b$

[0092] For instance, the Area under a ROC-Curve of the elements

$x_i^a,x_i^b$

is used as metric R.

[0093] The method comprises a step 222.

[0094] The step 222 comprises selecting at least one common concept depending on the ranks that are associated with the common concepts.

[0095] The method comprises a step 224.

[0096] The step 224 comprises outputting the selected at least one common concept.

[0097] The step 224 may comprise sending the at least one common concept via the interface 106 to the technical system 110. The technical system 110 may select depending on the at least one common concept digital content captured by the technical system 110 and send the selected digital content to the interface 106.

• • [0098] select the appropriate data matching the textual description.

[0099] The step 224 may comprise sending the at least one common concept to several technical systems, that are configured as described for the technical system 110.

[0100] For instance, the technical systems are vehicles of a fleet of vehicles. The textual description in the at least one common concept is sent to the fleet of vehicles. The vehicles are configured to apply a CLIP-based retrieval filter to select appropriate digital content matching the textual description and to send the selected digital content to the device 100. The vehicles for example apply the CLIP-based retrieval filter to select the appropriate digital content matching the textual description, and send the selected digital content.

[0101] The method may be applied in a training of a model. The model may be trained with the digital content of the elements, e.g. for classification or semantic segmentation.

[0102] Additional digital content for the training may be collected by sending the at least one common concept and receiving the selected digital content. Based on this collected digital content, the model may be retrained, e.g., in the step 224.

[0103] FIG. 3 schematically depicts a datastructure 300 for digital content processing.

[0104] The datastructure comprises at least one data field 302 for

• • [0105] a first dataset, wherein the first dataset comprises elements,
  • [0106] a second dataset, wherein the second dataset comprises elements, wherein a digital content of a respective element of the elements comprises a digital image, for example a video image, a radar image, a LiDAR image, an ultrasonic image, a motion image, or a thermal image, or wherein a content of a respective element of the elements comprise a digital audio signal,
  • [0107] a first set of descriptions, generated, in particular with a data-to-text model, wherein the first set comprises an element-wise description of the elements of the first dataset, wherein the description of the respective element of the first dataset is determined depending on the content of the respective element of the first dataset,
  • [0108] a second set of descriptions generated, in particular with a data-to-text model, wherein the second set comprises an element-wise description of the elements of the second dataset, wherein the description of the respective element of the second dataset is determined depending on the content of the respective element of the second dataset,
  • [0109] common concepts in the first dataset that are non-existent in the second dataset or less frequent in the second dataset than in the first dataset, in particular common concepts determined with a large language model,
  • [0110] a first plurality of text-data-similarities determined, in particular with a text-data-similarity metric, for the elements of the first dataset, wherein the first plurality comprises the element-wise and common concept-wise text-data-similarity of pairs of the content of one element of the first dataset and one common concept,
  • [0111] a second plurality of text-data-similarities determined, in particular with the text-data-similarity metric, for the elements of the second dataset, wherein the second plurality comprises the element-wise and common concept-wise text-data-similarity of pairs of the content of one element of the second dataset and one common concept,
  • [0112] the average text-data similarity that is associated with the respective common concept according to the first plurality determined for the first plurality common concept-wise,
  • [0113] the average text-data similarity that is associated with the respective common concept according to the second plurality determined for the second plurality common concept-wise,
  • [0114] ranks associated with the common concepts common concept-wise, wherein the rank is determined by the average text-data similarities associated with the common concepts according to the first plurality and by the average text-data similarities associated with the common concepts according to the second plurality,
  • [0115] at least one common concept selected depending on the ranks that are associated with the common concepts.