US 12,694,263 B2Grant

Modelling causation in machine learning

Issue Date:2026-07-28

•20 Claims

•15 Drawing Sheets

Abstract

A method comprising: sampling a temporal causal graph from a temporal graph distribution specifying probabilities of directed causal edges between different variables of a feature vector at a present time step, and from one variable at a preceding time step to another variables at the present time step. Based on this there are identified: a present parent which is a cause of the selected variable in the present time step, and a preceding parent which is a cause of the selected variable from the preceding time step. The method then comprises: inputting a value of each identified present and preceding parent into a respective encoder, resulting in a respective embedding of each of the present and preceding parents; combining the embeddings of the present and preceding parents, resulting in a combined embedding; inputting the combined embedding into a decoder, resulting in a reconstructed value of the selected variable.

Metadata

Assignee

Microsoft Technology Licensing, LLC

Inventors

Wenbo Gong
Cheng Zhang
Nick Pawlowski
Joel Jennings
Karen Fassio
Marife Defante
Steve Thomas
Alice Horan
Chao Ma
Matthew Ashman
Agrin Hilmkil

Application Information

Application Number:US 17/936,347

Filing Date:2022-09-28

Priority Date:2022-09-21

Art Unit:2141

Classifications

IPC:

G06N3/04G06N3/0455G06N3/084G06N7/01G16H20/00G16H50/20

Field of Search:

G06N 3/04G06N 3/045G06N 3/088G06N 3/0455G06N 3/084G06N 20/20G06N 5/025G06N 5/01G06N 7/01G16H 50/20G16H 50/50G16H 20/00G16H 50/70G06F 17/18G06F 18/29

Patent Drawings (15 sheets)

Description

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. provisional application No. 63/376,550 filed on Sep. 21, 2022, entitled “MODELLING CAUSATION IN MACHINE LEARNING,” the entirety of which is hereby incorporated by reference herein.

BACKGROUND

[0002] Neural networks are used in the field of machine learning and artificial intelligence (AI). A neural network comprises plurality of nodes which are interconnected by links, sometimes referred to as edges. The input edges of one or more nodes form the input of the network as a whole, and the output edges of one or more other nodes form the output of the network as a whole, whilst the output edges of various nodes within the network form the input edges to other nodes. Each node represents a function of its input edge(s) weighted by a respective weight, the result being output(s) on its output edge(s). The weights can be gradually tuned based on a set of training data so as to tend towards a state where the output of the network will output a desired value for a given input.

[0003] Typically the nodes are arranged into layers with at least an input and an output layer. A “deep” neural network comprises one or more intermediate or “hidden” layers in between the input layer and the output layer. The neural network can take input data and propagate the input data through the layers of the network to generate output data. Certain nodes within the network perform operations on the data, and the result of those operations is passed to other nodes, and so on.

[0004] Each node is configured to generate an output by carrying out a function on the values input to that node. The inputs to one or more nodes form the input of the neural network, the outputs of some nodes form the inputs to other nodes, and the outputs of one or more nodes form the output of the network. At some or all of the nodes of the network, the input to that node is weighted by a respective weight. A weight may define the connectivity between a node in a given layer and the nodes in the next layer of the neural network. A weight can take the form of a scalar or a probabilistic distribution. When the weights are defined by a distribution, as in a Bayesian model, the neural network can be fully probabilistic and captures the concept of uncertainty. The values of the connections between nodes may also be modelled as distributions. The distributions may be represented in the form of a set of samples or a set of parameters parameterizing the distribution (e.g. the mean μ and standard deviation a or variance σ²).

[0005] The network learns by operating on data input at the input layer, and adjusting the weights applied by some or all of the nodes based on the input data. There are different learning approaches, but in general there is a forward propagation through the network from input layer to output layer, a calculation of an overall error, and a backward propagation of the error through the network from output layer to input layer. In the next cycle, each node takes into account the back propagated error and produces a revised set of weights. In this way, the network can be trained to perform its desired operation.

[0006] Training may employ a supervised approach based on a set of labelled training data. Other approaches are also possible, such as a reinforcement approach wherein the network each data point is not initially labelled. The learning algorithm begins by guessing the corresponding output for each point, and is then told whether it was correct, gradually tuning the weights with each such piece of feedback. Another example is an unsupervised approach where input data points are not labelled at all and the learning algorithm is instead left to infer its own structure in the experience data.

[0007] Other forms of machine learning model are also known, other than just neural networks, for example clustering algorithms, random decision forests, and support vector machines.

[0008] Some machine learning models can be designed to perform causal discovery using observational data or both observational and interventional data. That is, for a set of variables (e.g. [x₁, x₂, x₃]), the model when trained can estimate a likely causal graph describing the causal relationships between these variables. E.g. in the case of three variables a simple causal graph could be x₁→x₂→x₃, meaning that x₁causes x₂and x₂causes x₃(put another way, x₃is an effect of x₂and x₂is an effect of x₁). Or as another example, x₁→x₂←x₃, means that x₁and x₃are both causes of x₂(x₂is an effect of x₁and x₃). However, in the past such models were only used for causal discovery alone, not for treatment effect estimation for decision making. Another type of model aims to perform treatment effect estimation, which commonly assumes that the causal graph is already given by the user. Such methods did not previously work with unknown causal graphs.

[0009] “Deep End-to-End Causal Inference” [DECI] (Geffner et al, Microsoft Research, arxiv.org/pdf/2202.02195.pdf) appreciated that in many applications, users would benefit from the ability to perform treatment effect estimation for decision making with observational data only, without needing to know the causal graph. Accordingly, Geffner et al provided an integrated machine learning model that both models the causal relationships between variables and performs treatment effect estimation, by averaging over multiple possible causal graphs sampled from a distribution. This advantageously allows the method to exploit a model that has been trained for causal discovery in order to also estimate treatment effects. The method thus enables “end-to-end” causal inference.

SUMMARY

[0010] It is recognized herein that there is still further scope to improve on the DECI model developed by Geffner et al, or the like. Particularly, existing models do not model well time series data. The nature of cause-and-effect is such that causes in the past cause effects in the future. However, in a model based on only a static causal graph, there is nothing to stop the model learning or predicting that a cause in the future causes an effect in the past.

[0011] To address such issues or similar, the present disclosure discloses a model in which, instead of a static graph that contains only edges between variables at single snapshot in time, the model samples a temporal causal graph which contains causal edges between variables at different time steps.

[0012] According to one aspect disclosed herein, there is provided computer-implemented method comprising: A) selecting a selected variable from among variables of a feature vector; B) sampling a temporal causal graph from a temporal graph distribution, the temporal graph distribution specifying probabilities of directed causal edges between different ones of the variables of the feature vector at a present time step, and from one of the variables of the feature vector at a preceding time step to one of the variables of the feature vector at the present time step; and C) from among of the variables of the feature vector, identifying a present parent which is a cause of the selected variable in the present time step according to the temporal causal graph, and identifying a preceding parent which is a cause of the selected variable from the preceding time step according to the temporal causal graph. The method further comprises: D) inputting an input value of each of the identified present and preceding parent into a respective encoder, resulting in a respective embedding of each of the present and preceding parents; E) combining the embeddings of the present and preceding parents, resulting in a combined embedding; and F) inputting the combined embedding into a decoder associated with the selected variable, resulting in a reconstructed value of the selected variable.

[0013] Augmenting the model to include a temporal graph will advantageously lead to more accurate predictions as it will more accurately model the causal reality of the modelled scenario. Alternatively or additionally, the model may also be used to predict the best order in which to apply a series of two or more treatments, and/or the timing with which to apply one or more treatments; and/or to predict how long a treatment will take to take effect.

[0014] Another potential issue to take into account when handling time series data, is that if the modelled noise is static, then the model may not be optimally specified. For instance consider a scenario where a symptom of a modelled patient, or a sensor reading from a modelled device, remains relatively smooth during some periods, but becomes more erratic during other periods. It would be desirable to take this into account in the modelled noise.

[0015] Optionally to address this, the presently disclosed model may also include a history dependent noise term, which is generated based on values of variables from past time steps.

[0016] Hence in embodiments, the method may comprise generating a history dependent noise term based on embeddings of the preceding parents; and combining the history dependent noise term with the reconstructed value of the selected variable, resulting in a simulated value of the reconstructed variable.

[0017] Making the simulated noise dependent on history will lead to more optimal predictions, such as estimated treatment effects, as the model will again be more likely to be a more realistic representation of the ground truth.

[0018] Optionally, in some embodiments, the disclosed model may also take into account the presence of possible confounders, which have the potential to bias existing models. Consider two variables x₁and x₂and a model which is attempting to discover whether there is a causal edge between them, or to predict a treatment effect based on the modelled causation. A hidden confounder is a third variable which is not observed and which is a cause of both x₁and x₂. This could lead to a false conclusion that x₁is the cause of x₂(or vice versa) when in fact there is no causal link (or a weaker causal link) and instead both x₁and x₂are an effect of a common causal variable u₁₂(a confounder) which is not present in the input data (i.e. not one of the variables of the input feature vector). For instance in the healthcare field x₁could measure the presence or absence of a certain condition (e.g. disease) in a subject, whilst x₂could be a lifestyle factor such as an aspect of the subject's diet (e.g. salt or fat intake, etc.) or whether they are a smoker, and the confounder u₁₂could be a measure of a socioeconomic circumstance of the patient (e.g. annual income). Ignoring the socioeconomic circumstance may give the false impression that the lifestyle factor x₂causes the condition x₁, whereas in fact the ground truth is that both x₁and x₂are effects the socioeconomic circumstance. To address such possibilities, embodiments of the present disclosure provide a machine learning model which models the possibility of hidden confounders as latent variables model.

[0019] Accordingly, in embodiments B) may further comprise sampling a second causal graph from a second graph distribution, the second causal graph modelling presence of possible confounders, a confounder being an unobserved cause of both of two variables in the feature vector. In this case C) further comprises, from among of the variables of the feature vector, identifying a parent variable which is a cause of the selected variable according to the first causal graph, and which together with the selected variable forms a confounded pair having a respective confounder being a cause of both according to the second causal graph; and D) further comprises inputting the input value of the parent variable and an input value of the selected variable into an inference network, resulting in a latent value modelling the respective confounder of the confounded pair, and inputting the latent value into a second encoder, resulting in an embedding of the confounder of the confounded pair; and in E) the combining includes combining the embedding of the present and preceding parents with the embedding of the confounder of the confounded pair, thereby resulting in said combined embedding.

[0020] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Nor is the claimed subject matter limited to implementations that solve any or all of the disadvantages noted herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To assist understanding of embodiments of the present disclosure and to show how such embodiments may be put into effect, reference is made, by way of example only, to the accompanying drawings in which:

[0022] FIG. 1 is a schematic block diagram of a system in accordance with embodiments disclosed herein,

[0023] FIG. 2 is a schematic computation diagram illustrating an example of a machine learning model,

[0024] FIG. 3 schematically illustrates an example of a causal graph,

[0025] FIG. 4 is schematic sketch of an example of a probabilistic distribution,

[0026] FIG. 5 schematically illustrates another example of a causal graph,

[0027] FIG. 6 is a schematic computation diagram illustrating a further machine learning model,

[0028] FIG. 7 is a schematic flowchart of a method of training a model in accordance with the present disclosure,

[0029] FIG. 8 is a schematic flowchart of a method of making treatment effect estimations using a trained model in accordance with embodiments disclosed herein,

[0030] FIG. 9 schematically illustrates a causal graph (or part thereof) including a possible confounder between a pair of variables,

[0031] FIG. 10 is a schematic computation diagram illustrating a machine learning model that models the possible presence of hidden confounders in accordance with embodiments disclosed herein,

[0032] FIG. 11 is a schematic flow chart showing the core functionality of the model of FIG. 10,

[0033] FIG. 12 schematically illustrates an example of a temporal causal graph,

[0034] FIG. 13 is a schematic computation diagram illustrating a machine learning model that models temporal relationships between time series data in accordance with embodiments disclosed herein,

[0035] FIG. 14 is a schematic computation diagram of a history-dependent noise model in accordance with embodiments disclosed herein,

[0036] FIG. 15 is a schematic flow chart of a method of training the model of FIG. 13,

[0037] FIG. 16 is a schematic flow chart of a method of using the model of FIG. 13 for treatment effect estimation once trained; and

[0038] FIG. 17 gives additional exemplary detail expanding upon the flow chart of FIG. 16.

DETAILED DESCRIPTION OF EMBODIMENTS

Example System Overview

[0039] FIG. 1 illustrates an example system according to embodiments of the present disclosure. The system comprises a server system 102 of a first party, a network 112, and a client computer 114 of a second party. The server system 102 and client computer 114 are both operatively coupled to the network 112 so as to be able to communicate with one another via the network 112. The network 112 may take any suitable form and may comprise one or more constituent networks, e.g. a wide area network such as the Internet or a mobile cellular network, a local wired network such as an Ethernet network, or a local wireless network such as a Wi-Fi network, etc.

[0040] The server system 102 comprises processing apparatus comprising one or more processing units, and memory comprising one or more memory units. The, or each, processing unit may take any suitable form, e.g. a general purpose processor such as a CPU (central processing unit); or an accelerator processor or application specific processor such as a dedicated AI accelerator processor or a repurposed GPU (graphics processing unit), DSP (digital signal processor), or cryptoprocessor, etc. The, or each, memory unit may also take any suitable form, e.g. an EEPROM, SRAM, DRAM or solid state drive (SSD); a magnetic memory such as a magnetic disk or tape; or an optical medium such as an optical disk drive, quartz glass storage or magneto-optical memory; etc.

[0041] In the case of multiple processing units and/or memory units, these may be implemented in the same physical server unit, or different server units in the same rack or different racks, or in different racks in the same data centre or different data centres at different geographical sites. In the case of multiple server units, these may be networked together using any suitable networking technology such as a server fabric, an Ethernet network, or the Internet, etc. Distributed computing techniques are, in themselves, known in the art.

[0042] The memory of the server system 102 is arranged to store a machine learning (ML) model 104, a machine learning algorithm 106, training data 108, and an application programming interface (API) 110. The ML model 104, ML algorithm 106 and API 110 are arranged to run on the processing apparatus of the server system 102. The ML algorithm 106 is arranged so as, when run, to train the ML model 104 based on the training data 108. Once the model 104 is trained, the ML algorithm 106 may then estimate treatment effects based on the trained model. In some cases, after the ML model 104 been trained based on an initial portion of training data 108 and been made available for use in treatment effect estimation, training may also continue in an ongoing manner based on further training data 108, e.g. which may be obtained after the initial training.

[0043] The API 110, when run, allows the client computer 114 to submit a request for treatment effect estimation to the ML algorithm 106. The ML model 104 is a function of a plurality of variables. The request may specify a target variable to be examined, and may supply input values of one or more other variables (including intervened-on values and/or conditioned values). In response, the ML algorithm 106 may control the ML model 104 to generate samples of the target variable given the intervened-on and/or conditioned values of the one or more other variables. The API 110 returns the result of the requested causal query (the estimated treatment effect) to the client computer 114 via the network 112.

[0044] In embodiments, the API may also allow the client computer to submit some or all of the training data 108 for use in the training.

Causal Machine Learning Model

[0045] FIG. 2 schematically illustrates an example implementation of a machine learning model 104 of a type as disclosed in “Deep End-to-End Causal Inference” (DECI) by Geffner et al. The ML model 104 comprises a respective encoder

[0046]

g_{i}^{e}

and a respective decoder

[0047]

g_{i}^{d}

for each of a plurality of variables x_i, where i is an index running from 1 to D where D>1. The set of variables x_i. . . x_Dmay be referred to as the input feature vector.

[0048] Each variable represents a different property of a subject being modelled. The subject may for example be a real-life entity, such as a human or other living being; or an object such as a mechanical, electrical or electronic device or system, e.g. industrial machinery, a vehicle, a communication network, or a computing device etc.; or a piece of software such as a game, operating system software, communications software, networking software, or control software for controlling a vehicle or an industrial processor or machine. Each property could be an inherent property of the being or object, or an environment of the being or object which may affect the being or object or be affected by the being or object.

[0049] For instance in a medical example, the variables represent different properties of a person or other living being (e.g. animal). One or more of the variables may represent a symptom experienced by the living being, e.g. whether the subject is exhibiting a certain condition such as a cough, sore throat, difficulty breathing, etc. (and perhaps a measure of degree of the condition), or a measured bodily quantity such as blood pressure, heart rate, vitamin D level, etc. One or more of the variables may represent environmental factors to which the subject is exposed, or behavioural factors of the subject, such as whether the subject lives in an area of high pollution (and perhaps a measure of the pollution level), or whether the subject is a smoker (and perhaps how many per day), etc. And/or, one or more of the variables may represent inherent properties of the subject such as a genetic factor.

[0050] In the example of a device, system or software, one or more of the variables may represent an output state of the device, system or software. One or more of the variables may represent an external factor to which the device, system or software is subjected, e.g. humidity, vibration, cosmic radiation, and/or a state of one or more input signals. For instance the variables may comprise a control signal, and/or an input image or other sensor data captured from the environment. Alternatively or additionally, one or more of the variables may represent an internal state of the device, system or software, e.g. an error signal, resource usage, etc.

[0051] One, more or all of the variables may be observed or observable. In some cases, one or more of the variables may be unobserved or unobservable.

[0052] Each respective encoder

[0053]

g_{i}^{e}

is arranged to receive an input value of its respective variable x_i, and to generate a respective embedding e_i(i.e. a latent representation) based on the respective input value. As will be familiar to a person skilled in the art, an embedding, or latent representation or value, is in itself a known concept. It represents in the information in the respective input variable an abstracted form, typically in a compressed form, which is learned by the respective encoder during training. The embedding may be a scalar value, or may be a vector of dimension ‘embedding_dim’ which is greater than 1.

[0054] Note that a “value” as referred to herein could be a vector value or a scalar value. For example if one of the variables is an image (e.g. a scan of the subject), then the “value” of this vector variable is the array pixel values for the image.

[0055] The different variables x_imay have a certain causal relationship between them, which may be expressed as a causal graph. A causal graph may be described as comprising a plurality of nodes and edges (note that these are not the same thing as the nodes and edges mentioned earlier in the context of a neural network). Each node represents a respective one of the variables x_iin question. The edges are directional and represent causation. I.e. an edge from x_i=kto x_i=lrepresents that x_kcauses x_l(x_lis an effect of x_k). A simple example involving three variables is shown in FIG. 3. In this example x₂causes x₁, and x₁causes x₃. For example x₃may represent having a respiratory virus, x₁may represent a lung condition and x₂may represent a genetic predisposition.

[0056] Of course other possible graphs are possible, including those with more variables and other causal relationships.

[0057] A given graph G may be expressed as a matrix, with two binary elements for each possible pair of variables x_i=k, x_i=l; one binary element to represent whether an edge exists between the two variables, and one to represent the direction of the edge if it exists (e.g. 0=no edge from k to l, 1=edge from k to l, or vice versa). For example for three variables this could be written as:

[0058]

G = (\begin{matrix} Exists 1, 2 & Dir 1, 2 \\ Exists 1, 3 & Dir 1, 3 \\ E x i sts 2, 3 & Dir 2, 3 \end{matrix})

[0059] Other equivalent representations are possible. E.g. an alternative representation of the probabilities of existence and direction of edges in a graph would be:

[0060]

G = (\begin{matrix} E x i sts 1 \to 2 & E x i sts 1 \to 3 & E x i sts 2 \to 3 \\ E x i sts 2 \to 1 & E x i sts 3 \to 1 & E x i sts 3 \to 1 \end{matrix})

[0061] For any given situation, the actual causal graph may not be known. A distribution q_φ of possible graphs may be expressed in a similar format to G, but with each element comprising a parameter φ (phi, also drawn φ) representing a probability instead of a binary value.

[0062]

q_{φ} = (\begin{matrix} φ_exists1, 2 & φ_dir1, 2 \\ φ_exists1, 3 & φ_dir1, 3 \\ φ_exists2, 3 & φ_dir2, 3 \end{matrix})

[0063] (Or an equivalent representation.) In other words the parameter φ_exists1,2 represents the probability that an edge between x₁and x₂exists; φ_dir1,2 represents the probability that the direction of the possible edge between x₁and x₂is directed from x₁to x₂(or vice versa); parameter φ_exists1,2 represents the probability that an edge between x₁and x₃exists; etc.

[0064] Returning to FIG. 2, the ML model 104 further comprises a selector custom character

, a combiner 202 and a demultiplexer 204. It will be appreciated that these are schematic representations of functional blocks implemented in software. The selector custom character

is operable to sample a causal graph G from the distribution. This means selecting a particular graph G (with binary elements) whereby the existence and direction of the edges are determined pseudorandomly according to the corresponding probabilities in the distribution q_φ.

[0065] Preferably the possible graphs are constrained to being directed acyclic graphs (DAGs), for the sake of practicality and simplicity of modelling.

[0066] The selector custom character

also receives a value of the index i for a selected target variable x_i. For the currently selected variable x_i(node of index i in the graph), the selector custom character

selects the respective embeddings e_Pa(i)generated by the respective encoders

[0067]

g_{P a (i)}^{e}

of the parents Pa(i) of the node i (variable x_i) in the currently sampled graph G, and inputs these into the combiner 202.

[0068] The combiner 202 combines the selected embeddings e_Pa(i)into a combined embedding e_c. In embodiments the combination is a sum. In general a sum could be a positive or negative sum (a subtraction would be a sum with negative weights). The combined (summed) representation thus has the same dimension as a single embedding, e.g. ‘embedding_dim’. In alternative implementations however it is not excluded that another form of combination could be used, such as a concatenation.

[0069] The demultiplexer 204 also receives the index i of the currently selected variable x_i, and supplies the combined embedding e_cinto the input of the decoder

[0070]

g_{i}^{d}

associated with the currently selected variable x_i. This generates a value of a respective noiseless reconstructed version x_i′ of the respective variable x_ibased on the combined embedding e_c.

[0071] The encoders

[0072]

g_{i = 1 \dots D}^{e}

and decoders

[0073]

g_{i = 1 \dots D}^{d}

are constituent machine learning models comprised by the overall ML model 104. They are each parameterized by respective sets of parameters θ which are tuned during learning. In embodiments, each of the encoders

[0074]

g_{i = 1 \dots D}^{e}

and decoders

[0075]

g_{i = 1 \dots D}^{d}

is a respective neural network (in which case their parameters may be referred to as weights). However it is not excluded that some or all of them could instead be implemented with another form of constituent machine learning model.

Training

[0076] FIG. 7 schematically represents a method of training the DECI type model 104 based on a training data 108. The training data 108 comprises a plurality of training data points. Each data point [x₁. . . x_D] comprises a set of input values, one respective value for each of the variables x_i.

[0077] The method processes each of the training data points, at least in an initial portion of the training data 108. At step S10 the method takes a new data point from memory, and at step S20 the index i is set to a first value (e.g. i=1) to specify a first target variable x_ifrom among the set of variables in the data point. At step S30, this causes the selector custom character

to select the target variable x_iwith the currently set value of the index i as the variable to be processed. At step S40 the selector custom character

samples a random graph G from the distribution q_φ. At step S50, the selector custom character

selects the parents Pa(i) of the target variable x_i(node i in the graph) and supplies the respective embeddings e_Pa(i)from the encoders

[0078]

g_{Pa (i)}^{e}

of the selected parents into the combiner 202. At step S60, the combiner 202 combines (e.g. sums) the embeddings of the selected parents Pa(i) into the combined embedding e_c, and the demultiplexer 204 selects to supply the combined embedding e_cinto the decoder

[0079]

g_{i}^{d}

of the target variable x_i. The respective decoder

[0080]

g_{i}^{d}

is thus caused to generate a noiseless reconstruction x_i′ of the selected target variable x_i.

[0081] This could equally be expressed as:

[0082]

x_{i}^{'} = g_{i}^{d} (\sum_{j \in P a (i)} g_{j}^{e} (x_{j}))

where Σ_j∈Pa(i)is another way of writing the operations of the selector custom character

and combiner 202. Preferably, G is a DAG (directed and acyclic) so that it is valid to generate the value of any node in this way. The difference x_i−x_i′ may be referred to as the residual.

[0083] The process is repeated for each variable x_iin the set of variables, i.e. for i=1 . . . D, thus creating a full reconstructed set of values [x₁′ . . . x_D′] for the data point in question. This is represented schematically by steps S70 and S80 in FIG. 7, looping back to S30. However note that this the “loop” may be a somewhat schematic representation. In practice some or all of the different variables x_ifor a given data point could be processed in parallel.

[0084] At step S90 the ML algorithm 106 applies a training function which updates the parameters (e.g. weights) θ of the encoders g

[0085]

g_{i = 1 \dots D}^{e}

and decoders

[0086]

g_{i = 1 \dots D}^{d} .

Simultaneously it also updates the parameters φ of the distribution q_φ of possible graphs. The training function attempts to update the parameters θ, φ in such a way as to reduce a measure of overall difference between the set of input values of the set of input variables [x₁. . . x_D] and the reconstructed version of the set variables [x₁′ . . . x_D′]. For example in embodiments the training function is an evidence lower bound (ELBO) function. Training techniques of this kind are, in themselves, known in the art, e.g. based on stochastic back propagation and gradient descent.

[0087] During the training phase, x_iis an observed data value that is attempted to be reconstructed. The residual x_i-x_i′ between the reconstruction and the observation may be used to compute the ELBO objective.

[0088] In embodiments, the model 104 may be a probabilistic model that it assigns a joint probability (likelihood) p (X₁, . . . , X_D) to all the variables X₁, . . . , X_D; where X_irepresents the noiseless reconstruction x_icombined with a random noise term z_i, e.g. additively (X_i=x_i′+z_i), and z_imay be sampled from a random distribution, z_i{tilde over ( )}p(z_i). In such embodiments, then in order to train the model 104, one first collects a real-world data point with D-dimensional features (x₁, x₂, . . . , x_D), and then maximizes the likelihood of it being generated by the model. That is to say, the training operates to try to maximize p(X₁=x₁, X₂=x₂, . . . , X_D=x_D). This is done by adjusting the parameters (e.g. weights) θ of the model 104. The ELBO mentioned previously is an example of a numerical approximation to the quantity p(X₁=x₁, X₂=x₂, . . . , X_D=x_D). ELBO is typically much easier to compute than p(X₁=x₁, X₂=x₂, . . . , X_D=x_D).

[0089] The above-described process is repeated for each of the data points in the training data 108 (or at least an initial portion of the training data). This is represented schematically in FIG. 7 by step S100 and the loop back to S10. Though again, this form of illustration may be somewhat schematized and in some implementations, batches of data points could be reconstructed in parallel, and the model parameters updated based on the batches.

[0090] At the beginning of the overall training method the graph distribution q_φ may start with some predetermined set of probabilities, e.g. all elements 0.5 for the existence and direction of each possible edge, or using some prior domain knowledge to inform which edges are possible or impossible, or more or less likely. Then, as the model is trained with more and more data points, the probabilities (i.e. the parameters φ of q_φ) are gradually learned in parallel with the parameters θ (e.g. weights) of the encoders and decoders

[0091]

g_{i = 1, \dots, D}^{e}, g_{i = 1, \dots, D}^{d} .

Treatment Effect Estimation

[0092] Once the model 104 has been trained sufficiently based on all the data points in the training data 108 (or at least an initial portion of the initial training data), then at step S110 the trained ML model 104 is made available to be used for treatment effect estimation (i.e., answering causal queries). In embodiments, this may comprise making the model 104 available to via the API 110 to estimate treatment effects/answer causal queries requested by the client computer 114.

[0093] A method of using the ML model 104 to estimate a treatment effect is shown schematically in FIG. 8.

[0094] At step T10 the index i is set to that of a target variable x_iwhose treatment effect is to be estimated. In addition, the input values of one or more “intervened-on” variables are also set to their known values. The intervened-on variables are variables whose values are set to some specified value, to represent that the property that they represent has been controlled (the treatment, i.e. an intervention on the modelled property). An “intervened-on” variable could also be referred to as a treated variable or controlled variable. For example, in the medical application, the intervened-on variable(s) may represent one or more interventions performed on the subject, and the target variable may represent a possible symptom or condition of the subject (e.g. the presence of a certain disease). Or in the case where the subject is a device or software, the intervened-on variable(s) may represent one or more states that are set to defined values, and the target variable may represent a condition or state of the device or software that is being diagnosed.

[0095] At step T20, the selector custom character

samples a graph G pseudorandomly from the distribution q_φ according to the probabilities φ. In other words the chance of there existing an edge in given direction between each pair of nodes i=k, i=l in the sampled graph G is determined by the pair of parameters φ_exists(k,l); φ_dir(k, l). Although optionally, some of the learned probabilities could be overridden to predetermined values based on prior knowledge (e.g. in some scenarios a given causality could be ruled out—probability set to zero—or may be known to be unlikely, either based on a priori or empirical knowledge).

[0096] At step T30 the selector custom character

selects the parents Pa(i) of the target variable x_iin the sampled graph G, and supplies the respective embeddings e_Pa(i)from the encoders

[0097]

g_{Pa (i)}^{e}

of the selected parents into the combiner 202 to be combined (e.g. summed) into the combined embedding e_c. The demultiplexer 204 selects to pass the combined embedding e_cinto the decoder

[0098]

g_{i}^{d}

of the target variable x_i, thus causing it to generate a noiseless reconstruction of

[0099]

x_{i}^{'} .

[0100] Thus the selection custom character

selects the parents of node i in the graph G, so it depends on both i and G. The index is set to a particular target node i that one is trying to estimate (as well as selecting a graph G from q_φ), and the model 104 outputs a noiseless reconstruction of x_i, denoted by x_i′. During training, such reconstruction was formed of each node given the actual data for its parent nodes.

[0101] Now at test time, the model 104 is used in a slightly different way for treatment effect estimation. Here, the target variable x_iwhich is now treated as unknown, and the goal is to generate simulated values of the target variable, given other interventional variables. In some embodiments the simulated value could just be taken as the noiseless reconstructed value x_i. However for some purposes, the noiseless reconstruction x_i′ of the target variable x_imay not be considered preferable, and instead the full interventional distribution may be taken into account. This can characterized by the following sources of uncertainties: 1), different realizations of the graph G that could happen to have been selected during sampling (even an unlikely one), which can be modelled by simulating different graphs from q_φ.; and 2), the randomness of the residual noise random variables z_i. The latter can be taken into account by the following simulation equation:

[0102]

X_{i} = g_{i}^{d} (\sum_{j \in P a (i)} g_{j}^{e} (x_{j})) + z_{i}, z_{i} \sim p (z_{i}) .

[0103] In other words, during the treatment effect estimation phase, X_iis a simulated value that is obtained through random sampling of z_i, wherein X_i=x_i′+z_i(except for x_i=T, the intervened-on variable). More generally in other possible implementations, the noise z_icould be combined with the noiseless reconstruction x_i′ in other ways in order to obtain Xi, not necessarily additively.

[0104] In the following description, reference may be made to X_ias the simulated value of target variable x_i. However generally the simulated value could simply be the noiseless reconstructed value x_i′, or X_i(which takes into account noise, e.g. X_i=x_i′+z_i), or a value which is simulated based on the reconstructed value x_i′ in any other way.

[0105] By whatever means the simulated value is determined, according to the present disclosure, an average is taken over multiple sampled graphs. In embodiments, an average is taken over multiple sampled graphs and residual noise variables z_i. In other words, for a given target variable x_ito be estimated, multiple values of variable X_iare simulated based on different respective sampled graphs G, each time sampled randomly from both q_φ and z_i

[0106] This is represented schematically in FIG. 8 by the loop from step T50 back to T20, to represent that multiple simulated values of x_iare determined (for a given target variable x_i), each determined based on a different randomly sampled graph G (each time sampled from the distribution q_φ) and optionally noises z_i. Again the illustration as a loop may be somewhat schematized, and in embodiments some or all of the different simulated values for a given target variable x_i, based on the different sampled graphs and sampled residual noises z, may in fact be computed in parallel.

[0107] In embodiments the average could be taken as a simple mean, median or mode of the different values of the simulated values x_i′ or X_i′ of the variable x_i(as simulated with the different sampled graphs and optionally residual noises). In embodiments, such averaging is based on estimating an expectation of the probabilistic distribution of the simulated values.

[0108] FIG. 4 schematically illustrates the idea of a probabilistic distribution p(x_i). The horizontal axis represents the value of the target variable being simulated, x_i, and the vertical axis represents the probability that the variable takes that value. If the distribution is modelled as having a predetermined form, it may be described by one or more parameters, e.g. a mean μ and standard deviation a or variance σ²in the case of a Gaussian. Other more complex distributions are also possible, which may be parameterized by more than two parameters, e.g. a spline function. As a convenient shorthand the probabilistic distribution may be referred to as a function the target variable of x_i, i.e. p(xi), in the sense that it models the distribution of the target variable x_i, but in fact it will be appreciated that the distribution is in fact determined based on the corresponding simulated values x_i′ or X_i.

[0109] Based on the trained model 104, it is possible using statistical methods to estimate an expectation E[p(x_Y)|do(x_T=val1)] of the probabilistic distribution p of a particular target variable x_i=Ygiven the intervened value val1 of another, treatment/intervention variable x_i=T. In embodiments, the target variable x_Ymay model an outcome of a treatment modelled by x_T. Note that “treatment” as used most broadly herein does not necessarily limit to a medical treatment or a treatment of a living being, though those are certainly possible use cases. In other examples, the treatment may comprise applying a signal, repair, debugging action or upgrade to an electronic, electrical or mechanical device or system, or software, where the effect may be some state of the device, system or software which is to be improved by the system. In embodiments, the actual real-world treatment may be applied in dependence on the estimation (e.g. expectation) of the effect of the modelled treatment, for example on condition that the treatment estimation (e.g. expectation) is above or below a specified threshold or within a specified range.

[0110] One suitable measure of expectation may be referred to as the average treatment effect (ATE). This may be expressed as:

[0111]

E [p (x_{Y}) ❘ do (x_{T} = val 1)] - E [p (x_{i = Y}) ❘ (do (x_{T} = val 2)]

or

E [p (x_{Y}) ❘ do (x_{T} = val 1)] - E [p (x_{i = Y})]

which could also be expressed E[p(x_Y)|do(x_T=val1)]−E[p(x_i=Y)|do(x_T=mean(x_T))],
where val1 is some treatment, val2 is some other treatment, and “do” represents applying the treatment. In other words, the ATE is the difference between: a) the expectation of the distribution of the effect x_Ygiven the value val1 of a treatment x_Tand b) the expectation of the distribution of the effect x_Ygiven the value val2 of a treatment x_T, or the difference between a) the expectation of the distribution of the effect x_Ygiven the value val1 of a treatment x_Tand b) the expectation of the distribution of the effect x_Ywithout applying a value of the treatment x_T. Note that in variants the equality x_T=val1 or x_T=val2 could be replaced with another form of expression such as a “greater than”, “less than” or range type expression.

[0112] An estimation of E[p(x_Y)|do(x_T=treatment) may be determined based on the modelled distributions of noise, together with the learned graph and arrow functions. This may be done using the noise distribution at node x custom character

, and using noise samples from other upstream variables, plus knowledge of the graph and its behaviour under the intervention do(x_T=treatment), in order to thus make the ATE estimation calculation. The expectation E[p(x_Y)|do(x_T=val)] of a target variable x_Ygiven a treatment x_T=val may be computed as:

[0113]

E [g_{Y}^{d} (\sum_{i \in Pa (Y)} e_{i}) + z_{Y}]

where z_Yis a noise term. E is an average over everything that is random in the equation. Consider for example a simple graph:

[0114]

x_{1} \to x_{2} \leftarrow x_{3}

where x₁is the treatment x_T, and x₂is the target variable x_Y. In the computation of E, x₁is set to its known value. E may then be written:

[0115]

E [g_{Y}^{d} (g_{1}^{e} (x_{1} = val) + g_{3}^{e} (x_{3})) + z_{Y}]

[0116] To compute E, one example method is simply to take the mean of the different sampled values x_Y′ or X_Yof the target variable x_Y, based on the different sampled graphs, optionally also including some random noise (e.g. additive noise) in each sample. Another option is to fit the sampled values of x_Yto a predetermined form of distribution, such shown as in FIG. 4, e.g. a Gaussian, normal or spline function (again optionally also including random noise). The average may then be determined as the average of the fitted distribution.

[0117] Note that during the determination of the multiple instances of x_i′ it is possible, especially for some distributions, that the same graph G ends up being sampled multiple times, or even every time. However the graph is still being freshly sampled each time and even if the sampled graphs turn out to be the same, they may still be described as different instances of the sampled graph, in the sense that they are individually sampled anew each time. In the case where the multiple graphs happen to be the same, then the averaging only averages the noise z.

[0118] In some cases, if the treated (i.e. controlled) variable has one or more parents, the graph G may be mutilated: G→G_do(xT=val). E.g. consider the graph:

[0119]

x_{4} \to x_{1} \to x_{2} \leftarrow x_{3}

where again x₁is the treatment x_T, and x₂is the target variable x_Y. In the computation of E, x₁is set to its known value. Therefore x₄has no effect on the outcome of x₂. So in the determination of the expectation, the graph is mutilated to remove the node x₄and the edge from x₄→x₁. In other words, fixing the value of the known (controlled) variable x₁means that any effect of the edge from the parent of the known variable x₁.

[0120] Another type of average which may be determined according to embodiments disclosed herein may be referred to as the conditional ATE (CATE). This is the difference between: a) the expectation of the target variable x_Ygiven the treatment x_T=val1 conditional on an observation γ of at least one other of the variables x_Cin the set, and b) the expectation of the target variable x_Ywithout the treatment (either with a different treatment val2 or no treatment) but still conditional on the same observation of x_C. That is, or:

[0121]

E [p (x_{Y}) ❘ do (x_{T} = val 1), x_{C} = Y] - E [p (x_{Y}) ❘ do (x_{T} = val 2), x_{C} = Y]

or :

E [p (x_{Y}) ❘ do (x_{T} = val 1), x_{C} = Y] - E [p (x_{Y}) ❘, x_{C} = Y]

which could also be expressed E[p(x_Y)|do(x_T=val1), x_C=γ]−E[p(x_i=Y)|do(x_T=mean(x_T)) x_C=γ].

[0122] Note that in variants the equality x_T=val1, x_T=val2, or x_C=γ could be replaced with another form of expression such as a “greater than”, “less than” or range type expression.

[0123] FIG. 5 illustrates by way of example why estimating the conditional treatment effect is not necessarily straightforward. In this example x₃is the target xv whose treatment effect will be estimated and x₄is the treatment x_T, where the treatment is the cause of the target effect. In addition, there is another, unobserved cause x₁of the target effect, and another observable effect x₂of the unobserved cause x₁. The variable x₂is to be the observed condition x_C. For instance, in a medical example the target effect x₃could be some condition or symptom of the subject (e.g. a respiratory problem), the treatment x₄could be a possible medical intervention (e.g. taking some drug or vitamin), the unobserved cause x₁may be a genetic factor, and the other observable cause x₂may be some observable physical quality of the subject's body (e.g. body mass index). In general the unobserved cause could be unobservable, or merely unobserved.

[0124] The observable condition x₂contains information about the unobserved cause x₁, which in turn can reveal information about the desired effect x₃(=x_Y). For example if it is known that an athlete is in the Olympics and that they are a footballer, this reduces the probability that they are also a rower. In fact for any two variables that are effects of a common cause and whose intersection is not 100%, knowing something about one gives information about the other. E.g. if it is known that a subject has lung cancer and that they were exposed to a carcinogenic chemical other than tobacco, this reduces the probability that they were are smoker.

[0125] However the causal direction is from x₁→x₂, and the model 104 of FIG. 2 is only configured to learn effects of causes in the direction from cause to effect—it is not configured to learn inferences of effect from cause. I.e. it is not configured to “go against the arrows” in the figure (the directional causal edges).

[0126] To address this, as illustrated in FIG. 6, in embodiments the ML model 104 may be adapted to include at least one inference network h disposed between at least one observable condition x_c(x₂in the example) and at least one unobservable potential cause (x₁in the example) of the condition x_c. As illustrated in FIG. 6, in some such embodiments, the inference network h (or individual such networks) may be disposed between the unobserved cause and multiple potential effects (up to all the other variables). This will allow the model to learn which variable(s) may be an effect of the unobserved cause, if relationship is not prior knowledge.

[0127] The inference network(s) h may be trained at the training stage simultaneously along with the encoders g^eand decoders g^dand the parameters of the graph distribution q_φ, or alternatively after the rest of the model (see below). In embodiments the inference network h may comprise a neural network, in which case training the inference network comprises tuning the weights of the inference network. Alternatively the use of other forms of machine learning is not excluded for the inference network.

[0128] The inclusion of the inference model makes it possible to estimate a conditional expectation such as E[x_Y|do(x_T=val1), x_C=γ].

[0129] To estimate the conditional expectation, then initially during the estimation of the target variable x_Y, the conditional variable x_cis not fixed. In other words, the method proceeds as described above with respect to ATE, to obtain multiple different samples of x_Ybased on multiple respective sampled graphs. At the same time, respective simulated samples x_cof the conditional variable are also obtained in the same way based on the respective sampled graphs. This gives a two dimensional set of samples (x_Y, x_c), which could be described as a plot or “scatter plot”. Then a predetermined form of function is fitted to the 2D set of samples (x_Y, x_c), such as a straight line, a curve, or a probabilistic distribution. After that, xc is set to its observed value in the fitted function, and a corresponding value of x_Yis read out from the fitted function. This is taken as the conditional expectation of x_Ygiven x_c.

[0130] At least two alternative variants to computing CATE may be employed, depending on implementation.

[0131] Variant I: estimate CATE using the same approach as used to estimate ATE, but performing a re-weighing of the terms inside of the expectations such that the condition (that gives CATE its name) is satisfied. This type of approach is known as an importance sampling technique. The weights of the different samples used to compute the expectation are provided by an inference network, which is trained together with the rest of the ML model 104.

[0132] Variant II: after the model 104 of FIG. 2 has been trained and a specific CATE query is received (e.g. via the API 110), the inference network h is trained to estimate the effect variable from the conditioning variable. To train this model, data simulated from the trained model of FIG. 2 is used, while applying some treatment specified in the query. Then the conditional average treatment effect is estimated by inputting the relevant value of the conditioning variable into the inference network h. It returns a distribution over effects from which the expected effect can be computed.

[0133] The main difference between the two approaches is that the first solely uses components learnt from the observed data during model training, while the second requires learning a new network after the model 104 has been trained. The reasoning for proposing both methods is that there are specifics settings where one or the other are more computationally efficient.

[0134] Note that the disclosed methods are not limited to the controlled (i.e. treated) variable x_Tbeing a direct parent of the target variable xv (i.e. the variable whose treatment effect being estimated). The treated variable or the effect variable can be any arbitrary variable in the set x_{i=1 . . . D}, e.g. a grandparent of the target variable x_Y.

[0135] In embodiments, the simulation of the target variable takes into account a potential effect of all causal variables across the sampled graph. An example implementation of this is as follows. This may be used in conjunction with any of the ways of averaging discussed above, or others.

[0136] The method of estimating the target variable x_Y(e.g. the treatment effect) may comprise an inner and an outer loop. In the inner loop, the method loops through i=1 . . . D, so as to generate the simulated value X_i(or just x_i′) of each of the input variables x_i(even those that are not the target variable), except skipping over the one or more controlled (treated or known) variables x_Twhich are set to their known, fixed values (any edges into those nodes have been “mutilated” away, i.e. removed from the sampled graph). Then, proceeding to the next round (iteration) of the outer loop, the simulated values X_iof the non-controlled variables are fed back to the respective inputs of the model 104. In other words (other than for the one or more controlled variables x_T), the simulated values X_ifrom the previous round or cycle (iteration) of the outer loop become the input values x_iof the current iteration of the outer loop to generate an updated set of values for the simulated variables x_i. This may be repeated one or more further times, and the simulated values will start to converge (i.e. the difference between the input layer and output layer of the model 104 will get smaller each time). If noise is included the noise is frozen throughout a given inner loop, then re-sampled each outer loop. The total number of iterations of the outer loop may be predetermined, or the outer loop may be iterated until some convergence criterion is met. In embodiments the outer loop is iterated at least D-1 times, which guarantees convergence without needing to evaluate a convergence criterion.

[0137] This method advantageously allows causal effects to propagate throughout the graph. For example if x₁causes x₂and x₂causes x₃, and an intervention is performed on x1 then the outer loop will be run at least two times to propagate the effect through to x3.

[0138] However the method of using an inner and outer loop is not essential. An alternative would be to perform a topological sort of the nodes and propagate effects through in a hierarchical fashion starting from the greatest grandparents or “source nodes” (those which are only causes and not effects). Or as another alternative, though less preferred, it is not necessary to propagate effects through multiple generations and instead only the effects of immediate parents of the target variable may be taken into account.

Optional Extension to Include Latent Confounders

[0139] The following describes an optional extension to the model to accommodate the possible presence of “confounders”.

[0140] FIG. 9 schematically illustrates the issue of the presence of possible “hidden confounders”, which may also be referred to just as confounders. A confounder is a variable that is both A) unobserved (i.e. hidden, so not part of the input feature vector); and B) a cause of at least two variables that are among the variables of the input feature vector [x₁. . . x_D]. These two variables may be referred to herein as the “confounded” variables or a “confounded pair” of the respective confounder. In the example of FIG. 9 the confounded variables are labelled x₁and x₂and the confounder between them is labelled u₁₂. The confounder could be unobservable or merely unobserved. Of course the causal graph may be more complex than shown in FIG. 9, and confounders may exist between more than one pair of variables of the input feature vector.

[0141] An issue recognized herein is that existing machine learning models do not take into account the presence of possible confounders, which may lead to an erroneous bias in the training of such models or predictions made by such models. Consider for example a scenario where x₁is a variable measuring the presence or absence of a certain condition (e.g. disease) in a subject (e.g. patient), and x₂is a variable measuring a property of the subject which may or may not affect the possible presence of the condition. For instance x₂could measure a lifestyle factor such as an aspect of the subject's diet (e.g. salt or fat intake, etc.), whether they are a smoker, a dosage of a certain medication they are taking, or an environmental factor such as living in an area of high pollution or near an electricity pylon. Say then that the model is being used to try to learn whether there is a causal link between x₁and x₂, or to predict what would be the effect on x₁of treating x₂. However there may be an unobserved common cause of the two variables, i.e. a confounder. For instance, the confounder u₁₂could represent a factor in the socioeconomic circumstances of the patient (e.g. annual income, education or family circumstances). Ignoring the confounder (e.g. socioeconomic circumstance) may give the false impression that the lifestyle factor x₂causes the condition x₁, whereas in fact the ground truth is that both x₁and x₂are effects of the socioeconomic circumstance, in which case x₂may not actually be a cause of x₁or may only be a weaker cause than it would otherwise appear.

[0142] The presently disclosed extension—which in embodiments may be applied as an extension to the previous work by Geffner et al summarised in the Background section—introduces the modelling of possible hidden confounders into a causal machine learning model by introducing a second causal graph sampled from a second graph distribution.

[0143] The extended model works in a similar manner to the model 104 described in relation to FIG. 2 and like elements function in the same manner except where stated otherwise. FIG. 10 schematically illustrates the extension to the model on the encoder side. The decoder side works the same as described in relation to FIG. 2.

[0144] As illustrated in FIG. 10, as in the base model 104 of FIG. 2, the extended model 104′ comprises a respective first encoder g^e_ifor each variable x_i(i.e. each feature) of the input feature vector [x₁. . . x_D], each arranged to encode the value of its respective variable x_iinto a respective embedding e_i.

[0145] The extended model 104′ additionally comprises an inference network H, and a respective second encoder g^e_ijfor each of a plurality of pairs of input variables x_i, x_jin the input feature vector—preferably one for each possible pair (i.e. every combination of two variables from the input vector). For each of the plurality of pairs of variables x_i, x_j, the inference network H is arranged to encode the respective values of one or more of the input variables x₁. . . x_D(e.g. in training, at least the two respective values of the pair together) into a respective latent value u_ijfor the respective pair. This latent value represents a modelled value of the (possible) confounder that potentially exists between the respective pair of input variables x_i, x_j. The inference network H may also be a function of a pseudorandom noise term ε_i. These noises will be added as part of the input. That is, u=H(x, ε), as shown in FIG. 10.

[0146] In embodiments, the inference network H is implemented as one common inference network that encodes all the variables x₁. . . x_Dtogether into the latent confounder value u_ijfor a given pair. Alternatively it is not excluded that the inference network H could be structured as a respective constituent inference network H_ijfor each of the plurality of pairs of input variables input variables x_i, x_j. Other internal structures of H are also possible.

[0147] The extended model 104′ further comprises a respective second encoder g^e_ijfor each respective latent value u_ij(corresponding to each respective pair of input variables x_ij). Each respective second encoder g^e_ijis arranged to encode the respective latent value u_ijof the respective confounder into a respective embedding e_ij.

[0148] The extended model 104′ comprises a selector custom character

′ analogous to the selector custom character

described in relation to the base model of FIG. 2, but with additional functionality. As in FIG. 2, a causal graph G_φ is sampled from the graph distribution q_φ, and a certain selected variable x_iis selected; and the selector custom character

then selects the parent variables Pa_ithat are parents of the selected variable x_iaccording to sampled graph G_φ, and inputs these into the combiner 202 (e.g. adder or concatenator). The graph distribution used for this is the same as the graph distribution q_φ described previously with respect to FIG. 2. I.e. it comprises matrix whose elements represent the probabilities of directed causal edges existing between the different possible pairs of variables in the input vector [x₁. . . x_D]. In the context of the extended model 104′, this may also be described as the first graph distribution, or the directed graph distribution; and the graph sampled therefrom may be referred to as the first causal graph or the directed graph. In other words:

[0149]

q_{ϕ} = q_{1} = q_{directed}

G_{ϕ} = G_{1} = G_{directed}

[0150] However, in addition, the extended model 104′ also samples a second causal graph G₂from a second graph distribution q₂. These may also be referred to herein as the bidirected graph distribution and bidirected graph:

[0151]

q_{2} = q_{bidirected}

G_{2} = G_{bidirected}

[0152] The second graph distribution q₂is a graph distribution representing the probabilities that confounders exist between pairs of variables. It may be expressed as a matrix in which the different elements represent the probabilities that there exists a confounder between different pairs of variables x_i, x_j, preferably having an entry for each possible pair of variables in the input feature vector [x₁. . . x_D] (i.e. every combination of two variables from the input vector). So for a simple three variable input vector (D=3), the second graph distribution would have three entries:

[0153]

ψ_exists 12

ψ_exists13

ψ_exists23

where Ψ_exists(i,j) represents the probability that a confounder exists between variables with indices i and j. More generally the second graph distribution q₂may be expressed as:

[0154]

q_{2} = (\begin{matrix} ψ_exists1, 1 \\ ⋮ & ⋱ \\ ψ_exists1, D & \dots & ψ_exists (D - 1), D \end{matrix})

[0155] Other equivalent representations may also be possible.

[0156] To sample an actual graph G₂from the second graph distribution q₂, the presence or absence of each edge in the graph G₂is determined pseudorandomly according to the probability specified in the corresponding element of the distribution q₂. E.g. if ψ_exists1,1=0.7, then the element (1,2) in G2—representing whether or not a confounder u₁₂exists between variables x₁and x₂—has a 70% chance of being 1 (meaning confounder present) and a 30% chance of being 0 (meaning no confounder present). Again, other equivalent mathematical expressions of the same functionality are also possible. The value of the confounder, if it exists in the sampled graph G₂, taken from the output of the inference network H. In embodiments H may be configured to model a probabilistic distribution, which may also be described as a third distribution q₃, from which the values of u_ijare sampled.

[0157] Based on the sampled graphs G₁and G₂, in addition to the embeddings e_iof the parents Pa(i) of the selected variable x_idetermined from the first causal graph G₁(=G_φ), as sampled from the first graph distribution q₁(=q_φ), the selector custom character

′ also selects the embeddings e_ijof any confounded pairs that include the selected variable x_i—i.e. of any confounders u_ijthat affect the selected variable x₁—as determined from the second causal graph G₂sampled from the second graph distribution q₂. These embeddings e_ijof the selected confounders u_ijare input into the combiner 202 (e.g. adder or concatenator) along with the embeddings e_iof the parents Pa(i) of the selected variable x_i. The combiner 202 then combines (i.e. sums or concatenates, depending on implementation) all of the embeddings e_iof the selected parents and the embeddings e_ijof the selected confounders together into a combined embedding e_C.

[0158] As shown in FIG. 2, the combined embedding e_Cis then input into the decoder g^d_iassociated with the selected variable x_i, which decodes the combined embedding to produce a respective reconstructed value x_i′ of the input variable x_i. Optionally the pseudorandom noise z_imay be added (or included in some other, non-additive way) to create the noisy reconstructed value X_i. Either may be used as the simulated value of x_i. This side of the model 104′ (the decoder side) works the same as in the base model 104, as described previously with respect to FIG. 2.

[0159] It will be appreciated that the inference network H, second encoders g^e_ij, and selector custom character

may be implemented as modules of software in a similar manner to the other components as described in relation to the model of FIG. 2. As with the first encoders g^e_iand decoders g^d_i, each of the inference network H and second encoders g^e_ijmay be implemented as a respective neural network, or alternatively other types of constituent ML models such as random forests or clustering algorithms, or any other form of parametric or non-parametric function, are not excluded.

[0160] The core functionality of the extended model 104′ is represented in the flow chart of FIG. 11. By core functionality here, it is meant the mechanism of the model for sampling a causal graph that includes latent confounders, identifying parents of a selected variable according to the sampled graph, including any confounders, and then combining and decoding the embeddings of the identified parents, including the embeddings of any confounders. This mechanism may be used in both training and in subsequent treatment effect estimation.

[0161] At step U10 the method selects one of the variables x_ifrom among the features of the input feature vector [x₁. . . x_D]. When used in training this variable x_iis one of the variables to be reconstructed in order to determine a loss effect. When used in treatment effect estimation the selected variable x_iwill be the target variable, and the intervened-on variable (i.e. variable to be treated) will also be selected separately (though may be null). At step U20 the method samples a first causal graph G₁from the first graph distribution q₁and samples a second causal graph G₂from the second graph distribution q₂. At step U30 the method determines which of the other variables [x₁. . . x_i−1, x_i+1. . . x_D] in the feature vector are parents Pa(i) of the selected variable x_iin the sampled first graph G₁. Note this may involve mutilating the graph as described previously. The method also determines which of these parents Pa(i) share a confounder u_ijwith the selected variable x_iin the sampled second graph G₂. In general this could be none, one, some or all of the parents Pa(i) identified from the first graph G₁.

[0162] At step U40 the method generates the reconstructed value x_i′ of the selected variable x_i. This is done by inputting the input values of each of the selected parents Pa(i) (as determined from the first graph G₁) into the respective first encoders g^e_iin order to generate their respective embeddings e_i; and inputting at least the input value of the parent in each selected confounded pair x_i, x_j(as determined from the second graph G₂) into the inference network H to generate the respective latent value u_ij, and inputting this into the respective second encoder g^eijfor the respective confounder in order to generate the respective embedding e_ijfor each respective pair of confounded variables. In some circumstances, the values input into the inference network H to generate the latent value u_ijfor a confounded pair i,j (and in turn the respective embedding e_ij) may also comprise one or more additional input values in addition to the parent of the respective pair. E.g. in training these one or more additional input values preferably comprise the observed input value of the selected variable as well. And in training or inference (e.g. treatment effect estimation), the one or more additional input values may comprise input values of one or more other of the variables of the feature vector {x₁. . . x_D} (other than the selected variable x_ior the respective parent x_jof the pair i,j), as one or more of these variables may potentially also comprise information about the possible confounder. In general, H can generate u_ijfor the selected variable x_ibased on what is known about any or all of the other variables. The more is known, the better that H will predict u_ij. One (of many) ways to it is to take the average: H(x_j)=sum over x_i{H(x_i, x_j)*p(x_j)}. So H can generate u_ijfrom only a specified value of the parent x_j. H can even generate u_ijgiven nothing as input, meaning H is just generating u_ijrandomly from a distribution learned at training. When estimating average treatment effects, this is acceptable, since in the one will marginalize out all the variables x in the entire population anyway.

[0163] Continuing step U40, all the generated embeddings e_i; e_i,jfor the selected parents and confounders are input into the combiner 202 to be combined (e.g. summed or concatenated) into the combined embedding e_C, which is then decoded by the respective decoder g^d_ifor the selected variable x_ito produce the respective reconstructed value x_i′. Optionally a noise term z_imay be added to produce the simulated value X_i. Alternatively the noiseless reconstructed value x_i′ could be used as the simulated value.

[0164] It will be appreciated that FIG. 11 is somewhat schematized and all the steps of FIG. 11 do not necessarily have to be performed linearly. E.g. the method could identify the parents Pa(i) from the first graph and begin generating the parents' embeddings e_ibefore it has begun identifying all of the confounded pairs x_i, x_j. Or the method could begin generating embeddings for some parents while still selecting other parents from the graph G₁, etc.

[0165] Training of the model 104′ may be performed in an analogous manner to that already described in relation to FIGS. 2 and 7, by replacing steps S30-S70 in FIG. 7 with steps U10-U40 from FIG. 11. In other words, training begins with a given training data point comprising a set of respective values of at least some the variables (features) of the feature vector x_i=[x₁. . . x_D]. Preferably this will be values of all the variables of the feature vector if available, but the function H can take any number of observations as input, so if one or more values are unobserved then they are simply not passed through H. The method then cycles through the indices of each variable i=1 . . . D observed in the feature vector, and for each performs the method of FIG. 11 with the current variable as the selected variable in order to produce a respective simulated value x_i′ or X_i. Over the cycle, the method will thus produce a simulated value x_i′ or X_ifor each variable in the feature vector (or at least those that are observed), which together thus result in a simulated feature vector x_i′=[x₁′ . . . x_D′] or X_i=[X_i. . . X_D]. The method then determines a measure of the discrepancy between the input feature vector x_iand the simulated feature vector x_i′ or X_ifor the given training data point, and based on this tunes the parameters (e.g. weights) of all the constituent models (e.g. neural networks) of the ML model 104′, including the first encoders g^e_i, second encoders g^e_ij, inference network H, and decoders g^d_i, as well as the first and second graph distributions q₁and q₂. E.g. this may be done using an ELBO function, or any other suitable learning function. Algorithms for comparing an input feature vector with a simulated version in order to tune the parameters of a ML model are, in themselves, known in the art. The method may be repeated over multiple training data points in order to refine the training.

[0166] Once the model 104′ has been trained to at least some extent, it may be used to estimate whether a likely confounder exists between a given target pair of variables x_i, x_j(i.e. a pair of variables of interest). This may be done by accessing the trained second graph distribution q₂from memory and reading out the value of the probability ψ_exists(i, j). If the probability is high (e.g. above a threshold) it may be determined that an unobserved confounder is likely to exist between the two variables x_iand x_j, whereas conversely, if the probability is low (e.g. below a threshold) it may be determined that such a confounder is not likely to exists between the variables in question.

[0167] In dependence on this, a decision may be made as to whether or not to action a treatment on a first, intervened-on one of the two variables x_jas a means of affecting a second, targeted one of the two variables x_i. Consider a scenario where x_jis a apparently cause of x_iaccording to the first graph distribution q₁, because the relevant elements—e.g. φ_exists(l,k), φ_dir(i,j)—of the first graph distribution q₁have been read out from memory and indicate a high probability (e.g. above a threshold probability) of a directed edge existing from x_jto x_i. Thus from the first graph distribution q₁alone, it may appear that treating x_jis an effective way of affecting x_i. However, if ψ_exists(i, j) in the second graph distribution q₂also indicates a high probability (e.g. above a threshold probability) of a hidden confounder existing between x_jand x_i, then it may be determined that in fact x_jshould not be treated as a means of affecting target variable x_i.

[0168] The trained model 104′ may also be used to estimate the value of such a confounder. This may be done by reading out the value of u_ijfrom the output of the inference network after a suitable amount of training. In embodiments the inference network H may model a probabilistic distribution, which could also be described as q₃, from which the value u_ijis sampled. The decision as to whether to action the treatment may also be dependent on the value of the estimated confounder. E.g. if a confounder is determined likely to be present but having a weak effect, it may be decided still to action the treatment of x_jas a means of affecting target variable x_i.

[0169] In alternative or additional applications, the trained model 104′ may be used to perform treatment effect estimation in an analogous manner to that already described in relation to FIGS. 2 and 8, by replacing steps T10-T40 in FIG. 8 with steps U10-U40 from FIG. 11. In other words, the target variable is selected as the selected variable x_i, and input values of one or more other, intervened-on (i.e. controlled) variables x_jis/are set to its/their intended value (i.e. the proposed value for treatment). As discussed earlier the inference network H can take any number of observations, so if any variables are not intervened-on or observed, their values are simply not passed through H. The method of FIG. 11 is then used to generate a simulated value x_i′ or X_iof the target variable x_i. In embodiments this may be repeated over multiple different sampled instances of the first and second causal graphs G₁, G₂; i.e. repeating the steps of FIG. 11 but newly re-sampling the first causal graph G₁from the first graph distribution q₁and the re-sampling the second causal graph G₂from the second graph distribution q₂afresh each time (while keeping the same selection of the target variable and the same fixed values of the intervened on variable or variables x_j). The estimated effect of the treatment may then be determined by averaging over these multiple rounds of graph sampling. E.g. this may comprise determining the average treatment effect (ATE) or any other averaging technique described previously with respect to FIGS. 2 and 8.

[0170] A decision about whether or not to action the proposed treatment of the (to-be) intervened-on variable or variables x_jmay be made in dependence on the estimated treatment effect. If the treatment is determined to be effective and to have a positive effect on the target variable x_i(e.g. having above a threshold effect), then the treatment may be actioned; but otherwise the treatment may not be actioned.

Extension to Accommodate Time Series Data

[0171] FIG. 12 illustrates the concept of a temporal causal graph by way of example. The modelled scenario may be divided into a series of two or more time steps, including a current time step t representing a present time, and one or more preceding time steps t−1, t−2, . . . representing one or more points in time in the past. Conceptually the nodes of the graph may be thought of as arranged into layers, each layer corresponding to a given time step. Each node in the graph represents a particular one of the variables of the input feature vector at a given time step. In the layer representing the present time t, the graph includes nodes representing a plurality of variables x_1,t. . . x_D,tof the feature vector at the present time (up to all the features of the feature vector). In the layer representing the first preceding time step t−1, the graph includes nodes representing a plurality of variables x_1,t−1. . . x_D,t−1at the previous time step. These may include some or all of the same variables x_1,t. . . x_D,tas included at the first time step, i.e. representing the same features of the feature vector, but instantiated at the preceding time step (the values of variables may change over time). Preferably it is the same set of variables as in the first time step. Similarly each further preceding layer before that (t−2 . . . t−T), if present, represents variables of the feature vector at earlier times going backward in time. Each layer may represent some or all of the same set of variables as included in other layers, but at different steps in time. In the case of more than two time steps, these may or may not be evenly spaced in time. Preferably the time steps are evenly spaced.

[0172] As well as edges 1202 between between nodes in the layer representing variables x_1,t. . . x_D,tat the present time t, the temporal graph is able to include one or more edges 1204 directed from the nodes representing one or more variables x_1,t−1. . . x_D,t−1; . . . x_1,t−T. . . x_D,t−Tin layers representing one or more preceding steps in time t−1 . . . t−T to nodes in the layer representing variables x_1,t. . . x_D,tat the present time t. These could include an edge from a given variable at a preceding time step to the same or a different variable at the present time step. The edges could include an edge from a given variable in a preceding time step to a single other variable in the current time step, or from a given variable in a preceding time step to multiple variables at the present time step. The edges could include edges from a plurality of different variables in a given preceding time step, each to a respective set of one or more variables in the present time step, where the sets may or may not include one, some or all of the same variables. The edges could include edges from variables in a plurality of different preceding time steps, each to a respective set of one or more variables in the present time step, where again the sets may or may not include one, some or all of the same variables. In general there may be any combination of edges between any one or more variables of the feature vector at the present time step or any one or more preceding time steps and any one or more variables of the feature vector at the present time step; excepting only that edges between time steps must be directed forward in time, i.e. from a preceding time steps to the present time step. Preferably however, the graph formed by edges from present time steps to present time steps should not contain any directed cycles. (i.e. it should form a directed acyclic graph).

[0173] A particular instance of a temporal graph G may be sampled from a temporal graph distribution q_φ where q_φ now represents a temporal graph distribution rather than a static graph distribution as specified earlier. In other words q_φ now represents not only the probabilities of causal edges existing between various possible combinations of variables x_1,t. . . x_D,tat the present time t; but also from each of one or more of variables x_1,t−1. . . x_D,t−1; . . . x_1,t−T. . . x_D,t−Tat one or more preceding time steps t−1 . . . t−T each to a respective one, more or all of the variables x_1,t. . . x_D,tat the present time step t. Preferably q_φ includes an entry representing the probability of an edge existing between every possible combination variables in preceding and present time-step layers.

[0174] As a temporal graph distribution, q_φ may now be expressed as a matrix: q_φ={q^φ_t, q^φ_t−1, . . . , q^φ_t−T}; where q^φ_tis the same as the static version of q_φ that was defined earlier in relation to FIG. 2, i.e.:

[0175]

q_{t}^{φ} = (\begin{matrix} φ_exists1, 2 & dir 1, 2 \\ ⋮ & ⋱ & ⋮ & ⋱ \\ φ_exists1, D & \dots & φ_exists {(D - 1, D)}^{'} & dir 1, D & \dots & dir (D - 1, D \end{matrix})

[0176] Which could also be written as q^φ_t={φ_exists(i,j); φ_dir(i,j)} counting through all modelled possible combinations of i and j.

[0177] The parameter φ_exists(i,j) represents the probability that a causal edge exists between x_iand x_j; and φ_dir(i,j) represents the probability that the direction of the possible edge between x_iand x_jis directed from x_ito x_j(or vice versa). Other equivalent mathematical representations of the same functionality may also be possible. Note this submatrix q^φ_tis just for present time to present time, and so does not included existence of edges {1,1}, {2,2} . . . {D, D}.

[0178] The other submatrices q^φ_t−1. . . q^φ_t−Tmodelling the distribution of edges from historical time steps do not contain entries for direction as edges from the past, if they exist, must always be directed forward in time. However these submatrices may include edges {1,1}, {2,2} . . . {D, D}. Therefore

[0179]

q_{t - τ}^{φ} = (\begin{matrix} φ_{τ}_exists1, 1 & \dots & φ_{τ}_exists D, 1 \\ ⋮ & ⋱ & ⋮ \\ φ_{τ}_exists1, D & \dots & φ_{τ}_exists (D, D) \end{matrix})

where τ=1 . . . T. In other words: q^φ_t−τ={φ_τ_exists(i,j)} counting through all modelled possible combinations of i and j. The parameter φ_τ_exists(i,j) represents the probability that an edge exists between x_iat past time τ and x_jat the present time t. Again other equivalent mathematical representations of the same functionality may also be possible.

[0180] FIG. 13 shows schematically an extension of the model of FIG. 2 to accommodate the modelling of temporal causality using a temporal graph sampled from a temporal graph distribution. Based on this the extended model includes embeddings for different time steps.

[0181] The extended model 104″ is the same as the base model 104 or 104′ described in relation to FIG. 2 or 10, except where specified otherwise where the extended model 104″ builds on the base model. As well as a respective first encoder g^e_i,tfor each of the variables x_i,tat present time t, as in the model of FIG. 2, the extended model 104″ also includes a respective first encoder g^e_i,t−τ for each of a plurality of the variables x_i,t−τof the feature vector for each of the one or more preceding time steps t−τ, where τ=1 . . . T. Each first encoder g^e_i,t; g^e_i,t−τis arranged to receive an input value of its respective variable x_i,t; x_i,t−τ at its respective time step t, and to encode into a respective embedding e_i,t; e_i,t−τ for its respective combination of variable index i and time step t.

[0182] The selector custom character

receives a selection of a particular selected variable of index i from among the variables (i.e. features) of the input feature vector {x₁. . . x_D}. The selector custom character

also also takes as an input a particular sampled graph G from the graph distribution q_φ, remembering that q is now a temporal graph distribution and so G may now be a temporal graph containing one or more edges from one or more variables at a past time t−τ to one or more variables at the present time t (depending on what edges get sampled from the distribution q_φ on any given graph sampling event). The edges present in G are determined to exist or not pseudorandomly according to the corresponding probabilities specified in the temporal graph distribution q_φ.

[0183] The selector custom character

then selects the parents Pa_ithat are causes of the selected variable x_iin the sampled temporal graph G, including any parents from past time steps t−1 . . . t−T. For these parents, the respective embeddings e_j,t; e_j,t−τare generated from the respective encoder g^e, and the selector inputs these into the combiner 202 (e.g. adder or concatenator) to be combined (e.g. summed or concatenated) into the combined embedding e_Cfor the selection variable x_i, which may also be written {e_C}_lto indicate that it will vary with i. The combined embedding e_Cis input into the respective decoder g^d_iassociated with the selected variable in order to thereby generate a respective reconstructed value x_i′, in the same way as the decoder side of FIG. 2, except that this is now specifically the reconstructed value of the selected variable x_ifor in the present time step t.

[0184] Using a temporal graph can lead to more accurate predictions of treatment effects or the like since the model 104″ has the capacity to more accurately reflect the causation of the modelled scenario, which may be temporal in nature. In embodiments, the model 104″ may also be used to predict the best order in which to apply treatments, the best timing in which to apply one or more treatments, and/or to predict the likely timing of an effect of one or more treatments. In example use cases, the model 104″ (once trained) may be used to determine treatments to apply to a data centre or cloud server system, by predicting the effects of one or more server units failing or being taken offline, or having a high load. In another example, the model 104″ may be used to determine the order in which to apply controls in an autonomous vehicle or robot, or to apply certain actions in an industrial process or farming (e.g. when to apply pesticides and fertilizer, etc.).

[0185] In embodiments the reconstructed value may be combined (e.g. summed) with a noise term to generate the simulated value X_i. Alternatively the noiseless reconstructed value x_i′ could simply be used as the simulated value without including noise. If noise is included, this may be combined with the reconstructed value x_i′ as an additive noise term, or alternatively other non-additive means of combining a noise term with the reconstructed value x_i′ may be used. Either way, in some embodiments a static noise term z_ias discussed previously could be used. However more preferably, the static noise term is replaced with a history-dependent noise term ε_i,t.

[0186] The history-dependent noise term ε_i,tis a noise term that takes into account the values of the preceding parent variables Pa_i⁹(<t) of the selected variable x_iin the sampled graph G. This means the model will take into account the past smoothness or noisiness of time series data in which at least one variable has different values over time. Thus by including the optional history-dependent noise term, the model 104″ will be a better specified representation of the real-world scenario being modelled, e.g. taking into account the level of fluctuation in a symptom of a patient, or a sensor reading measuring the state of a device or its environment, or traffic conditions experienced over a network, or a condition of a crop of plants, or so forth.

[0187] A simulated value X_i,tthat is determined based on a temporal causal graph G, and which includes history dependent noise, could be expressed as:

[0188]

\begin{matrix} X_{i, t} = f_{i, t} [{Pa}_{i}^{G} (< t), {Pa}_{i}^{G} (t), ϵ_{t, i}] & (1) \end{matrix}

[0189] In embodiments disclosed herein, this formula is broken down and implemented as follows.

[0190]

\begin{matrix} X_{i, t} = f_{i} [{Pa}_{i}^{G} (< t), {Pa}_{i}^{G} (t)] + g_{i} [{Pa}_{i}^{G} (< t), ϵ_{t, i})] & (1 a) \end{matrix}

[0191] Equation 1 is not very practical in terms of modelling as it is too general. On the other hand Equation 1a (also called equation 5 later in the subsection “Example Implementation: Auto-Regressive DECI) breaks the representation down into a part that is based only on the parents from the present and preceding time steps, but not the noise; and a part that is based only on the parents from previous time steps and the noise, but not the parents from the present time step. If the model is too general as in Equation 1, then it may not work well as it cannot be identified from observational data alone. On the other hand if the model is made too specific, then it may also not work well because sometimes the data may violate the assumptions of the equation. The inventors have found that Equation 1a provides a good workable trade-off between specificity and generality.

[0192] Equation 1a may be implemented as shown in FIGS. 13 and 14. The embeddings e_i,t−τ of the identified parents Pa_i^G(<t) from any preceding times steps—but not the parents Pa_i^G(t) from the current time step—are selected by a selector 1401 and input into a noise model 1302. In embodiments these are the same embeddings that you use to generate the reconstructed values x_i′ in FIG. 13, generated by the same encoders g^e_i. Alternatively however the embeddings used here in the noise model 1302 of FIG. 14 could be separate embeddings generated by separate encoder networks than used in FIG. 13. This selector 1401 may be implemented by some or all of the same code as the selector custom character

(it will be appreciated that the figures are somewhat schematized and do not necessarily imply that the illustrated functions have to be implemented by separate modules of code). These embeddings e_i,t−τ are input into the history dependent noise model 1302 which is configured to generate the history-dependent noise term ε_i,tin dependence on the input embeddings.

[0193] An example implementation of the history-dependent noise model 1302 is shown in FIG. 14. The noise model comprises a combiner 1402 (e.g. adder or concatenator), a demultiplexer 1404, and a respective decoder g^d_iassociated with each of the possible variables x_i. . . x_Dthat may be selected as the selected variable x_i. The combiner 1402 of the noise model 1302 may share some or all of the same code with the combiner 202 of the wider model 104/104″ as described previously with respect to FIGS. 2 and 13. The demultiplexer 1404 of the noise model 1302 may share some or all of the same code with the demultiplexer 204 of the wider model 104/104″ as described previously with respect to FIGS. 2 and 13. In embodiments, the decoders g^d_iused here in the noise model 1302 of FIG. 14 may be different than those used to decode the combined embedding e_Cin the wider model 104/104′/104″ as described previously with respect to FIGS. 2 and 13. For example the x′_idecoder would only output a single value: x′_i, but in embodiments the output η_ithat is the input to the noise generator 1406 actually has a slightly different form than the output of the decoder that produces x′_i. E.g. Specifically, η_imight be the mean and variance (or standard deviation) of a distribution such as a Gaussian distribution (two values) and the noise generator 1406 would generate a sample from this distribution. Alternatively however, in other embodiments the same decoders g^d_imay be used both to decode η_iin the noise model 1402 of FIG. 14 and to decode C_iin the wider model. Again it will be appreciated that the figures are somewhat schematized and do not necessarily imply that the components 1402, 1404 and g^d_iare separate modules of code than those shown in FIG. 13).

[0194] The noise model 1302 further comprises a noise generator 1406. The combiner 1402 combines (e.g. sums or concatenates) the embeddings e_i,t−τof the selected past parents Pa_i^G(<t) into a further combined embedding C_i. The demultiplexer 1404 routes this further embedding C_iinto the respective decoder g^d_iassociated with the selected variable x_i. This decodes the embedding C_iinto a set of one or more parameter values η_i, which are the values of one or more parameters of a probabilistic distribution. The noise generator 1406 then generates the history dependent noise term ε_i,tbased on the probabilistic distribution as parameterized by the one or more parameter values. E.g. the probabilistic distribution could be a Gaussian or a spline function. FIG. 4 shows an example of a probabilistic distribution such as a Gaussian that may be parameterized by two parameters, a centre point (e.g. mean μ) and a width (e.g. standard distribution a or variance σ²). In one example implementation, the noise generator 1406 could generate the distribution parameterized by the parameter(s) η_i(the mean μ and standard deviation σ), and then sample the value of the history dependent noise term ε_i,tdirectly from the generated probabilistic distribution. Alternatively however, more preferably the noise generator 1406 implements the sampling by taking a sample E_ifrom a fixed distribution, e.g. a unit distribution such as a unit Gaussian (with mean of 0 and standard deviation of 1), and then apply a transform function F which transforms the combination of the sample E_ifrom the fixed distribution and the generated parameter(s) η_iinto a sample ε_i,tfrom the parameterized distribution (without having to actually generate the entire parameterized distribution). In other words ε_i,t=F(E_i, η_i).

[0195] The various encoders g^e_i,t; g^e_i,t−τ; decoders g^d_iand noise model 1302 may be implemented as modules of software as part of the model 104″, stored in memory and arranged to run on one or more processors in a similar manner as described previously in relation to FIG. 2.

[0196] The extended model 104″ may be trained in the same way as described previously with reference to FIGS. 2 and 7, but using a temporal graph sampled from the temporal graph distribution q_φ={q^φ_t, q^φ_t−1, . . . q^φ_t−T} instead of a static graph sampled from the static graph distribution, and including the embeddings of the past parents Pa_i^G(<t) as well as the present parents Pa_i^G(t), and optionally also including the history dependent noise ε_i,t.

[0197] FIG. 15 shows the method of training the extended, temporal model 104″. At step V10 the method receives a new temporal training data point comprising values of some or all of the variables (features) of the feature vector {x_i. . . x_D} at two or more time steps. In embodiments it is all of the variables, but alternatively the model could be extended to handle missing data. At step V20 the method samples a temporal causal graph G from the temporal graph distribution q_φ={q^φ_t, q^φ_t−1, . . . q^φ_t−T}. At step V30 the method selects a selected variable x_iwith index i from amongst the observed variables of the input feature vector.

[0198] At step V40 the method selects the instantaneous parents Pa_i^G(t), i.e. from the present time step t, of the selected variable x_i. At step V45 the method selects the lagged parents Pa_i^G(<t), i.e. from the one or more preceding time steps t−1 . . . t−T, of the selected variable x_i. At step V50 the method generates the noiseless reconstructed value of x_i′ of the selected variable x_i, by combining the embeddings e_i,t; e_i,t−1; . . . e_i,t−Tof the selected instantaneous and lagged parents. Optionally at step V55 the method also generates the history dependent noise ε_i,t. In embodiments however, only the noiseless reconstruction x′_iis used during training, whereas the simulation X_iwith noise is used in treatment effect estimation. At step V60 the method determines the difference between the simulated value X_iof the selected variable and the actual input value x_i. The simulated value may include the noise term, or in less preferred embodiments the noiseless reconstruction x_i′ could be used as the simulated value in step V60.

[0199] At step V70 the method loops back to step V30 via step V80 where it selects a new one of the observed variables of the feature vector, and repeats the method until reconstructed x_i′ values have been generated for the whole feature vector. At step V90 the method then updates the parameters (e.g. weights) of all the constituent models (e.g. NNs) which include at least the encoders g^e, decoders g^dand temporal graph distribution q_φ. The update is done based on the evaluating a measure of overall loss between the simulated value X_ior reconstructed value x′_i, of the feature vector and the input value x_i. E.g. this may be done using an ELBO function. In embodiments only the noiseless reconstruction is used during training, whereas the simulation with noise is used in treatment effect estimation. At step V100 the method loops back to V10 and repeats with a new training data point, tuning the parameters of the model 104″ each time. This continues until the last training data point in the batch has been used, then the method proceeds to step V110 where the model 104″ is made available to be used for making predictions, e.g. treatment effect estimation.

[0200] The method of using the extended temporal model 104″ to perform treatment effect estimation is analogous to that described earlier with reference to FIGS. 2 and 8. FIG. 16 shows a particular example method of using the model 104″ to estimate that treatment effect that will be experienced at a future point in time.

[0201] At step W10 the method selects a target variable x_Aas the selected variable and a future time t′ as a target time. A here is the value of the index i for the target variable. The method also selects at least one variable x_Bto be intervened on (i.e. treated). This may be referred to herein as the intervened-on variable, meaning the variable for a which an intervention (treatment) is proposed, i.e. an effect of intervening on that variable is to be estimated. B is the value of the index i of the intervened on variable x_B. The method also selects a value l of the proposed intervention on x_B, and a time t{tilde over ( )} as the time of the proposed intervention. The target time t′ is the time for which the effect on the target variable is to be estimated, i.e. the time of interest. The time t{tilde over ( )} on the other hand is the time at which it is proposed to perform the intervention l on the intervened-on variable x_B.

[0202] The method begins with t=t{tilde over ( )} where t{tilde over ( )} is the start time, and at W20 samples a temporal graph G. At step W30 the method extracts instantaneous observations of the feature vector {x_i}, where i counts through the indices 1 . . . D of some or all of the features of the feature vector. This may include at least those that are observed and/or treated, and/or at least those that are immediate parents of the target variable i=A in the sampled graph. For any treated (i.e. intervened-on) variables x B these are set to their fixed, treated values l. For any observed values these are set to their observed values. In embodiments it may be assumed that observed or treated values are available for the entire feature vector at the start time t=t{tilde over ( )}. At step W35 the method extracts any past observations of the feature vector {x₁. . . x_D}, i.e. for times t−T . . . t−1. At step W50 the method uses the values of the present and previous parents to generate the reconstructed value x′_i,t, cycling through the indices i=1 . . . D of some or all of the variables as the selected variable so as to reconstruct some or all of the feature vector in the manner discussed previously. The reason for reconstructing multiple variables, even though only one target variable x_Amay be ultimately of interest, is that when iterated over multiple time steps then these may have a knock on effect on the target variable x_Aeven if not immediate parents of x_A. At optional step W40 the method generates the noise distribution parameter(s) η_i, for some or all of i=1 . . . D, based on the historical data and sampled past parents, in the manner described previously; and then step W55 generates the history dependent noise based on this. At step W60 the method generates the simulated values {X_i,t} for i=1 . . . D.

[0203] Note: the representation of steps W20-W60 in FIG. 16 may be somewhat of a simplification compared to the actual implementation. In embodiments these steps may be arranged into a sub-loop to count through i=1 . . . D, reconstructing each variable in turn, where the order of the indices from 1 to D preferably represent a topological order of the nodes in the sampled graph. That is, in the first cycle of the subloop the value of the variable i=1 with the highest topological order is determined, and then in the next cycle of the subloop the value of variable i=2 with the next highest topological order is determined, which may depend on the reconstructed or simulated value of the first variable, and so forth. Operating in this manner means that the method can takes into account that some variables may have a knock-on effect on others (not necessarily just their immediate parents). For example at a given t, x₂might have knock-on effects on x₃(or similar), and as such the sample X_i,tcan have an impact on X_j,t. The sub-loop will take account of this. This is shown in FIG. 17.

[0204] At step W70 the method loops back, via step W75 where t is incremented by one step, to the point of branching to steps W30/W35/W40. Each time the reconstructed or simulated values from the preceding step are fed back to the input of the model as the input values, unless intervened-on or observed values of the variables are of be used for any intermediate time steps between start time t=t{tilde over ( )} and target time t=t′ in which case those values are used for those variables. The method continues repeating through to step W60 until future time t′ is reached. After this the method proceeds to step W80 where, if not finished, it loops back to step W20 and repeats one or more times, resampling the graph at step W20 each time. Once the method has completed all the desired graph sampling loops, it finally proceeds to step W90 are an average treatment effect for the target variable x_iof index i is determined averaged over all the sampled graphs.

[0205] The temporal graph is only sampled once throughout the entire time t to t′. Since there is additional time information, both the variable index i and a future time t′ are specified (t′>=t) for the target variable. Therefore, in order to simulate a variable in the future, one needs to start with the current time t and simulate the all variables at the current time t (W30-W60). Then, one can proceed to simulate the variables at t+1. Repeat the above procedure until reaching t′ (the loop of W70). This will generate simulation values of the target variable w.r.t the current temporal graph. Since it is desired to average multiple graphs, the procedure is then repeated one or more times (the loop of W80). Note also again that the flow charts may be somewhat schematized and in practice one or more steps and/or loops shown sequentially may in fact be performed in parallel.

[0206] FIG. 17 shows explicitly the sub-loop mentioned previously with respect to steps W20-W60 of FIG. 16. FIG. 17 may be considered as a more detailed expansion of FIG. 16. At step W10 the method selects the variable x_Awith index i=A as the target variable, and selects at least one variable x_B(index i=B) as the intervened-on variable, and l as the value of the intervention. The method also selects time t{tilde over ( )} as the time of the intervention and t′ as the target future time. For example if the model 104″ is being used to predict the effect of a heating control loop, x_Acould be the temperature of a target environment or device, x_Bcould be the thermostat setting, l the value to which the thermostat is proposed to be set (the proposed treatment), time t{tilde over ( )} is the time of applying the thermostat setting, and time t′ is the time for which it is desired to predict the resulting change in the temperature x_A. The method begins with the current time t set to an initial value, e.g. t=1, which could be the treatment time t{tilde over ( )} or a time earlier than the treatment time.

[0207] At step W20 the method samples a temporal graph G from the graph distribution q_φ. At step W25 the method sets i=1. Preferably the index i counts in topological order of the nodes in the sampled graph, such that the variable x₁indexed by i=1 has no parents, then the variable x₂indexed by i=2 has either no parents or only x₁as a parent, and so forth. At step W27 the method determines whether the current time t equals the treatment time t{tilde over ( )} AND the current value of the variable index i equals B, that of the intervened-on variable (i.e. treated variable) x_B. If both of these conditions are met then the method branches to step W30a where it sets the treated variable x_Bto the value of the intervention l. Though only a single treatment index value B is shown, more generally B could represent a set of one or more index values and l could represent a set of one or more corresponding intervention values, such that one or more variables i will be set to their corresponding intervened-on values l as the method counts though i via the loop W30b. Also, though not shown, optionally step W27 could check whether there exists an observed value of the current variable i for the current time t, and if so branch to W30a where that variable is set to its observed value. In other words, more generally step W27 could represent determining whether the current variable i at the present time t has a pre-specified value determined by some other means than reconstruction by the model 104″ (whether because it is intervened-on or observed) and step W30a may represent setting any such variable to its specified value. In some cases the method could begin with a full set of observed and/or treated variables at the initial time t=1, such that at the initial time step all the variables of the feature vector are set to specified values.

[0208] If however it is determined at step W27 that the value of the present variable x_i,tat index i at the present time t is not specified, then the method branches to step W29 where it extracts the values of any instantaneous parent(s) of x_i,tfrom the sampled graph, and step W35a where it extracts the values of any lagged parents of x_i,taccording to the sampled graph. The method then proceeds to step W50 where it generates a noiseless reconstructed value x′_i,tof the present variable x_i,tfor the present time t, using means as discussed previously in the description of the model 104″. Optionally steps W45 and W55 are also performed to generate the history dependent noise term e a as also discussed previously. In this case then at step W60 the noise is added to (or otherwise combed with) the reconstructed variable x′_i,tto generate the noisy simulated sample X_i,tcorresponding to the present variable x_i,t.

[0209] The method then proceeds to step W65 where it determines whether the counting index i has reached the value D of the index of the last variable in the feature vector (preferably sorted in topological order). If not the method branches to step W30b where it increments i by 1 and then loops back to step W25 and repeats until it has reconstructed the entire vector (or at least as much of it as is to be reconstructed). The steps W30a and W35a and the loop via W30b may be considered to correspond to the steps W30 and W35 in FIG. 16, in that steps W30a sets pre-specified (treated or observed) variables to their specified values and step W35 extracts the historical values, while the loop via W30b ensures that a reconstructed value x′_<i,tis generated for any unspecified variable that comes before the present variable x_i,tin the counting order (e.g. topological order). This is relevant since these variables x′_<i,tcould have an effect on the present variable. Counting through all variables i=1 . . . D will mean that, over the entire count (i.e. all cycles of the subloop W27 to W30b), the past and present values of all the variables in the feature vector will be extracted as illustrated at a higher level of abstraction in steps W30 and W35 in FIG. 16.

[0210] If the last index D in the feature vector has been reached for the present time t, then at step W the method checks whether the present time t has reached the target time t′ yet. If not, the method branches to step W75 where t is incremented by 1, and then loops back to step W25 where it repeats with the new value oft. When the method does reach the target time t′, then the method proceeds to step W80 where it determines whether any more graphs are to be sampled. If so, the method loops back to step W20 where it repeats with a newly sampled temporal graph. Once the method has been repeated with the desired number of rounds of graph sampling, then it proceeds to step W90 where the average treatment effect on the target variable x_Ais determined, by averaging over the multiple sampled graphs as discussed previously.

Example Use Cases

[0211] The variables of the feature vector x_i=[x₁. . . x_D] may represent properties of a tangible real-world entity being modelled, or an environment in which the entity resides. The entity may comprise a living being; or a physical object such as an electronic, electrical or mechanical device or system. Alternatively or additionally the object may comprise a piece of software running or to be run on a computer system.

[0212] In one example use case, the modelled real-world entity may comprise a living being, e.g. a human or animal. The disclosed model 104, 104′ or 104″ may be used to determine whether to action a treatment to the being in order to affect a condition of the being (e.g. to cure or alleviate a disease by which the being is afflicted). In this case the target variable x_irepresents a measure of the condition, e.g. a symptom experienced by the living being as a result of the condition, and one or more intervened-on variables x_jrepresent one or more properties of the being or its environment which are susceptible to being controlled as a treatment (e.g. dosage of a drug, or a change in habit or environment, etc.). E.g. this may be used in a medical setting to treat a patient, where the potential treatment in question could be, for instance, a drug, surgery, or a lifestyle change. In another scenario the living being may comprise a plant or crop of plants, such as an agricultural crop, and the potential treatment could be the timing of sowing seeds, when to water or cover the plant/crop, or whether or when to apply a chemical such as a fertilizer or pesticide and/or in what quantity, etc. Whatever form the living being takes, a decision as to whether to perform the proposed treatment on the intervened-on variable(s) as a means to treat the target condition may be made in dependence on whether a confounder is estimated to exist between the proposed treatment and the target condition, and/or an estimate strength of the confounder. And/or, the decision may be made in dependence on the estimated treatment effect, e.g. ATE.

[0213] In another example use case, the real-world entity may comprise a mechanical, electrical or electronic system or device; and the disclosed model 104, 104′ or 104″ may be used to determine whether to action a treatment to affect a state of the system or device, such as to repair, maintain or debug the system or device. In this case the target variable x_imay represent a measure of the state of the system or device, e.g. sensor data measuring wear, operating temperature, operating voltage, output or throughput, etc. The one or more intervened-on variables x_jmay represent one or more properties of the device or its environment susceptible to being treated, e.g. vibration, humidity, ambient temperature, applied current or voltage level, etc. E.g. this could be used in an industrial setting to service industrial equipment such as factory equipment. A decision as to whether to perform the proposed treatment on the intervened-on variable(s) as a means to treat the target state may be made in dependence on whether a confounder is estimated to exist between the proposed treatment and the target state, and/or an estimate strength of the confounder. And/or, the decision may be made in dependence on the estimated treatment effect, e.g. ATE.

[0214] In another example use case, the real-world entity being modelled may comprise software that is run, or to be run, on one or more processors in one or more computer devices at one or more locations; and the disclosed model 104, 104′ or 104″ may be used to determine whether to action a treatment to try to optimize the running of the software. In this case the target variable x_imay comprise a measure of a current state of the software, e.g. memory or processing resource usage, or latency, etc. The one or more intervened-on variables x_jmay represent any property capable of affecting the running of the software, e.g. input data, or rebalancing or the load across a different combination of processors or devices. For example this could be used in a data centre or the cloud for load balancing of software run across different server devices. A decision as to whether to perform the proposed treatment on the intervened-on variable(s) as a means to optimize the running of the software may be made in dependence on whether a confounder is estimated to exist between the proposed treatment and the target state, and/or an estimate strength of the confounder. And/or, the decision may be made in dependence on the estimated treatment effect, e.g. ATE.

[0215] In another example use case the real-world entity being modelled may comprise a network, e.g. a mobile cellular network, a private intranet, or part of the internet (e.g. an overlay network such as a VoIP network overlaid on the network). The disclosed model 104, 104′ or 104″ may be used to determine whether to action a treatment to try to optimize the operation of the network. In this case the target variable x_imay represent any state of the network that it may be wished to improve, e.g. a property of network traffic such as end-to-end delay, jitter, packet loss, error rate, etc. The one or more intervened-on variables x_jmay represent and property capable of affecting the target variable, e.g. balancing of traffic across the network, routing or timing of traffic, encoding scheme used, etc. A decision as to whether to perform the proposed treatment on the intervened-on variable(s) as a means to optimize the network performance may be made in dependence on whether a confounder is estimated to exist between the proposed treatment and the target state, and/or an estimate strength of the confounder. And/or, the decision may be made in dependence on the estimated treatment effect, e.g. ATE. The decision(s) may be made adaptively (i.e. dynamically in response to changing conditions), or as part of network planning.

[0216] In another example use case, the real-world entity may comprise an autonomous or semi-autonomous self-locomotive device such as a self-driving vehicle or robot. The disclosed model 104, 104′ or 104″ may be used to determine whether to action a treatment in the form of a control signal to control the motion of the device. In this case the target variable x_imay represent sensor data providing information on the device's environment, e.g. image data or other sensor data (such as distance or motion sensor data) providing information on the environment of the device (such as presence of obstacles, location of another object to interact with). The one or more intervened-on variables x_jmay comprise a control signal that can be applied to control the device, e.g. to steer the vehicle or apply brakes or lights, or to move a robot arm in a certain way. A decision as to whether to perform a certain control operation (a type of “treatment”) represented by the intervened-on variable, as a means to achieve a certain effect on the environment (e.g. avoid an obstacle or interact with another object), may be made in dependence on whether a confounder is estimated to exist between the proposed variable to be controlled and the target outcome, and/or an estimate strength of the confounder. And/or, the decision may be made in dependence on the estimated treatment effect, e.g. ATE.

Example Implementation: Auto-Regressive Deci

[0217] Further details of some example implementations of the various concepts discussed above are now described, by way of example only.

[0218] Identifying the causal relations between different variables is a question in many fields of scientific enquiry. Randomized control trial is the gold standard for discovering such relationships. However, it may not be always feasible due to practical or ethical reasons. Therefore, with no access to the interventional data, causal discovery may be used to discover the causal relations between different variables using the observational data. Recently, there has been a increasing trend towards causal discovery for time-series data. This is a relevant task in many fields of research including engineering, biology and medicine.

[0219] In a temporal system, one key underlying assumption that greatly simplifies the analysis is that the arrow cannot go against the flow of time. Causal discovery methods for temporal data may be based on the framework of Granger causality, stating that a variable in the past Granger causes another in the current time if the lagged one contains some unique information about the current one. These methods can be roughly categorized into (1) vector autoregressive based methods and (2) deep learning based approaches. However, one hidden assumption of Granger causality is that the time sampling interval is frequent enough such that no aggregation effect exists. In practice, this assumption can be easily violated, where the aggregation manifest itself as the instantaneous effect, i.e. the arrow between variables can exist within a time frame.

[0220] There is another line of research based on structure equation models (SEM), which can simultaneously model the lagged and instantaneous effect. The linear vector auto-regressive may be combined with non-Gaussian noise for causal discovery. For example a general structure equation model for time-series data, called TiMINo, provides a general theoretical framework for structure identifiability. The above methods assume that the noises are mutually independent and stationary in a way such that their distributions have independent (and learnable) parameters. However, this may not hold in many scenarios. For example, under education context, the observational noise of whether a student correctly answers a question should depend on his/her past learning history. If the student correctly answered similar past questions in a consistent way, the observational noise should be small. On the other hand, if the answer history is similar to a random guess, the observational noise should be large. This history-dependent noise distribution is ubiquitous in real life and cannot modelled by the aforementioned SEM.

[0221] To address the aforementioned issue, a particular embodiment of the present disclosure provides a novel SEM for time-series data based on the framework known as deep end-to-end causal inference (DECI) (Geffner et al, 2022). This may be named herein auto-regressive DECI (AR-DECI), which can model both non-linear lagged and instantaneous effect with history-dependent noise distribution. To implement this, the following contributions are provided.

- [0222] A new formulation of temporal SEM with history-dependent noise, called AR-DECI, which combines SEM with vector auto-regressive to model both non-linear lagged and instantaneous effects. AR-DECI also adopts a Bayesian view of graph learning by using variational inference to approximate the graph posterior, which provides uncertainties over graphs under limited data. Also, we show one can compute interested treatment effect estimation by leveraging the fitted AR-DECI, e.g. conditional averaged treatment effect (CATE) with time-invariant interventions.
- [0223] Theoretically, we show that the proposed AR-DECI is structure identifiable under assumptions. To achieve this, we provide a general framework for showing structure identifiability with history-dependent noise, where AR-DECI is a special case. Furthermore, we show AR-DECI unifies several aforementioned approaches based on SEM and Granger causality.

[0224] First we will briefly introduce necessary preliminaries required for building AR-DECI model. In particular, we will focus on the basic concepts of structure equation model, DECI (Geffner et al. 2022), and typical approaches for causal time series: Granger causality and vector auto-regressive models. Note that capital X may now be used to represent both input values and simulated values of the variables.

[0225] Structure Equation Models (SEM): Consider a multivariate time-series X_t=(Xⁱ_t)_i∈Vwhere V is a set of nodes and |V|=D. SEM describes the functional relationships between individual variable Xⁱ_tacross and within the time frame given a graph G. In particular, a general form of SEM is

[0226]

\begin{matrix} X_{t}^{i} = f_{i, t} ({Pa}_{G}^{i} (< t), {Pa}_{G}^{i} (t), ϵ_{t, i}) & (1) \end{matrix}

where Pa_Gⁱ(<t) contains the parent node specified by G in previous time frame (lagged parents) (i.e. parent nodes at time t−1, t−2, . . . ), Pa_Gⁱ(t) is the parent node at current time t (instantaneous parents), ϵ^t,iis the exogenous noise that is mutually independent to other variables in the model and f_i,tdescribes the functional relationships between the parents and child node Xⁱ_t. The above SEM induces a joint probability distribution over the entire time series {X_t}_{t=0 . . . T}. Throughout the following, we make two underlying assumptions for SEM: (1) the function f_ifor all i∈V and t is differentiable w.r.t all the arguments and (2) the induced joint distribution is absolutely continuous w.r.t the Lebesgue measure. Although the above form is very general, it is not very practical to be directly used for causal discovery due to the non-identifiability issue.

[0227] Deep End-to-end Causal Inference (DECI): Recently, Geffner et al. (2022) disclosed an end-to-end framework, called DECI, for causal inference with static data. DECI is based on a specific type of SEM, name additive noise model (ANM), which is structure identifiable under mild conditions. Specifically, the SEM of DECI is defined as:

[0228]

\begin{matrix} X_{i} = f_{i} [{Pa}_{G} (X_{i})] + ϵ_{i} & (2) \end{matrix}

where we drop its dependence of time t compared to Eq. (1) due to its static nature, and ϵ_iis the additive independence noise. DECI also adopts a Bayesian view for graph learning by applying variational inference to approximate the graph posterior p(G|D) where D is the training dataset. By leveraging the trained DECI, one can not only perform causal discovery but also inference tasks such as estimating (conditional) average treatment effect, which provides us an end-to-end pipeline from observation data to the interested causal quantities.

[0229] Granger Causality: Granger causality has been extensively used as the underlying principle for identifying causal relationships in time series data. It is based on the idea that Xⁱdoes not Granger causes X^j_tif the history of Xⁱdoes not help the prediction of X^j_tgiven the past of all other time series X_kfor k≠i. Formally, we can express Granger causality under the SEM framework:

[0230] Definition 1 (Granger Causality): Given a multivariate stationary time series {X_t}_{t=0 . . . T}and a SEM fi defined as:

[0231]

\begin{matrix} X_{t}^{j} = f_{j, t}, [{Pa}_{G}^{i} (< t), ϵ_{t, j}] & (3) \end{matrix}

where Paⁱ_G(<t) is the lagged parents of node X^j_tand ϵ_t,jis the independent noise. Under this setup, time-series i Granger causes j if ∃l ∈[1, t] such that X^j_t−l∈Paⁱ_G(<t) and f_j,tdepends on Xⁱ_t−l.

[0232] It has been shown that Granger causality is equivalent to causal relations for directed acyclic graph (DAG) if there are no latent confounders and instantaneous effect. One shortcoming of Granger causality is its incapability of handling the instantaneous effect, which can happen when the time sampling interval is not frequent enough.

[0233] Vector Auto-regressive Model: To overcome the aforementioned issue of Granger causality, another line of research focuses on directly fitting the identifiable SEM to the observational data with instantaneous parents. One typical approach is called vector auto-regressive model with the SEM defined as:

[0234]

\begin{matrix} X_{t}^{i} = β^{i} + \sum_{τ = 0}^{K} \sum_{j = 1}^{D} B_{ji}^{τ} X_{t - τ}^{j} + ϵ_{t, i} & (4) \end{matrix}

where β^jis the offset, τ is the lag, B^τ∈ custom character

^D×Dis the weighted adjacency matrix specifying the connections in the DAG (i.e. if B^τ_ji=0 means no connection between X^j_t−τto Xⁱ_t) and ϵ_t,iis the independent noise. Under certain assumptions, the above SEM has been shown to be structure identifiable, which validates its usage in causal discovery/inference tasks.

[0235] However, in real life, linear relationship is too restricted to capture the complex data generating mechanism and noise distribution may also be history-depended as described earlier. Therefore, a crucial research question is how to design an SEM for such effects with structure identifiability guarantees.

AR-DECI: Auto-Regressive Deep End-To-End Causal Inference

[0236] In this subsection, we will first focus on formulating the SEM of AR-DECI, followed by disclosing a variational objective with continuous DAG relaxation for training. In the end, we will discuss how to estimate CATE with time-invariant interventions.

[0237] Model formulation: For a multivariate stationary time-series {X_t}_{t=0 . . . T}with a set of nodes V and |V|=D, we assume it follows a temporal graph adjacency matrix G_0:Kwhere K is the maximum lag K. We define G_τ,ij=1 means there is a directed connection Xⁱ_t−τ→X^j_tand 0 means no connection. In the following, we interchange the usage of the notation G and G_0:Kfor brevity. We provide the following SEM for AR-DECI:

[0238]

\begin{matrix} X_{t}^{i} = f_{i} ({Pa}_{G}^{i} (< t), {Pa}_{G}^{i} (t)) + g_{i} ({Pa}_{G}^{i} (< t), ϵ_{t, i}) & (5) \end{matrix}

where f_i, g_iare general differentiable non-linear functions, Paⁱ_G(<t) is the lagged parents of node Xⁱ_tspecified by adjacency G_1:K, Paⁱ_G(t) is the corresponding instantaneous parents specified by G₀, and ϵ_t,iis the independent noise. We also assume the above model satisfies causal Markov and minimality conditions. The above SEM has a similar additive structure as ANM, which has been leveraged in previous work to develop causal time-series models. A distinction of AR-DECI is that the noise ϵ_t,iis transformed by a non-linear history-dependent function g_i, meaning that the transformed noise distribution is history dependent and may not be independent to the other variables.

[0239] For the detailed design of the non-linear function fi, it should respect the relationships specified by graph G_0:K. Namely, if X^j_t−τ∉Paⁱ_G(<t)∪Paⁱ_G(t), then ∂f_i/∂X^j_t−τ=0. Building on the design of DECI, we disclose:

[0240]

\begin{matrix} f_{i} (P a_{G}^{i} (t), {Pa}_{G}^{i} (t)) = ζ_{i} (\sum_{τ = 0}^{K} \sum_{j}^{D} G_{τ, j i} ℓ_{τ j} (X_{t - τ}^{j})) & (6) \end{matrix}

where ζ_iand custom character

_τi(i=1, . . . ,D and τ=0, . . . ,K) are MLPs. To reduce the number of neural networks, we adopts a weight sharing scheme across nodes and lags: ζ_i(⋅)=ζ(⋅,u_0,i) and custom character

_τj(⋅)=

(⋅,u_τ,j), where u_τ,iis the trainable embedding for node i at time t−τ.

[0241] For the design of the noise transformation gi, it allows very flexible design choices. For the ease of evaluation and training, we limit our choice to be a conditional normalizing flow. Specifically, we choose conditional spline flow for g_iwith Gaussian noise ϵ_t,ifor all t and i. Due to the invertibility of g_i, we have:

[0242]

\begin{matrix} p_{g_{i}} (g_{i} (ϵ_{t, i}) | P a_{G}^{i} (< t)) = P_{ϵ} (ϵ_{t, i}) ❘ \frac{δ g_{i}^{- 1}}{{δϵ}_{t, i}} ❘ & (7) \end{matrix}

where we omit the dependency of g_iwith Paⁱ_G(<t) for clarity.

Variational Inference for AR-DECI

[0243] AR-DECI adopts a Bayesian view of causal discovery, which aims to learn a graph posterior distributions instead of inferring a single graph. For N observed multivariate time series X⁽¹⁾_0:T, . . . ,X^(N)_0:T, the joint likelihood of AR-DECI is:

[0244]

\begin{matrix} p (X_{0 : T}^{(1)}, \dots, X_{0 : T}^{N}, G) = p (G) \overset{N}{\prod_{n = 1}} p_{θ} (X_{0 : T}^{(n)} | G) & (8) \end{matrix}

where θ is the model parameter. Once fitted, the posterior p(G|X⁽¹⁾_0:T, . . . ,X^(N)_0:T) incorporates the belief of the underlying causal relationships.

[0245] Graph Prior: To design a proper graph prior, we aim at three criteria: (1) constrain the graph to be a DAG; (2) favour sparse graph; (3) support prior knowledge. We choose the prior as:

[0246]

\begin{matrix} p (G) \propto \exp (- λ_{s} { G_{0 : K} }_{F}^{2} - ρ h^{2} (G_{0}) - α h (G_{0}) - λ_{p} { G_{0 : K} - G_{0 : K}^{p} }_{F}^{2} & (9) \end{matrix}

where h(G)=tr(e^G⊙G)−D is the DAG penalty proposed in and is 0 if and only if G is a DAG; ⊙ is the Hadamard product; G^Pis the prior knowledge of the graph; λ_s, λ_pspecifies the strength of the graph sparseness and prior knowledge, respectively; α, ρ characterize the strength of the DAG penalty. Since the connections from the history node to the current one can only follow the direction of the time flow, only the instantaneous connections can violate the DAG constraint. Thus, DAG penalty is only applied to G₀.

[0247] Variational Objective: Unfortunately from the Bayes' rule, the exact graph posterior p(G|X⁽¹⁾_0:T, . . . ,X^(N)_0:T) is intractable due to the large combinatorial space of all DAGs. To overcome this challenge, we adopt variational inference, which uses variational distribution q_φ(G) to approximate the true posterior. We choose independent Bernoulli distribution for each directed edge as q_φ(G). We derive the evidence lower bound (ELBO) as:

[0248]

\begin{matrix} \log p_{θ} (X_{0 : T}^{(1)}, \dots, X_{0 : T}^{(N)}) \geq \underset{\underset{ELBO (θ, ϕ)}{︸}}{𝔼_{q_{ϕ} (G)} [\sum_{n = 1}^{N} \log p_{θ} (X_{0 : T}^{(n)} | G) + \log p_{θ} (G)] + H (q_{ϕ} (G))} & (10) \end{matrix}

where H(q_φ(G)) is the entropy of the variational distribution. From the causal Markov assumption and auto-regressive nature, we can derive:

[0249]

\begin{matrix} \log p_{θ} (X_{0 : T}^{(n)} | G) = \overset{T}{\sum_{t = 0}} \overset{D}{\sum_{i = 1}} \log p_{θ} (X_{t}^{i, (n)} | P a_{G}^{i} (< t), {Pa}_{g}^{i} (t)) & (11) \end{matrix}

and

\begin{matrix} \log p_{θ} (X_{t}^{i, (n)} | P a_{G}^{i} (< t), {Pa}_{G}^{i} (t)) = p_{g_{i}} (z_{t}^{i, (n)} | P a_{G}^{i} (< t)) & (12) \end{matrix}

where z^i,(n)_t=X^i,(n)_t−f_i[Paⁱ_G(<t), Paⁱ_G(t)] and p_giis defined in Eq. (7). The parameters θ, φ are optimized by maximizing the ELBO, where a Gumbel-softmax trick is used to estimate the gradients w.r.t φ.

[0250] Treatment effect estimation: We now show how to leverage the fitted AR-DECI for estimating the CATE. For simplicity, we only consider a special case of CATE defined as:

[0251]

\begin{matrix} CATE (a, b) = 𝔼_{q_{ϕ}} [𝔼_{p (X_{t + τ}^{Y} | X_{< t}, do (X_{t}^{I} = a), G)} [X_{t + τ}^{Y}] - 𝔼_{p (X_{t + τ}^{Y} | X_{< t}, do (X_{t}^{I} = b), G)} [X_{t + τ}^{Y}]] & (13) \end{matrix}

[0252] We assume the conditioning variable can only be X_<t(i.e. the entire history before t), and the intervention and target variable can only be either at current time t or sometime in the future t+τ. This formulation is for simplicity, and AR-DECI can be easily extended to more general cases using the CATE tricks in DECI (Geffner et al). Once fitted, the idea is to draw target samples X^Y_t−τ from the interventional distribution p(X^Y_t+τ|X_<t, do(X^l_t),G) for each graph sample G~q_φ(G). Then, unbiased Monte Carlo estimation can be used to compute CATE. For sampling from the interventional distribution, we can use the “mutilated” graph G_do(X^l_t) to replace G, where all incoming edges to X^l_tare removed. The intervention samples can be obtained by simulating the AR-DECI with history X_<t, X^l_t=a or b and G_do(X^l_t).

Example Implementation: Confounded Deci

[0253] Further details of some example implementations of the various concepts discussed above are now described, by way of example only.

[0254] As discussed earlier, in practice it is not always possible to directly perform interventional studies (i.e., randomised AB testings) to estimate treatment effects, and it may be required to use observational studies. That is, the causal graph and the associated causal effects are to be inferred based on observational data, in embodiments solely based on observational data. Therefore, this would appear to require an assumption that all the data provided by users already contains all the information that is needed (i.e. there should not be any unmeasured confounders). However, this assumption is often unrealistic, as the modelled scenario is also impacted by certain variables that cannot be directly measured.

[0255] Therefore, it would be desirable to provide an effective and theoretically principled strategy for handling latent confoundings when performing joint causal discovery & inference. However, the existence of unmeasured confounding poses questions for causal discovery since there might exist multiple contradicting causal structures that are compatible with observations. Traditional methods for causal discovery mainly focus on estimating the causal relationships between observed variables and does not recover the causal relationships that involve unmeasured confounders. There also exists several related methods for causal discovery with latent confounders, but each comes with certain limitations on assumptions. Following the previous work by Geffner at al of developing deep end-to-end causal inference (DECI), this framework can be extended in order to handle latent confounders. The outcomes of this is to implement latent variable variants/baselines of DECI that explicitly take latent confounders into account. These variants/baselines should preferably: 1), be able to perform joint graph learning and functional learning; 2), have computational costs comparable to DECI; and 3), subsume DECI, in the sense that it should perform similarly to DECI when there is no latent confounders.

[0256] To recap the causal discovery without latent confounders using DECI, recall that in DECI, we perform causal discovery under no latent confounder assumptions, but modelling the causal graph G on all observed vertexes X jointly with the observations x¹, . . . , x^Nas:

[0257]

\begin{matrix} p_{θ} (x^{1}, \dots, x^{N}, G) = p (G) \prod_{n} p_{θ} (x^{n} | G) & (1) \end{matrix}

Here, p(G) is a prior over DAG, implemented by:

[0258]

\begin{matrix} p (G) \propto \exp (- λ_{s} { G }_{F}^{2} - ρ {h (G)}^{2} - α h (G)) & (2) \end{matrix}

where G is the adjacency matrix representation, and

[0259]

\begin{matrix} h (G) = tr (e^{G ⊙ G}) - D & (3) \end{matrix}

is the DAG penalty.

[0260] We aim to fit θ, the parameters of our non-linear ANM, using observational data by maximizing (an lower bound of) log p_θ(x¹, . . . , x^N). Given θ, the question of causal discovery can be simply answered by the posterior, p_θ(G|x¹, . . . , x^N). In the original DECI, we maximize the ELBO of log p_θ(x¹, . . . , x^N) and use mean-field approximation qφ(G) to perform approximate inference.

[0261] Causal discovery under ADMGs: in the present disclosure, we will extend DECI's framework to consider unmeasured confounders. Unfortunately, if some variables in a DAG is unobserved, the resulting set of (conditional) independences no longer corresponds to a DAG (on the observed variables).

[0262] To deal with unobserved variables, a common extension is the so called acyclic directed mixed graphs (ADMGs). In a ADMGs, a bidirected edge x↔y indicates the existence of a unmeasured founder u_xy, such that x←u_xy→y. Note that this would be different than a undirected edge x-y used in (CP)DAG, which indicates either x←y or x→y. Finally, the term acyclic in ADMG means that there is no cycles that contains only directed edges.

[0263] Therefore, causal discovery under latent confounders can be formulated as the same equation as the original DECI,

[0264]

\begin{matrix} p_{θ} (x^{1}, \dots, x^{N}, G) = p (G) \prod_{n} p_{θ} (x^{n} | G) & (4) \end{matrix}

except that the prior p(G) will be based on the constraint that G is a ADMG (instead of a DAG). We will discuss how to parameterize the ADMG constraint and how to parameterize p_θ(x¹, . . . , x^N, G) later.

[0265] Note that in principle, an ADMG causal discovery algorithm is able to express search results using (a subset of) the following structures:

- [0266] x→y: x is the cause of y;
- [0267] x↔y: indicates the existence of a unmeasured founder u_xy, such that x←u_xy→y.
- [0268] x-y: either x←y or x→y.
- [0269] x⇒y: either x↔y, or x→y.

[0270] In the following, we mainly consider representing our posteriors using the first two, where they have better granularity than the last two.

D-DECI (Deconfounded DECI)

[0271] Magnification-based representation: We perform causal discovery on the highest granularity, i.e. explicitly model the latent confounders (denoted by U), and learn a DAG G′ on X∪U. In other words, G′ (on X∪U) is a magnification of G (on X); and the marginalized graph G over observed variables X will then be a ADMG. To relate G′ to the previous notation of G₁and G₂, G₁is the adjacency matrix on X; and G₂is the adjacency matrix on U. G′ is basically a big matrix on both X and U, obtained by concatenating and unpacking G₁and G₂. In other words, G′=M(G₁, G₂), where M(·) is a pre-known mathematical function. G′ may also be called magnified matrix. In short, it is obtained by concatenating and aggregating G₁and G₂. For example, for a graph x₁→x₂↔x₃, the corresponding matrices will be:

[0272]

G_{1} = (\begin{matrix} 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{matrix})

G_{2} = (\begin{matrix} 0 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{matrix})

G^{'} = (\begin{matrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 1 & 1 & 0 \end{matrix})

where the 4th row of G′ corresponds to the 4th variable (additional latent confounder) implied in the bidirected edge x₂↔x₃. That is, the 4th variable should point to both x₂and x₃, hence those two 1s in the second and third cell of the 4th row.

[0273] Under this idea, we model the causal graph G′ jointly with the observations x¹, . . . , x^N, as well as latent confounders u¹, . . . , u^N:

[0274]

\begin{matrix} p_{θ} (x^{1}, \dots, x^{N}, u^{1}, \dots, u^{N}, G^{'}) = p (G^{'}) \prod_{n} p_{θ} (x^{n}, u^{n} | G^{'}) & (5) \end{matrix}

[0275] We will define constraints on G′ and the parameterization of p_θ(xⁿ, uⁿ|G') later. We call Eq. (5) magnification-based parameterization, since the original ADMGs formulation does not explicitly assume any functional forms. To optimize θ and perform approximate inference on G′, we define:

- [0276] 1) A variational distribution q_φ(G) to approximate the intractable posterior p_θ(x¹, . . . x^N). Similar to the original DECI, this can be parametrized as the product of independent Bernoulli distributions for each potential directed edge in G′. We parametrize edge existence and edge orientation separately, using the ENCO parametrization.
- [0277] 2) An inference network q_λ(u|x) to approximate the intractable posterior p_θ, G′(u|x), e.g., a Gaussian inference network used in VAEs. The inference network q_λ(u|x) has also been referred to as H earlier.

[0278] We use them to build the ELBO, given by:

[0279]

\begin{matrix} ELBO (θ, \emptyset, λ) = 𝔼_{q_{ϕ} (G) q_{λ} (u | x)} [\log \frac{p (G) \prod_{n} p_{θ} (x^{n}, u^{n} | G)}{q_{ϕ} (G) \prod_{n} q_{λ} (u^{n} | x^{n})}] \leq \log p_{θ} (x^{1}, \dots, x^{N}) & (6) \end{matrix}

[0280] By committing to optimize Eq. (6), we implicitly make the following assumptions:

- [0281] We already know the exact number of unmeasured variables. That is, the latent vertexes U is given and fixed.
- [0282] For all unmeasured variables in U, they only possess out-going edges. That is, all variables U serve as confounders. This in practice and be implemented by model.set graph constraint( ) in DECI code base.

[0283] Prior over ADMGs: The ADMG constraint on the graph G′ can be imposed in two ways:

- [0284] Directly apply the DAG constraint on G′. That is,

[0285]

\begin{matrix} h (G^{'}) = tr (e^{G' ⊙ G}) - D = 0 & (7) \end{matrix}

[0286] This ensures that once G′ can be converted to an ADMG (by marginalizing U and only keeping X).

- [0287] OR
- [0288] First marginalize G′ to the graph G on X, and then apply ADMG constraints on G. This can be done via the following three ADMG constraints:
  - [0289] Ancestral ADMG constraint:

[0290]

\begin{matrix} h (G) = t r (e^{D}) - D + \sum (e^{G_{directed}} ⊙ G_{b i directed}) = 0 & (8) \end{matrix}

- [0291] where G_directed,i,j=1 if and only if x_i←x_j; and G_{bidirected,i,j}=1=G if and only if x_i↔x_j.
- [0292] Arid ADMG constraint:

[0293]

\begin{matrix} h (G) = tr (e^{D}) - D + G R EENERY (G_{directed}, G_{b i directed}) = 0 & (9) \end{matrix}

- [0294] Bow-free constraint:

[0295]

\begin{matrix} h (G) = t r (e^{D}) - D + \sum (G_{directed} ⊙ G_{b i directed}) = 0 & (10) \end{matrix}

By using any of those constraints, p(G′) can be implemented by:

[0296]

\begin{matrix} p (G^{'}) \propto \exp (- λ_{s} { G^{'} }_{F}^{2} - ρ {h (G^{'})}^{2} - α h (G^{'})) & (11) \end{matrix}

[0297] Parameterization of p_θ(xⁿ, uⁿ|G′): Similar to DECI, we can parameterize p_θ(xⁿ, uⁿ|G′) using auto-regressive ANM. That is, z=g_G′(x, u; θ)=[x, u]−f_G′Q([x, u]; θ). Where z are independent noise variables following the distribution Π_ip_zi. Then we can write the observational likelihood as:

[0298]

\begin{matrix} p_{θ} (x^{n}, u^{n} | G^{'}) = p_{z} (g_{G^{'}} ([x^{n}, u^{n}]; θ)) = \overset{D}{\prod_{i = 1}} p_{z_{i}} ({g_{G^{'}} ([x^{n}, u^{n}]; θ)}_{i}) & (12) \end{matrix}

where we omitted the Jacobian-determinant term because it is always equal to one for DAGs G′.

[0299] Structural identifiability under latent confounders: Preferably, we would like to find a set of suitable assumptions such that D-DECI should satisfy the structural identifiability definition, as follows.

[0300] Definition 1 (D-DECI Structural Identifiability). For a distribution p_θ(x;G′):=∫_up_θ(x, u|G′)du, the graph G′ is said to be structural identifiable from p_θ(x;G′) if there exists no other distribution p_θ'(x;G″) such that G′≠G″ and p_θ(x;G′)=p_θ′(x;G″).

[0301] Generally, such structural identifiability does not hold. However, under the constraint of bow-freeness, we will be prove identifiability, which will be discussed in the next section.

D-DECI Identifiability

[0302] Notation and assumptions: Let x∈ custom character

⁰denote the observed data, u∈ custom character

^Ddenote the unobserved data, v=[x; u]^T, and p(x, u; custom character

⁰) the ground truth data generating distribution, where custom character

⁰is a binary adjacency matrix representing the true causal DAG. DDECI uses the additive noise model (ANM) defining the structural assignment x_j′=f_i(pa(j); θ)+n_jwhere pa(j) denotes the parents of x_j. The function f corresponds to the decoders g^ddescribed earlier. We'll also use the notation pa_x(j) and pa_u(j) to denote the parents of x_jin the observed and unobserved set, respectively. n_jare mutually independent noise variables. The noise term n has also been referred to as z earlier.

[0303] Assumption 1 (Form of f_j): We assume that the functions f_jare non-linear and decompose as:

[0304]

\begin{matrix} f_{j} (p a_{x} (j), p a_{u} (j); θ) = f_{j, x} (p a_{x} (j); θ) + f_{j, u} (p a_{u} (j); θ) & (13) \end{matrix}

[0305] Assumption 2 (Unobserved variables are latent confounders): All unobserved variables are latent confounders of observed variables, i.e. have the causal structure x_j←u_k→x_i. Justification of this is given by the proposition that any latent variable can be reparameterised by absorbing exogenous noise to make it a latent confounder. A distribution p(x) satisfies minimality w.r.t. custom character

if it is Markov w.r.t. custom character

, but no to any proper subgraph of custom character

. Since we cannot access the true data generating distribution p(x, u; custom character

⁰), only the marginal distribution over observed variables p(x; custom character

⁰), we require a slightly different definition of minimality. We define this below.

[0306] Definition 1 (Minimality): p(x; custom character

) is minimal if it does not satisfy the local Markov condition with respect to the ADMG of any subgraph of custom character

. This form of minimality implies the bow-free constraint. To see this, note that the causal structure x_i←u_k→x_jtogether with x_i←x_j, and x_i←u_k→x_jalone imply the same conditional independencies. Since the latter is a subgraph of the former, only it satisfies the minimality condition.

[0307] Assumption 3 (Minimality): For a distribution p(x; custom character

) generated by D-DECI, we assume the minimality condition holds.

[0308] Comments on assumption of latent variables: Treatment effect is invariant to parameterisations if intervening on observed variables only.

[0309] Proof of structural identifiability: Here we prove structural identifiability of custom character

given the assumptions of the previous section.

[0310] Lemma 1: Let M denote a set of variables not containing v_i, and let s(v_i) denote a linear or nonlinear function of v_i. Then, the residual of s(v_i) regressed on M cannot be independent of n_i:

[0311]

\begin{matrix} \forall M \subseteq V \ {v_{i}}, g_{i}, [𝒮 (v_{i}) - g_{i} (M) n_{i}] & (14) \end{matrix}

[0312] Proof: Assume that [s(v_i)−g_i(M) custom character

n_i] holds, then M must contain at least one descendent of v_ias it must have dependence on the noise n_ito cancel effect of n_iin s(v_i). We can express g_i(M) as u_i(∪_j:v_l_∈Mn_j∪n_i). Since g_ioperates on variables defined by non-linear transformations of the exogenous noise terms, we cannot express u_ias a_i(∪_j:v_l_∈Mn_j)+b_i(n_i). M contains a descendent of v_i, so ∪_j:v_l_∈Mn_j∪n_iincludes at least one noise term n_kthat satisfies v_i custom character

n_k(i.e. is not in v_i). Thus, terms containing ni cannot be fully removed from s(v_i)−g_i(M) and so [s(v_i)−g_i(M) custom character

n_i] does not hold.

[0313] Lemma 2: If an only if is satisfied, there is a latent confounder, and no direct cause, between x_iand x_j.

[0314]

\begin{matrix} \forall g_{i}, g_{j}, M \subseteq (X \ {x_{i}}), N \subseteq (X \ {x_{j}}), [(x_{i} - g_{i} (M) (x_{j} - g_{j} (N)] & (15) \end{matrix}

[0315] Proof: Define g_iand g_jas g_i(M)=f_i,x(pa_x(i))+g′_i(M) and g_j(N)=f_j,x(pa_x(j))+g′_iN) respectively. Then, (15) becomes equivalent to:

[0316]

\begin{matrix} [(f_{i, u} ({pa}_{u} (i)) + n_{i} - g_{i}^{'} (M) (f_{j, u} ({pa}_{u} (j)) + n_{j} - g_{j}^{'} (N)] & (16) \end{matrix}

[0317] This is equivalent to:

[0318]

\begin{matrix} (f_{i, u} ({pa}_{u} (i)) + n_{i}) (f_{j, u} ({pa}_{u} (j)) + n_{j}) & (17) \end{matrix}

[0319] And since n_i custom character

n_j, we have:

[0320]

\begin{matrix} (f_{i, u} ({pa}_{u} (i)) n_{j}) \lor (n_{i} f_{j, u} ({pa}_{u} (j))) \lor f_{i, u} ({pa}_{u} (i)) f_{j, u} ({pa}_{u} (j))) & (18) \end{matrix}

[0321] The first implies the existence of an unobserved mediator between x_jand x_i, the second implies the existence of an unobserved mediator between x_iand x_j, and the third implies the existence of an unobserved confounder. Given Assumptions 2 and 3, this indicates the presence of a latent confounder and no direct cause between x_iand x_j.

[0322] Lemma 3. If (19) is satisfied, there is no causal relationship between x_iand x_jand no latent confounder.

[0323]

\begin{matrix} \exists g_{i}, g_{j}, M \subseteq (X \ {x_{i}, x_{j}}), N \subseteq (X \ {x_{i}, x_{j}}),  [(x_{i} - g_{i} (M) (x_{j} - g_{j} (N))] & (19) \end{matrix}

[0324] Lemma 4. If (20) and (21) are satisfied, x_jis a direct cause of x_iand there is no latent confounder.

[0325]

\begin{matrix} \forall g_{i}, g_{j}, M \subseteq (X \ {x_{i}, x_{j}}), N \subseteq (X \ {x_{j}}), [(x_{i} - g_{i} (M) (x_{j} - g_{j} (N)] & (20) \end{matrix}

\begin{matrix} \exists g_{i}, g_{j}, [(x_{i} - g_{i} (M) (x_{j} - g_{j} (N))] & (21) \end{matrix}

[0326] Thus, for any pair of variables we can determine whether there exists a latent confounder, and if not we can determine whether or not there is a direct cause and the orientation.

[0327] MLE recovers ground proof: We can use the exact same proof as in the DECI paper to show that MLE recovers ground truth—we can just replace custom character

with the ADMG corresponding to custom character

projected onto x.

[0328] D-DECI recovers ground truth: In the infinite data limit, we have:

[0329]

\begin{matrix} \lim_{N \to \infty} \frac{1}{N} \sum_{n = 1}^{N} 𝔼_{q_{ϕ} (G) q (u_{n} | x_{n})} [\log p_{θ} (x_{n}, u_{n} | 𝒢)] - \frac{1}{N} \sum_{n = 1}^{N} H (q (u_{n} | x_{n})) - \underset{\underset{\to 0}{︸}}{\frac{1}{N} KL [q (𝒢) ❘ ❘ p (𝒢)]} = \lim_{N \to \infty} \frac{1}{N} \sum_{n = 1}^{N} 𝔼_{q_{ϕ} (G)} [\log p_{θ} (x_{n} | 𝒢)] - \frac{1}{N} \sum_{n = 1}^{N} 𝔼_{q (𝒢)} [KL [q (u_{n} | x_{n}) ❘ ❘ p (u_{n} | x_{n}, 𝒢)]] . & (22) \end{matrix}

[0330] The posterior approximation q(u_n|x_n) should preferably be very accurate (i.e. custom character

[KL[q(u_n|u_x)∥p(u_n|u_x, custom character

)]]≈0) to recover the ground truth causal structure.

Final Remarks

[0331] It will be appreciated that the above embodiments have been described by way of example only.

[0332] More generally, according to a first aspect disclosed herein there is provided (Statement 1): a computer-implemented method comprising: A) selecting a selected variable from among variables of a feature vector; B) sampling a temporal causal graph from a temporal graph distribution, the temporal graph distribution specifying probabilities of directed causal edges between different ones of the variables of the feature vector at a present time step, and from one of the variables of the feature vector at a preceding time step to one of the variables of the feature vector at the present time step; C) from among of the variables of the feature vector, identifying a present parent which is a cause of the selected variable in the present time step according to the temporal causal graph, and identifying a preceding parent which is a cause of the selected variable from the preceding time step according to the temporal causal graph; D) inputting an input value of each of the identified present and preceding parent into a respective encoder, resulting in a respective embedding of each of the present and preceding parents; E) combining the embeddings of the present and preceding parents, resulting in a combined embedding; and F) inputting the combined embedding into a decoder associated with the selected variable, resulting in a reconstructed value of the selected variable.

[0333] In embodiments the method may optionally further comprise features as set out in any of the following Statements.

[0334] Statement 2. The method of Statement 1, wherein: the temporal causal graph distribution sampled in B) specifies probabilities of directed causal edges existing from each of a plurality of variables of the feature vector from one or more preceding time steps, each to a variable of the feature vector at the present time step; and C) comprises, from among of the variables of the feature vector, identifying each present parent that is a cause of the selected variable in the present time step according to the temporal causal graph, and identifying each preceding parent variable which is a cause of the selected viable from a preceding time step according to the temporal causal graph.

[0335] Statement 3. The method of Statement 2, further comprising: generating a history dependent noise term based on embeddings of the preceding parents; and combining the history dependent noise term with the reconstructed value of the selected variable, resulting in a simulated value of the reconstructed variable.

[0336] Statement 4. The method of Statement 3, wherein embeddings of the present parents are not input into the noise model.

[0337] Statement 5. The method of Statement 4, wherein the generating of the history dependent noise term comprises: combining the embeddings of the preceding parents, resulting in a further embedding; inputting the further embedding into a decoder associated with the selected variable, resulting in one or more parameter values; and generating the history dependent noise term from the probabilistic distribution as parameterized by the one or more parameter values.

[0338] Statement 6. The method of any preceding Statement, wherein: B) further comprises sampling a second causal graph from a second graph distribution, the second causal graph modelling presence of possible confounders, a confounder being an unobserved cause of both of two variables in the feature vector; C) further comprises, from among of the variables of the feature vector, identifying a parent variable which is a cause of the selected variable according to the first causal graph, and which together with the selected variable forms a confounded pair having a respective confounder being a cause of both according to the second causal graph; and D) further comprises inputting the input value of the parent variable and an input value of the selected variable into an inference network, resulting in a latent value modelling the respective confounder of the confounded pair, and inputting the latent value into a second encoder, resulting in an embedding of the confounder of the confounded pair; and in E) the combining includes combining the embedding of the present and preceding parents with the embedding of the confounder of the confounded pair, thereby resulting in said combined embedding.

[0339] Statement 7. The method of any of Statements 2 to 5, or Statement 6 when dependent on any of Statements 2 to 5, comprising: i) for a given training data point comprising a given combination of input values of the variables of the feature vector at the present times step and each preceding time step, repeating A)-F) over multiple selections, each selection selecting a different one of the variables of the feature vector as the selected variable thereby resulting in a respective reconstructed value, the multiple selections together thereby resulting in a reconstructed version of the training data point comprising the reconstructed values for the training data point; ii) evaluating a measure of difference between the training data point and the reconstructed version; and iii) training model parameters of the encoders, decoders, and temporal graph distributions, based on the evaluated measure.

[0340] Statement 8. The method of Statement 7, comprising repeating i)-iii) over multiple input data points, each comprising a different combination of input values of the variables of the feature vector.

[0341] Statement 9. The method of Statement 7 or 8, wherein: each selection in i) further comprises a history dependent noise model generating a respective history dependent noise term based on embeddings of the preceding parents; and wherein the training further comprises training model parameters the history dependent noise model based on the evaluated measure.

[0342] Statement 10. The method of Statement 9, wherein in each selection in i), the generating of the history dependent noise term by the history-dependent noise model comprises: combining the embeddings of the preceding parents, resulting in a respective further embedding; inputting the respective further embedding into the decoder associated with the selected variable, resulting in a one or more parameter values; and generating the respective history dependent noise term based on the one or more parameter values.

[0343] Statement 11. The method of Statement 10, comprising repeating i)-iii) over multiple input data points, each comprising a different combination of input values of the variables of the feature vector.

[0344] Statement 12. The method of Statement 2, or any of Statements 3 to 11 when dependent on Statement 2, comprising: I) setting the input value of an intervened-on variable of the feature vector, other than the selected variable, to a specified value; and II) estimating an effect of the intervened-on variable on the selected variable based on the reconstructed value of the selected variable.

[0345] Statement 13. The method of Statement 12, wherein: I) comprises setting the input value of a plurality of intervened-on variables of the feature vector, other than the selected variable, to respective specified values; and II) comprises estimating the effect of the plurality of intervened-on variables based on the reconstructed value of the selected variable.

[0346] Statement 14. The method of Statement 13, wherein: I) comprises repeating A)-F) over multiple rounds, each using the same selected variable and setting the intervened-on variable to the same specified value, but performing the sampling of the temporal causal graph afresh each round, thereby resulting in a respective reconstructed value of the selected variable from each round; and II) comprises determining an average treatment effect of the intervened-on variable on the selected variable averaged over the multiple rounds.

[0347] Statement 15. The method of Statement 14, wherein each graph-sampling event further comprises: generating a history dependent noise term based on the embeddings of the preceding parents; and combining the history dependent noise term with the reconstructed value of the selected variable, resulting in a simulated value of the reconstructed variable; wherein said estimating based on the reconstructed values in II) comprises: estimating the effect of the intervened-on variable on the selected variable based on the simulated values.

[0348] Statement 16. The method of Statement 14 or 15, wherein one or both of: each round sets a plurality of intervened-on variables and/or observed variables of the feature vector to specified values (and in the case of multiple intervened-on variables, II comprises determining the average treatment effect of the plurality of intervened-on variables on the selected variable); and/or each round comprises an interior loop of D)-F) repeated around multiple iterations with the same sampled graph but for incrementing values of the present time step with each iteration, wherein each but the last iteration comprises, in addition to the target variable, reconstructing or simulating values of one or more further variables of the feature vector, and feeding back the simulated or reconstructed values as the specified input values of the next iteration.

[0349] Statement 17. The method of any of Statements 12 to 16, wherein the intervened-on variable models a treatment on a real-world entity or an environment thereof and the target variable models an effect of the treatment applied to the real-world entity, and the method further comprises actioning the treatment on the real-world entity in dependence on the estimated treatment effect.

[0350] Statement 18. The method of Statement 17, wherein one of: the real-world entity comprises a living being, and the treatment comprises a medical treatment to the living being or an environment thereof; or the real-world entity comprises a mechanical, electrical or electronic device or system, or an environment thereof; and the treatment comprises an act of maintaining, debugging, upgrading or controlling the device or system, or controlling the environment thereof; or the real-world entity comprises a network or software, and the treatment comprises an act of controlling the network or software.

[0351] Statement 19. A system comprising: processing apparatus comprising one or more processors; and memory comprising one or more memory units, wherein the memory stores: a machine learning model comprising the encoder and decoders of any preceding Statement, and optionally the noise model and/or a inference network; and code arranged to run on the processing apparatus and being configured so as when run to perform the method of any of Statements 1 to 18.

[0352] Statement 20. A computer program embodied on non-transitory computer-readable storage, wherein the computer program comprises instructions configured so as when run on one or more processors to perform the method of any of Statements 1 to 18.

[0353] According to a second aspect disclosed herein there is provided (Statement 1A) a computer-implemented method comprising: a) selecting a selected variable from among variables of a feature vector; b) sampling a first causal graph from a first graph distribution and sampling a second causal graph from a second graph distribution, the first causal graph modelling causation between variables in the feature vector, and the second causal graph modelling presence of possible confounders, a confounder being an unobserved cause of both of two variables in the feature vector; c) from among of the variables of the feature vector, identifying a parent variable which is a cause of the selected variable according to the first causal graph, and which together with the selected variable forms a confounded pair having a respective confounder being a cause of both according to the second causal graph; d) inputting an input value of the parent variable into a first encoder, resulting in a respective embedding of the parent variable; e) inputting at least the input value of the parent variable (and optionally also an input value of the selected variable and/or an input value of one or more others of the variables of the feature vector) into an inference network, resulting in a latent value modelling the respective confounder of the confounded pair, and inputting the latent value into a second encoder, resulting in an embedding of the confounder of the confounded pair; f) combining the embedding of the parent variable with the embedding of the confounder of the confounded pair, resulting in a combined embedding; and g) inputting the combined embedding into a decoder, resulting in a reconstructed value of the selected variable.

[0354] In embodiments there may optionally be provided a method in accordance with any of the following statements.

[0355] Statement 2A. The method of Statement 1A, wherein c) comprises: identifying each of the variables of the feature vector that is a respective parent variable of the selected variable according to the first causal graph, and identifying each respective confounded pair that comprises the selected variable and a respective one of the identified parent variables according to the second causal graph; d) comprises: inputting a respective input value of each of the parent variables into a respective first encoder for the parent, resulting in a respective embedding of each respective parent variable; e) comprises: for each identified confounded pair, inputting the input value of the respective parent variable and the input value of the selected variable into the inference network, thereby resulting in a respective latent value of each of the respective confounders, and inputting each of the latent values into a respective second encoder for the respective latent value, resulting in a respective embedding of each of the respective confounders; f) comprises combining the embeddings of all the identified parent variables and confounders together, thereby resulting in said combined embedding; and g) comprises inputting the combined embedding into a respective decoder associated with the selected variable, resulting in a reconstructed value of the selected variable.

[0356] Statement 3A. The method of Statement 2A, comprising: i) for a given training data point comprising a given combination of input values of the variables of the feature vector, repeating a)-g) over multiple selections, each selection selecting a different one of the variables of the feature vector as the selected variable thereby resulting in a respective reconstructed value, the multiple selections together thereby resulting in a respective reconstructed version of the training data point comprising the reconstructed values for the training data point; ii) evaluating a measure of difference between the training data point and the reconstructed version; and iii) training parameters of the inference network, first and second encoders, decoders, and first and second graph distributions, based on the evaluated measure.

[0357] Statement 4A. The method of Statement 3A, comprising repeating i)-iii) over multiple input data points, each comprising a different combination of input values of the variables of the feature vector.

[0358] Statement 5A. The method of Statement 4A, comprising, after the training over the multiple training data points, observing the second graph distribution to estimate whether a confounder exists between a target pair of the variables of the feature vector.

[0359] Statement 6A. The method of Statement 5A, comprising, after the training over the multiple training data points, observing the latent value of the respective confounder between the target pair of variables, resulting from the inference network, as an estimated value of the respective confounder.

[0360] Statement 7A. The method of Statement 5A or 6A, wherein the variables of the feature vector model properties of a real-world entity or an environment thereof, and the method comprises, in dependence in said estimating of whether a confounder exists between the target pair of variables, determining whether or not to apply a treatment to one of said target pair of variables to affect the other of said target pair of variables.

[0361] Statement 8A. The method of Statement 7A, wherein one of: the real-world entity comprises a living being, and the treatment comprises a medical treatment to the living being or the environment thereof; or the real-world entity comprises a mechanical, electrical or electronic device or system, or an environment thereof; and the treatment comprises an act of maintaining, debugging, upgrading or controlling the device or system, or controlling the environment thereof; or the real-world entity comprises a network or software, and the treatment comprises an act of controlling the network or software.

[0362] Statement 9A. The method of any of Statements 2A to 8A, comprising: I) setting the input value of an intervened-on variable of the feature vector, other than the selected variable, to a specified value; and II) estimating an effect of the intervened-on variable on the selected variable based on the reconstructed value of the selected variable.

[0363] Statement 10A. The method of Statement 9A, wherein: I) comprises setting the input value of a plurality of intervened-on variables of the feature vector, other than the selected variable, to respective specified values; and II) comprises estimating the effect of the plurality of intervened-on variables based on the reconstructed value of the selected variable.

[0364] Statement 11A. The method of Statement 9A or 10A, wherein: I) comprises repeating a)-g) over multiple graph-sampling events, each using the same selected variable and setting the intervened-on variable to the same specified value, but performing the sampling of the first and second causal graphs afresh each time, thereby resulting in a respective reconstructed value of the selected variable from each graph-sampling events; and II) comprises determining an average treatment affect of the intervened-on variable on the selected variable averaged over the multiple graph-sampling events.

[0365] Statement 12A. The method of Statement 11A, wherein each graph-sampling event sets a plurality of intervened-on values to specified values, the same values each time; and II) comprises determining the average treatment effect of the plurality of intervened-on variables on the selected variable.

[0366] Statement 13A. The method of Statement 9A, wherein the intervened-on variable models a treatment on a real-world entity or an environment thereof and the target variable models an effect of the treatment applied to the real-world entity, and the method further comprises actioning the treatment on the real-world entity in dependence on the estimated treatment effect.

[0367] Statement 14A. The method of Statement 13A, wherein one of: the real-world entity comprises a living being, and the treatment comprises a medical treatment to the living being or an environment thereof; or the real-world entity comprises a mechanical, electrical or electronic device or system, or an environment thereof; and the treatment comprises an act of maintaining, debugging, upgrading or controlling the device or system, or controlling the environment thereof; or the real-world entity comprises a network or software, and the treatment comprises an act of controlling the network or software.

[0368] Statement 15A. The method of any of Statements 2A to 14A, wherein the inference network comprises a respective constituent inference network for each pair of variables in the feature vector, and e) comprises: for each identified confounded pair, inputting the input value of the respective parent variable and the input value of the selected variable into the respective inference network for the respective confounded pair, thereby resulting in the respective latent value of the respective confounder. Alternatively the inference network may comprise a common inference network operable to encode all the variables of the feature vector together into the respective latent for each pair.

[0369] Statement 16A. The method of any of Statements 2A to 15A, wherein each of the first and second encoders and each of the decoders comprises a neural network.

[0370] Statement 17A. The method of any of Statements 1A to 16A, wherein the inference network comprises a neural network.

[0371] Statement 18A. The method of Statement 9A or any subsequent Statement when dependent thereon, wherein the method is performed on a server system of a first party, the server system comprising one or more server units at one or more sites; and the method further comprises, by the server system of the first party: providing an application programming interface, API, enabling a second party to contact the server system via a network; receiving a request from the second party over the network via the API; in response to the request, determining the estimated treatment effect on the target variable; and returning the estimated treatment effect to the second party over the network via the API.

[0372] Statement 19A. A computer program embodied on computer-readable storage, wherein the computer program comprises a machine learning model comprising a plurality of first encoders, a plurality of second encoders, a decoder and an inference network; and wherein the computer-program further comprises instructions configured so as when run on one or more processors to perform the method of any of Statements 1A to 18A.

[0373] Statement 20A. A system comprising: processing apparatus comprising one or more processors; and memory comprising one or more memory units, wherein the memory stores: a machine learning model comprising the first encoders, second encoders, decoders, and inference network of any preceding Statement; and code arranged to run on the processing apparatus and being configured so as when run to perform the method of any of Statements 1A to 18A.

[0374] The first aspect (Statement 1) or any embodiment thereof may be used independently or in conjunction with the second aspect (Statement 1A) or any embodiment thereof.

[0375] In some examples, the model 104/104′/104″ and associated computer executable instructions are provided using any computer-readable media that are accessible by the computing equipment 102. Computer-readable media include, for example, computer storage media such as memory and communications media. Computer storage media include volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or the like. Computer storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), persistent memory, phase change memory, flash memory or other memory technology, Compact Disk Read-Only Memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, shingled disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing apparatus. In contrast, communication media may embody computer readable instructions, data structures, program modules, or the like in a modulated data signal, such as a carrier wave, or other transport mechanism. As defined herein, computer storage media do not include communication media. Therefore, a computer storage medium should not be interpreted to be a propagating signal per se. Propagated signals per se are not examples of computer storage media. Although the computer storage medium may be described within the computing equipment 102, it will be appreciated by a person skilled in the art, that, in some examples, the storage is distributed or located remotely and accessed via a network or other communication link (e.g., using a communication interface).

[0376] Other variants or use cases of the disclosed techniques may become apparent to the person skilled in the art once given the disclosure herein. The scope of the disclosure is not limited by the described embodiments but only by the accompanying claims.

Claims

The invention claimed is:

1. A computer-implemented method comprising:

selecting a variable from among variables of a feature vector;

sampling a temporal causal graph from a temporal graph distribution, the temporal graph distribution specifying probabilities of directed causal edges between different ones of the variables of the feature vector at a present time step, and from one of the variables of the feature vector at a preceding time step to one of the variables of the feature vector at the present time step;

from among the variables of the feature vector, identifying a present parent which is a cause of the selected variable in the present time step according to the temporal causal graph, and identifying a preceding parent which is a cause of the selected variable from the preceding time step according to the temporal causal graph;

inputting input values of the present parent and the preceding parent into a respective encoder of a machine learning model, resulting in embeddings of the present and preceding parents;

combining the embeddings of the present parent and the preceding parent, resulting in a combined embedding;

inputting the combined embedding into a decoder associated with the selected variable, resulting in a reconstructed value of the selected variable, the machine learning model including the decoder;

setting input values of a plurality of intervened-on variables of the feature vector, other than the selected variable, to respective specified values;

estimating an effect of the plurality of intervened-on variables on the selected variable based on the reconstructed value of the selected variable;

determining a reconstruction loss based on the effect of the plurality of intervened-on variables on the selected variable; and

updating parameters of the encoder and the decoder of the machine learning model based on the reconstruction loss.

2. The method of claim 1, wherein the machine learning model is a trained machine learning model, the method further comprising:

receiving a request to estimate an effect of the selected variable, the request including the input values of the plurality of intervened-on variables.

3. The method of claim 2, further comprising:

controlling the machine learning model to generate samples of the selected variable based on the input values of the plurality of intervened-on variables.

4. The method of claim 1, wherein each variable models a treatment on a real-world entity or an environment thereof and an effect of the treatment applied to the real-world entity, and the method further comprises actioning the treatment on the real-world entity based on the estimated effect of the treatment.

5. The method of claim 4, wherein one of:

the real-world entity comprises a living being, and the treatment comprises a medical treatment to the living being or an environment thereof;

the real-world entity comprises a mechanical, electrical or electronic device or system, or an environment thereof, and the treatment comprises an act of maintaining, debugging, upgrading or controlling the device or system, or controlling the environment thereof, or

the real-world entity comprises a network or software, and the treatment comprises an act of controlling the network or software.

6. The method of claim 1, wherein the feature vector represents sensor measurements collected from a physical system, wherein the updated parameters of the encoder and the decoder of the machine learning model are used to control operations of the physical system.

7. The method of claim 1, wherein:

setting the input values comprises repeating the selecting, the sampling, the identifying, the inputting the input values, the combining, and the inputting the combined embedding over multiple rounds, each using the same selected variable and setting the intervened-on variable to the same specified value, and performing the sampling of the temporal causal graph afresh in each round, thereby resulting in a respective reconstructed value of the selected variable from each round; and

estimating the effect comprises determining an average treatment effect of the plurality of intervened-on variables on the selected variable averaged over the multiple rounds.

8. The method of claim 7, wherein each round further comprises:

generating a history dependent noise term based on the embeddings of the preceding parents; and

combining the history dependent noise term with the reconstructed value of the selected variable, resulting in a simulated value of the reconstructed variable;

wherein said estimating based on the reconstructed values comprises: estimating the effect of the plurality of intervened-on variables on the selected variable based on the simulated values.

9. The method of claim 8, wherein the generating of the history dependent noise term comprises:

combining the embeddings of the preceding parents, resulting in a further embedding;

inputting the further embedding into the decoder associated with the selected variable, resulting in one or more parameter values; and

generating the history dependent noise term based on the one or more parameter values.

10. The method of claim 7, wherein:

each round sets a plurality of intervened-on variables and/or observed variables of the feature vector to specified input values; and

each round comprises an interior loop of the inputting the input values, the combining, and the inputting the combined embedding repeated around multiple iterations with the same sampled graph but for incrementing values of the present time step with each iteration, wherein each but the last iteration comprises reconstructed or simulated values of one or more further variables of the feature vector, and feeding back the simulated or reconstructed values as the specified input values of the next iteration.

11. The method of claim 1, further comprising:

generating a history dependent noise term for the selected variable based on embeddings of the preceding parents; and

combining the history dependent noise term with the reconstructed value of the selected variable, resulting in a simulated value of the reconstructed variable.

12. The method of claim 11, wherein embeddings of the present parents are not used to generate the history dependent noise term for the selected variable.

13. The method of claim 11, wherein the generating of the history dependent noise term comprises:

combining the embeddings of the preceding parents, resulting in a further embedding;

inputting the further embedding into the decoder associated with the selected variable, resulting in one or more parameter values; and

generating the history dependent noise term based on the one or more parameter values.

14. The method of claim 1, wherein:

the sampling further comprises sampling a second causal graph from a second graph distribution, the second causal graph modelling presence of possible confounders, a confounder being an unobserved cause of both of two variables in the feature vector;

the identifying further comprises, from among the variables of the feature vector, identifying a parent variable which is a cause of the selected variable according to the temporal causal graph, and which together with the selected variable forms a confounded pair having a respective confounder being a cause of both according to the second causal graph;

the inputting the input values further comprises inputting the input value of the parent variable and an input value of the selected variable into an inference network, resulting in a latent value modelling the respective confounder of the confounded pair, and inputting the latent value into a second encoder, resulting in an embedding of the confounder of the confounded pair; and

the combining includes combining the embeddings of the present and preceding parents with the embedding of the confounder of the confounded pair, thereby resulting in said combined embedding.

15. The method of claim 1, comprising:

for a given training data point comprising a given combination of input values of the variables of the feature vector at the present time step and each preceding time step, repeating the selecting, the sampling, the identifying, the inputting the input values, the combining, and the inputting the combined embedding over multiple selections, each selection selecting a different one of the variables of the feature vector as the selected variable thereby resulting in a respective reconstructed value, the multiple selections together thereby resulting in a reconstructed version of the training data point comprising the reconstructed values for the training data point;

evaluating a measure of difference between the training data point and the reconstructed version; and

training model parameters of the encoders, decoders, and temporal graph distributions, based on the evaluated measure.

16. The method of claim 15, comprising repeating the repeating, the evaluating, and the training over multiple input data points, each comprising a different combination of input values of the variables of the feature vector.

17. The method of claim 15, wherein:

each selection in the repeating further comprises a history dependent noise model generating a respective history dependent noise term based on embeddings of the preceding parents;

wherein the training further comprises training model parameters of the history dependent noise model based on the evaluated measure.

18. The method of claim 17, wherein in each selection in the repeating, the generating of the history dependent noise term by the history dependent noise model comprises:

combining the embeddings of the preceding parents, resulting in a respective further embedding;

inputting the respective further embedding into the decoder associated with the selected variable, resulting in one or more parameter values; and

generating the respective history dependent noise term based on the one or more parameter values.

19. A system comprising:

processing apparatus comprising one or more processors; and

memory comprising one or more memory units, wherein the memory stores:

a machine learning model comprising a plurality of encoders and a plurality of decoders, and

code arranged to run on the processing apparatus and being configured so as when run to perform a method comprising:

selecting a variable from among variables of a feature vector;

sampling a temporal causal graph from a temporal graph distribution, the temporal graph distribution specifying probabilities of directed causal edges between different ones of the variables of the feature vector at a present time step, and probabilities of causal edges existing from each of a plurality of variables of the feature vector at one or more preceding time steps to one or more variables of the feature vector at the present time step;

from among the variables of the feature vector, identifying a present parent which is a cause of the selected variable in the present time step according to the sampled temporal causal graph, and identifying a preceding parent which is a cause of the selected variable from any preceding time step according to the sampled temporal causal graph;

inputting input values of the present parent and the preceding parent into a respective one of the encoders of the machine learning model, resulting in a respective embedding of the present and preceding parents;

combining the embeddings of the present and preceding parents, resulting in a combined embedding;

inputting the combined embedding into a respective one of the decoders associated with the selected variable, resulting in a reconstructed value of the selected variable, the machine learning model including the decoder;

setting input values of a plurality of intervened-on variables of the feature vector, other than the selected variable, to respective specified values;

estimating an effect of the plurality of intervened-on variables on the selected variable based on the reconstructed value of the selected variable;

determining a reconstruction loss based on the effect of the plurality of intervened-on variables on the selected variable; and

updating parameters of the encoders and the decoders of the machine learning model based on the reconstruction loss.

20. A computer program embodied on non-transitory computer-readable storage, wherein the computer program comprises instructions configured so as when run on one or more processors to perform a method comprising:

selecting a variable from among variables of a feature vector;

sampling a temporal causal graph from a temporal graph distribution, the temporal graph distribution specifying probabilities of directed causal edges between different ones of the variables of the feature vector at a present time step, and probabilities of causal edges existing from each of a plurality of variables of the feature vector at one or more preceding time steps to one or more variables of the feature vector at the present time step;

inputting input values of the present parent and the preceding parent into a respective first encoder of a machine learning model, resulting in a respective embedding of the present and preceding parents;

combining the embeddings of the present and preceding parents, resulting in a combined embedding;

inputting the combined embedding into a decoder associated with the selected variable, resulting in a reconstructed value of the selected variable, the machine learning model including the decoder;

setting input values of a plurality of intervened-on variables of the feature vector, other than the selected variable, to respective specified values;

estimating an effect of the plurality of intervened-on variables on the selected variable based on the reconstructed value of the selected variable;

determining a reconstruction loss based on the effect of the plurality of intervened-on variables on the selected variable; and

updating parameters of the encoders and the decoders of the machine learning model based on the reconstruction loss.

Patent Citations (29)

Patent	Date	Inventor	Cited By
US7117185(B1)	2006-10-01	Aliferis	Examiner
US10699450(B2)	2020-06-01	Weissbrod	Examiner
US11468348(B1)	2022-10-01	Cui	Applicant
US12034747(B2)	2024-07-01	Raghavendra	Examiner
US2009/0006289(A1)	2009-01-01	Jaros	Examiner
US2011/0112998(A1)	2011-05-01	Abe	Examiner
US2011/0167031(A1)	2011-07-01	Kleinberg	Examiner
US2013/0046721(A1)	2013-02-01	Jiang	Examiner
US2016/0292248(A1)	2016-10-01	Garcia	Examiner
US2017/0220937(A1)	2017-08-01	Wada	Examiner
US2019/0326019(A1)	2019-10-01	Ramazzotti	Examiner
US2020/0233920(A1)	2020-07-01	Meeds	Examiner
US2020/0245009(A1)	2020-07-01	Saini	Examiner
US2020/0279417(A1)	2020-09-01	Lee	Applicant
US2021/0064517(A1)	2021-03-01	Wong	Examiner
US2021/0256406(A1)	2021-08-01	Lau	Examiner
US2021/0295955(A1)	2021-09-01	Lee et al.	Applicant
US2022/0093271(A1)	2022-03-01	Huang	Examiner
US2023/0065173(A1)	2023-03-01	Shibahara	Examiner
US2023/0072173(A1)	2023-03-01	Gupta	Examiner
US2023/0128111(A1)	2023-04-01	Greenewald	Examiner
US2023/0229906(A1)	2023-07-01	Zhang et al.	Applicant
US2023/0359867(A1)	2023-11-01	Park	Examiner
US2023/0376734(A1)	2023-11-01	Liu	Examiner
US2023/0401423(A1)	2023-12-01	Sim	Examiner
US2024/0047042(A1)	2024-02-01	Daza	Examiner
US2024/0104370(A1)	2024-03-01	Ma	Applicant
US2024/0126945(A1)	2024-04-01	Bellot	Examiner
WO-2007144804(A1)	2007-12-01		Examiner

Non-Patent Literature (91)

Liu, Yan, et al. “Learning temporal causal graphs for relational time-series analysis.” Proceedings of the 27th International Conference on Machine Learning (ICML-10). 2010. (Year: 2010).Examiner
Gong, Mingming, et al. “Causal discovery from temporally aggregated time series.” Uncertainty in artificial intelligence: proceedings of the . . . conference. Conference on Uncertainty in Artificial Intelligence. vol. 2017. 2017. (Year: 2017).Examiner
Gootjes-Dreesbach, Luise, et al. “Variational autoencoder modular Bayesian networks for simulation of heterogeneous clinical study data.” Frontiers in big Data 3 (2020): 16. (Year: 2020).Examiner
Zhang, Xin, and Jiali You. “A gated dilated causal convolution based encoder-decoder for network traffic forecasting.” IEEE Access 8 (2020): 6087-6097. (Year: 2020).Examiner
Wang, Yanbang, et al. “Inductive representation learning in temporal networks via causal anonymous walks.” arXiv preprint arXiv: 2101.05974v1 (2021). (Year: 2021).Examiner
Berrevoets, Jeroen, et al. “Disentangled counterfactual recurrent networks for treatment effect inference over time.” arXiv preprint arXiv:2112.03811v1 (2021). (Year: 2021).Examiner
Kaddour, Jean, et al. “Causal effect inference for structured treatments.” Advances in Neural Information Processing Systems 34 (2021): 24841-24854. (Year: 2021).Examiner
Moraffah, Raha, et al. “Causal Inference for Time series Analysis: Problems, Methods and Evaluation.” arXiv preprint arXiv: 2102.05829v1 (2021). (Year: 2021).Examiner
Chen, Y., Zhang, J. & Qin, X. Interpretable instance disease prediction based on causal feature selection and effect analysis. BMC Med Inform Decis Mak 22, 51 (2022). https://doi.org/10.1186/s12911-022-01788-8 (Year: 2022).Examiner
Wang, L., Adiga, A., Chen, J., Sadilek, A., Venkatramanan, S., & Marathe, M. (Jun. 2022). Causalgnn: Causal-based graph neural networks for spatio-temporal epidemic forecasting. In Proceedings of the AAAI conference on artificial intelligence (vol. 36, No. 11, pp. 12191-12199). (Year: 2022).Examiner
Wang, Xinyue, et al. “Meta-learning Causal Discovery.” arXiv preprint arXiv:2209.05598v1 (2022) (Year: 2022).Examiner
Zhang, Tao, Hao-Ran Shan, and Max A. Little. “Causal GraphSAGE: A robust graph method for classification based on causal sampling.” Pattern recognition 128 (2022): 108696. (Year: 2022).Examiner
Amornbunchornvej, et al., “Variable-Lag Granger Causality for Time Series Analysis”, In Proceedings of IEEE International Conference on Data Science and Advanced Analytics (DSAA), Oct. 5, 2019, pp. 21-30.Applicant
Bellot, et al., “Deconfounded Score Method: Scoring DAGs with Dense Unobserved Confounding”, In Repository of arXiv:2103.15106v2, May 25, 2021, 19 Pages.Applicant
Zhang, et al., “On the Identifiability of the Post-Nonlinear Causal Model”, In Repository of arXiv:1205.2599v1, May 9, 2012, pp. 647-655.Applicant
Bhattacharya, et al., “Differentiable Causal Discovery Under Unmeasured Confounding”, In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR, Apr. 13, 2021, 11 Pages.Applicant
Blei, et al., “Variational Inference: A Review for Statisticians”, In Journal of The American Statistical Association, vol. 112, Issue 518, Feb. 27, 2017, pp. 859-877.Applicant
Bussmann, et al., “Neural Additive Vector Autoregression Models for Causal Discovery in Time Series”, In Proceedings of International Conference on Discovery Science, Oct. 9, 2021, pp. 446-460.Applicant
Bussmann, et al., “Neural Additive Vector Autoregression Models for Causal Discovery in Time Series”, In Repository of arXiv:2010.09429v2, Oct. 18, 2021, 18 Pages.Applicant
Cai, et al., “Triad Constraints for Learning Causal Structure of Latent Variables”, In Proceedings of Advances in Neural Information Processing Systems, vol. 32, Dec. 8, 2019, 10 Pages.Applicant
Dang, et al., “seq2graph: Discovering Dynamic Dependencies from Multivariate Time Series with Multi-level Attention”, In Repository of arXiv:1812.04448v1, Dec. 7, 2018, 9 Pages.Applicant
Durkan, et al., “Neural Spline Flows”, In Proceedings of Advances in Neural Information Processing Systems, vol. 32, Dec. 8, 2019, 12 Pages.Applicant
Geffner, et al., “Deep End-to-end Causal Inference”, In Repository of arXiv:2202.02195v2, Jun. 20, 2022, 31 Pages.Applicant
Geffner, et al., “FCause: Flow-based Causal Discovery”, Retrieved From: https://openreview.net/attachment?id=HO_LL-oqBzW&name=pdf, Sep. 29, 2021, 19 Pages.Applicant
Glymour, et al., “Review of Causal Discovery Methods Based on Graphical Models”, In Journal of Frontiers in Genetics, vol. 10, Article 524, Jun. 4, 2019, 15 Pages.Applicant
Granger, Clivew. , “Investigating Causal Relations by Econometric Models and Cross-spectral Methods”, In Journal of Econometrica, vol. 37, Issue 3, Aug. 1, 1969, pp. 424-438.Applicant
Guo, et al., “A Survey of Learning Causality with Data: Problems and Methods”, In Proceedings of ACM Computing Surveys, vol. 53, Issue 4, Jul. 22, 2020, 37 Pages.Applicant
Heckerman, et al., “A Bayesian Approach to Causal Discovery”, In Publication of Springer Berlin Heidelberg, 2006, 28 Pages.Applicant
Hoyer, et al., “Nonlinear Causal Discovery with Additive Noise Models”, In Proceedings of Advances in Neural Information Processing Systems (NIPS), vol. 21, Dec. 8, 2008, 8 Pages.Applicant
Hyvärinen, et al., “Estimation of a Structural Vector Autoregression Model Using Non-Gaussianity”, In Journal of Machine Learning Research, vol. 11, May 1, 2010, pp. 1709-1731.Applicant
Jang, et al., “Categorical Reparameterization with Gumbel-Softmax”, In Repository of arXiv:1611.01144v1, Nov. 3, 2016, 13 Pages.Applicant
Kaiser, et al., “Unsuitability of NOTEARS for Causal Graph Discovery”, In Repository of arXiv:2104.05441v1, Apr. 12, 2021, 5 Pages.Applicant
Zheng, et al., “DAGs with NO Tears: Continuous Optimization for Structure Learning”, In Proceedings of Advances n Neural Information Processing Systems, vol. 31, Dec. 3, 2018, 12 Pages.Applicant
Lippe, et al., “Efficient Neural Causal Discovery without Acyclicity Constraints”, In Repository of arXiv:2107.10483v1, Jul. 22, 2021, 34 Pages.Applicant
Löwe, et al., “Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data”, In Proceedings of Machine Learning Research, vol. 140, Apr. 11, 2022, 17 Pages.Applicant
Maathuis, et al., “Handbook of Graphical Models”, In Publication of CRC Press, Nov. 12, 2018, 546 Pages.Applicant
Maddison, et al., “The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables”, In Repository of arXiv:1611.00712v2, Nov. 6, 2016, 17 Pages.Applicant
Monti, et al., “Autoregressive Flow-Based Causal Discovery and Inference”, In Repository of arXiv:2007.09390v1, Jul. 18, 2020, 6 Pages.Applicant
Moraffah, et al., “Causal Inference for Time Series Analysis: Problems, Methods and Evaluation”, In Journal of Knowledge and Information Systems, vol. 63, Nov. 23, 2021, pp. 3041-3085.Applicant
Naser, et al., “Causality, Causal Discovery, and Causal Inference in Structural Engineering”, In Repository of arXiv:2204.01543v3, Aug. 8, 2022, 27 Pages.Applicant
Ness, et al., “A Bayesian Active Learning Experimental Design for Inferring Signaling Networks”, In Proceedings of International Conference on Research in Computational Molecular Biology, May 3, 2017, 11 Pages.Applicant
Pamfil, et al., “DYNOTEARS: Structure Learning from Time-Series Data”, In Repository of arXiv:2002.00498v2, Apr. 27, 2020, 23 Pages.Applicant
Pamfil, et al., “DYNOTEARS: Structure Learning from Time-Series Data”, In Proceedings of International Conference on Artificial Intelligence and Statistics, Aug. 26, 2020, 10 Pages.Applicant
Pawlowski, et al., “Deep Structural Causal Models for Tractable Counterfactual Inference”, In Proceedings of Advances in Neural Information Processing Systems, vol. 33, Dec. 6, 2020, 13 Pages.Applicant
Pearl, Judea, “Causal Inference in Statistics: An Overview”, In Journal of Statistics Surveys, vol. 3, Oct. 13, 2009, pp. 96-146.Applicant
Pena, Jose M. , “Learning Acyclic Directed Mixed Graphs from Observations and Interventions”, In Proceedings of the Eighth International Conference on Probabilistic Graphical Models, PMLR, vol. 52, Sep. 6, 2016, pp. 392-403.Applicant
Pena, Josem. , “Reasoning with Alternative Acyclic Directed Mixed Graphs”, In Journal of Behaviormetrika, vol. 45, Issue 2, Oct. 2018, pp. 389-422.Applicant
Peters, et al., “Causal Inference on Time Series using Restricted Structural Equation Models”, In Proceedings of Advances in Neural Information Processing Systems, vol. 26, Dec. 5, 2013, 9 Pages.Applicant
Peters, et al., “Causal Inference on Time Series using Structural Equation Models”, In Repository of arXiv:1207.5136v1, Jul. 21, 2012, 11 Pages.Applicant
Peters, et al., “Elements of Causal Inference: Foundations and Learning Algorithms”, Published by MIT Press, Nov. 29, 2017, 289 Pages.Applicant
Peters, et al., “Identifiability of Causal Graphs using Functional Models”, In Repository of arXiv:1202.3757v1, Feb. 14, 2012, 10 Pages.Applicant
Reisach, et al., “Beware of the Simulated DAG! Varsortability in Additive Noise Models”, In Repository of arXiv:2102.13647v1, Feb. 26, 2021, 19 Pages.Applicant
Richardson, Thomas, “Markov Properties for Acyclic Directed Mixed Graphs”, In Scandinavian Journal of Statistics, vol. 30, No. 1, Feb. 28, 2003, pp. 145-157.Applicant
Runge, Jakob, “Causal network reconstruction from time series: From theoretical assumptions to practical estimation”, In Journal of Chaos: An Interdisciplinary Journal of Nonlinear Science, vol. 28, Issue 7, Jul. 24, 2018, 20 Pages.Applicant
Shojaie, et al., “Discovering Graphical Granger Causality Using the Truncating Lasso Penalty”, In Journal of Bioinformatics, vol. 26, Issue 18, Sep. 15, 2010, pp. 1517-i523.Applicant
Siggiridou, et al., “Granger Causality in Multivariate Time Series Using a Time-Ordered Restricted Vector Autoregressive Model”, In Journal of IEEE Transactions on Signal Processing, vol. 64, Issue 7, Apr. 2016, pp. 1759-1773.Applicant
Spirtes, et al., “Causation, Prediction, and Search”, In Publication of MIT Press, 2000, 546 Pages.Applicant
Tank, et al., “Neural Granger Causality for Nonlinear Time Series”, In Journal of Stat, vol. 1050, Feb. 19, 2018, 18 Pages.Applicant
Tian, et al., “A General Identification Condition for Causal Effects”, In Publication of eScholarship, University of California, Aug. 1, 2002, pp. 567-573.Applicant
Trippe, et al., “Conditional Density Estimation with Bayesian Normalising Flows”, In Repository of arXiv:1802.04908v1, Feb. 14, 2018, 19 Pages.Applicant
Wu, et al., “Discovering Nonlinear Relations with Minimum Predictive Information Regularization”, In Repository of arXiv:2001.01885v1, Jan. 7, 2020, 26 Pages.Applicant
Xu, et al., “Scalable Causal Graph Learning through a Deep Neural Network”, In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Nov. 3, 2019, 10 Pages.Applicant
International Search Report and Written Opinion received for PCT Application No. PCT/US2023/031000, Dec. 15, 2023, 19 pages.Applicant
International Search Report and Written Opinion received for PCT Application No. PCT/US23/031103, mailed on Jan. 9, 2024, 14 pages.Applicant
Lowe S., et al., “Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data,” arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Feb. 21, 2022, 27 pages.Applicant
Meng Y., “Estimating Granger Causality with Unobserved Confounders via Deep Latent-Variable Recurrent Neural Network,” arxiv.org, Cornell University Library, 201 Olin Library, Cornell University Ithaca, NY 14853, Sep. 9, 2019, 9 pages.Applicant
Wang, et al., “Estimating Individualized Causal Effect with Confounded Instruments”, Proceedings of the 2020 5th International Conference on Big Data and computing, Aug. 14, 2022, pp. 1857-1867.Applicant
Zecevic M., et al., “Relating Graph Neural Networks to Structural Causal Models,” arxiv.org, Cornell University Library, 201 Olin Library, Cornell University Ithaca, NY 14853, Oct. 22, 2021, 29 pages.Applicant
“International Search Report and Written Opinion Issued in PCT Application No. PCT/US23/010159”, Mailed Date: Apr. 19, 2023, 11 Pages.Applicant
Chen et al., “Heterogeneous Treatment Effect Estimation through Deep Learning”, arXiv:1810.11010v1, Oct. 25, 2018, 26 pages.Applicant
Fehner, Gerald, “Estimating casual effects of credit decisions”, International Journal of Forecasting, vol. 28, 2012, pp. 248-260.Applicant
International Preliminary Report on Patentability received for PCT Application No. PCT/US2023/031000, Apr. 3, 2025, 14 pages.Applicant
International Preliminary Report on Patentability received for PCT Application No. PCT/US23/031103, mailed on Apr. 3, 2025, 10 pages.Applicant
Non-Final Office Action mailed on Apr. 22, 2025, in U.S. Appl. No. 17/579,877, 59 pages.Applicant
Sanchez-Martin, et al., “VACA: Design of Variational Graph Autoencoders for Interventional and Counterfactual Queries”, arXiv:210.14690v1, Oct. 27, 2021, 31 pages.Applicant
Zheng, et al., “DAGs with NO Tears: Smooth Optimization for Structure Learning”, In Repository of arXiv:1803.01422v1, Mar. 4, 2018, 14 Pages.Applicant
Zhang, et al., “Advances in Variational Inference”, In Journal of IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, Issue 8, Dec. 24, 2018, pp. 2008-2026.Applicant
Zhang, et al., “On Estimation of Functional Causal Models: General Results and Application to the Post-Nonlinear Causal Model”, In Journal of ACM Transactions on Intelligent Systems and Technology (TIST), vol. 7, No. 2, Dec. 17, 2015, 23 Pages.Applicant
Final Office Action mailed on Jan. 8, 2026, in U.S. Appl. No. 17/936,338, 26 Pages.Applicant
Zheng, et al., “Dags with No. tears: Continuous optimization for structure learning”, arXiv:1803.01422, Nov. 3, 2018, 22 pages.Applicant
Mireshghallah, et al., “Privacy in Deep Learning: A Survey”, In Repository of arXiv:2004.12254, Version.5, Nov. 7, 2020, 24 Pages.Applicant
Non-Final Office Action mailed on Sep. 8, 2025, in U.S. Appl. No. 17/936,338, 39 Pages.Applicant
Sharp, et al., “Creating Causal Embeddings for Question Answering with Minimal Supervision”, In Repository of arXiv:1609.08097v1, Sep. 26, 2016, 11 Pages.Applicant
Zecevic, et al., “Relating Graph Neural Networks to Structural Causal Models”, In Repository of arXiv:2109.04173, vol. 3, Oct. 22, 2021, 29 Pages.Applicant
Final Office Action mailed on Nov. 13, 2025, in U.S. Appl. No. 17/579,877, 71 Pages.Applicant
First Examination Report Received for Indian Application No. 202417053257, mailed on Nov. 21, 2025, 07 Pages.Applicant
Summer Institute in Computational Social Science, “Tutorial on deep learning for causal inference”, Retrieved from URL: https://www.youtube.com/watch?v=v9uf9rDYEMg, Published on: Jun. 25, 2021, 01 Page.Applicant
Yu, et al., “Dag-gnn: Dag structure learning with graph neural networks”, In Proceedings of the 36th International Conference on Machine Learning, 2019, 10 pages.Applicant
Notice of Allowance mailed on Apr. 15, 2026, in U.S. Appl. No. 17/936,338, 18 Pages.Applicant
Kuipers, et al., “Efficient and Structure Learning of Bayesian Networks”, In Repository of arXiv:1803.07859v4, Nov. 9, 2021, 41 Pages.Applicant
Non-Final Office Action mailed on May 14, 2026, in U.S. Appl. 17/579,877, 37 Pages.Applicant

Abstract