Cost function engineering for estimating uncertainty correlated with prediction errors

[0001] The present application is a National Stage Entry of International Application No. PCT/US20/21057066, filed Oct. 28, 2021, which claims priority to U.S. Provisional Patent Application 63/107,294, filed Oct. 29, 2020, the entirety of which is incorporated by reference.

[0002] Aspects of the disclosure relate to computer programming. More specifically, aspects of the disclosure relate to cost function engineering using neural networks.

[0003] Deep learning techniques have been shown to be able to provide uncertainties together with point estimates for various classification and regression problems. Enabling these uncertainties to be correlated and calibrated to the error, however, is still considered challenging. Different methods may be used. Methods such as Bayesian neural networks (BNN) and those using dropout for Bayesian inference have become very popular for providing the epistemic uncertainty, while the use of log-likelihood loss function in artificial neural networks (ANN) have incorporated the aleatoric uncertainty.

[0004] In practice, the uncertainties are often found to be poorly correlated with the absolute error, questioning their value. BNNs have been traditionally used for epistemic uncertainty estimation by determining the posterior weight distributions of the neural networks, providing us with both a mean and variance of the predicted outputs using variational inference. Recently, dropout as a technique for approximating Bayesian inference has also been successfully introduced and has seen widespread uptake in the community. As these methods are sensitive to the epistemic uncertainty they have also been shown to be useful to differentiate in and out-of-distribution predictions. The aleatoric uncertainty which originates from the data and is inherent to the measurement, could be either homoscedastic or heteroscedastic.

[0005] Recently, conventional analysis has addressed the issue of sensitivity of the neural network model to the aleatoric uncertainty with a log-likelihood loss function. They have shown that when such an approach is combined with dropout based variational inference, sensitivity to both epistemic and aleatoric uncertainty can be achieved. But the inherent challenge that the total uncertainty, which is the sum of the aleatoric and epistemic uncertainty is not necessarily well correlated to the absolute error in the prediction, makes the reliance on these uncertainties questionable.

[0006] There is a need to provide an analysis technique that will provide aleatoric and epistemic uncertainty in analysis.

[0007] There is a further need to take the aleatoric and epistemic uncertainty and correlate this uncertainty to the absolute error in predictions.

[0008] There is a further need to provide analysis techniques that are more accurate than conventional analysis techniques.

[0009] So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized below, may be had by reference to embodiments, some of which are illustrated in the drawings. It is to be noted that the drawings illustrate only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective embodiments without specific recitation. Accordingly, the following summary provides just a few aspects of the description and should not be used to limit the described embodiments to a single concept.

[0010] In one example embodiment, a method for performing a calculation is disclosed. The method may comprise providing a neural network and accepting a data set into the neural network. The method may also comprise performing calculations with the neural network using the data set, wherein the calculations use a loss function and provide an aleatoric and epistemic uncertainty that is correlated to a value and displaying results of the calculations performed.

[0011] In another embodiment a non-transitory computer readable medium storing a program causing a computer to execute a process is disclosed. The process performed may comprise accepting a data set into a neural network. The process may further comprise performing calculations with the neural network using the data set, wherein the calculations use a loss function and provide an aleatoric and epistemic uncertainty that is correlated to a value and displaying results of the calculations performed.

[0018] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures (“FIGS”). It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation.

[0019] In the following, reference is made to embodiments of the disclosure. It should be understood, however, that the disclosure is not limited to specific described embodiments. Instead, any combination of the following features and elements, whether related to different embodiments or not, is contemplated to implement and practice the disclosure. Furthermore, although embodiments of the disclosure may achieve advantages over other possible solutions and/or over the prior art, whether or not a particular advantage is achieved by a given embodiment is not limiting of the disclosure. Thus, the following aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the claims except where explicitly recited in a claim. Likewise, reference to “the disclosure” shall not be construed as a generalization of inventive subject matter disclosed herein and should not be considered to be an element or limitation of the claims except where explicitly recited in a claim.

[0020] Although the terms first, second, third, etc., may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms may be only used to distinguish one element, components, region, layer or section from another region, layer or section. Terms such as “first”, “second” and other numerical terms, when used herein, do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed herein could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.

[0021] When an element or layer is referred to as being “on,” “engaged to,” “connected to,” or “coupled to” another element or layer, it may be directly on, engaged, connected, coupled to the other element or layer, or interleaving elements or layers may be present. In contrast, when an element is referred to as being “directly on,” “directly engaged to,” “directly connected to,” or “directly coupled to” another element or layer, there may be no interleaving elements or layers present. Other words used to describe the relationship between elements should be interpreted in a like fashion. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed terms.

[0022] Some embodiments will now be described with reference to the figures. Like elements in the various figures will be referenced with like numbers for consistency. In the following description, numerous details are set forth to provide an understanding of various embodiments and/or features. It will be understood, however, by those skilled in the art, that some embodiments may be practiced without many of these details, and that numerous variations or modifications from the described embodiments are possible. As used herein, the terms “above” and “below”, “up” and “down”, “upper” and “lower”, “upwardly” and “downwardly”, and other like terms indicating relative positions above or below a given point are used in this description to more clearly describe certain embodiments.

[0023] Aspects of the disclosure provide for a new loss function for ANNs which provides an uncertainty that captures both the epistemic and aleatoric contributions and explicitly correlates it to the prediction bias or error.

[0024] A custom loss function for neural networks with three terms as shown below in equation:

[0025] $\begin{array}{cc}L\left(x,y,\theta \right)=\frac{1}{N}{\sum }_{i=1}^N\frac{1}{{\sigma }_i^2}{f_{\theta }\left(x_i\right)-y_i}^2+\log \left[{\sigma }_i^2\right]+{\alpha \left({\sigma }_i-❘f_{\theta }\left(x_i\right)-y_i❘\right)}^2& \left(1\right)\end{array}$

[0026] The loss function is minimized with respect to the model parameters θ and variance

[0027] ${\sigma }_i^2$
represents the aleatoric uncertainty corresponding to output y_i. The first term represents the accuracy of the model while the second term prevents the assignment of very high uncertainty values for all data points. The first two terms are related to addressing aleatoric uncertainty. A third term, which forces the σ_i to be a function of not only the measurement y_i but also the model prediction ƒ_θ(x_i), (thus enabling it to absorb both the epistemic as well as the aleatoric uncertainties. In the limit of infinite data, the model can learn well, thus, from an expectation perspective, the following can be obtained σ_i|y_i−|. Therefore, the third term in cost function is expected to go to zero and the σ_i represents the aleatoric uncertainty. The parameter α is a regularization parameter and can be tuned to balance the trade-off between achieving a low mean square error (MSE) versus a high correlation between uncertainty and error.
Demonstration on Synthetic Data

[0028] The application of this approach is demonstrated on a synthetic dataset of 3,000 points with six input features (x₁ through x₆). The features were sampled from a normal distribution with mean values of 0, 3, 0, 5, 0, 0 and standard deviations of 4, 2, 2, 1, 2, 2 respectively. The dependent variable for each row of the synthetic data was generated using equation 1, with an additional noise term N(0, σ_i) with σ_i sampled from a uniform distribution U(0,2). The synthetic data was randomly split into 80% and 20% to create the training and in-distribution testing datasets, and were normalized based on the training set statistics.

[0029] $\begin{array}{cc}{y}_i=\sin \left(x_i\right)+x_2^2+2x_1{x}_3^2+\sqrt{x_4}+e^{x_2{x}_5^2}+\frac{3x_6}{0.2+❘x_1❘}+N\left(0,{\sigma }_i\right)& \left(2\right)\end{array}$

[0030] A neural network incorporating our custom loss function was constructed to predict the output point estimates as well as its uncertainty simultaneously, with the help of two output neurons. The network architecture consisted of 3 hidden layers with 20 neurons each. A ReLU activation function was used for the hidden layers and an RMSprop optimizer for minimizing the loss function. A dropout percent of 5% during training was added to each of the hidden layers to make the model robust to overfitting. To obtain the optimal a for our custom loss function, we employed a five-fold cross validation and plotted absolute-error versus uncertainty correlation as well as MSE for different alpha values as shown in FIG. 1.

[0031] There is an inherent tradeoff between error-uncertainty correlation and prediction accuracy as shown in FIG. 1. At higher α values, the third term of the custom loss function that enforces that the predicted standard deviation be correlated with the absolute error is weighted more than the first two terms. Hence, the error-uncertainty correlation coefficient (R) increases while the MSE, also increases. On the other hand, a very low value of α leads to a low MSE but is also accompanied by a low correlation coefficient. This trade-off between MSE and R obtained by varying α can be depicted in the form of a Pareto front in FIG. 2. Depending on the application, different values of α on this frontier can be selected, based on the importance of the tradeoffs between the two values.

[0032] In this example, the performance of the algorithm on a test data set for an optimal α of 750 is shown in FIG. 3. It is observed that the MSE on the figure is small while the uncertainties are also highly correlated with the prediction error.

[0033] For comparison we create a second neural network with the same architecture but optimized for MSE to predict the point estimate, incorporated with 20% dropout during testing and run 500 times to also help predict the standard deviation of the predictions. The performance of this network is shown in FIG. 4.

[0034] A third dropout BNN was also created but with its architecture tuned to obtain the best possible MSE for comparison. The results of this network on the same testing data is shown in FIG. 5. In the case of the custom loss function, we can substantially increase the correlation between the model's predicted uncertainty and prediction error in exchange for a slight sacrifice in the MSE as summarized in Table 1.

[0035]

TABLE 1
Prediction error and absolute-error/uncertainty
correlation for each model.
	Test set	Error-uncertainty
	prediction	correlation
	error (MSE)	(Pearson's R)
Neural network with custom loss	0.16	0.84
function (FIG. 3)
Neural network with same	0.13	0.15
architecture and dropout (FIG. 4)
Best neural network for dropout	0.15	0.40
(FIG. 5)

[0036] Aspects of the disclosure provide for a loss function that may be used, for example, with artificial neural networks. These artificial neural networks may be used to provide analysis of data to aid in various functions of data processing. For example, data may be obtained from a geological scan for a hydrocarbon recovery project, and may be analyzed through the use of an artificial neural network to allow for evaluation of potential hydrocarbons trapped within strata of a geological formation. The artificial neural network may be used to separate data that would be indicative of the presence of hydrocarbons from portions of a geological stratum that would not provide for any hydrocarbon recovery. The data may be grouped, for example, in such a situation to determine where the highest concentrations of hydrocarbons may occur in order to establish a model of the geological parameters. This model may be a two-dimensional or a three-dimensional model to allow geological engineers to determine the size, location and cost that would be associated with developing any hydrocarbon reservoir determined by the geological scan. A drilling plan may be then established to determine if a single or multiple wells may be created in order to exploit the hydrocarbon reserves located beneath the surface. In one example embodiment, the drilling plan may be used to determine if a single complex geometry well could be used to recover hydrocarbons from the geological formation to limit the amount of drilling and maximize the potential profit from the drilling operation.

[0037] In other embodiments, other types of data may be evaluated through the use of the artificial neural network that employs embodiments of the disclosure. For example, embodiments of the artificial neural network may be employed to evaluate data related to a wireline scan that is achieved to determine various geological formation parameters. As will be understood, certain geological parameters can significantly impact the analysis of determining the size and location of a hydrocarbon field. By quickly and efficiently setting up an artificial neural network using embodiments of the disclosure, data outliers may be excluded from analysis providing a more robust evaluation that may be more accurate and cost-efficient from a computing standpoint. Lessons learned from previous evaluations may be fed back into the artificial neural network thereby providing an even greater efficiency for the evaluation process. As will be understood, with greater training, the artificial neural network may include greater numbers of nodes and potential feedback thereby creating a system that will allow for identification of potential errors and exclusion of extraneous data from large and complex data sets. As will be understood, various numbers of nodes and levels may be provided in different aspects of the disclosure.

[0038] In embodiments, synthetic data may be used to repetitively train the artificial neural network using multiple repetitions to allow the artificial neural network to be capable of handling real-world data with minimal error the first time the artificial neural network is used. Such use of artificial neural networks may reduce the overall cost of an engineering operation for a hydrocarbon well based upon the accurate and swift evaluation of multiple data streams provided to the artificial neural network. As an example, data may be gathered together such that the important factor for the project is the minimal cost engineering expenditures for the establishment of wells to tap the hydrocarbon reservoir. Such an evaluation basis may be different than providing a drilling plan that would maximize the overall amount of hydrocarbons recovered from the geological stratum. In such instances, the use of the artificial neural network would minimize capital outlay for a potential drilling operation thereby allowing the drilling operation to be performed wherein operator may have minimal capital expenditure capabilities. Thus, a hydrocarbon recovery well may be established that takes cost engineering into account to exploit the geological formation in the most cost productive way possible.

[0039] In other evaluations, the artificial neural network may be used in well rework in order to reestablish hydrocarbon flow in a previously drilled well. Data may be obtained from various sources including seismic, wireline or other evaluation intensive techniques to allow an engineer the ability to maximize fluid flow from a pre-existing reservoir and a pre-existing well. Such rework may include use of chemicals within the well to stimulate fluids flowing within the well. Estimates may be determined based upon the data analyzed to establish a rate of return for the capital expended on the pre-existing well compared to the expected amount of hydrocarbons recovered from well activities. In using such neural networks, the amount of chemicals utilized for well stimulation may be minimized, thereby creating an environmental benefit for well stimulation activities. Minimizing the amount of chemicals used for stimulation of the well would also increase the overall capital return from well production activities as such initial cost expenditures would be minimized as well as the amount of rework performed in the field.

[0040] Prior analysis from other jobsites may be also inputted into the neural network to allow, for example, the amount and selection of chemicals used for stimulation activities. For example, a cost-benefit analysis of each chemical used in a wellbore setting may be provided to engineers to allow for the most cost-efficient selection of chemicals used for stimulation purposes. Thus, costs for development of the flow within the wellbore may be minimized while the maximum amount of hydrocarbons recovered is maximized.

[0041] In order to perform such evaluations, cost function engineering may be performed by the artificial neural network on a computer, a server, a laptop, or other computing apparatus at a jobsite or at a base location. The data may be fed into the artificial neural network which is programmed into the computing apparatus to allow for evaluation of the data into its desired format. Data may be shared with a client as it is evaluated out of the neural network or may be summarized and presented to a client based upon the choice of a computer architect. In embodiments, the computer architect may choose to provide a link to the artificial neural network through a data connection that would entail use of an Internet, wireless or other data transfer mechanism. In embodiments, the artificial neural network may be placed on the Internet for evaluation of data coming from multiple sources at one time, or the artificial neural network may be a stand-alone system that only provided evaluation capability of a single set of data fed into the network. The artificial neural network may be placed upon a compact disc, universal serial bus apparatus, server, or other recording and playback means to allow for transport of the artificial neural network to a field location or location other than a single site location. Evaluation of data may be performed in a real-time analysis wherein crews from a wireline operation may instantly feed data coming from a wellbore analysis to provide operators insights as to areas of interest of the wellbore during analysis types. Such capabilities would allow for further data to be obtained from these areas of interest greatly enhancing analysis in areas that are more profitable than without the use of the artificial neural network. When utilizing this capability, the artificial neural network provides a reduction in field activities whereby data obtained from the wellbore is of a higher-quality and concentrated in areas that are needed compared to providing an analysis of an entire wellbore that perhaps would generate significant amounts of data that are not pertinent to wellbore areas of interest.

[0042] The specific embodiments described above have been shown by way of example, and it should be understood that these embodiments may be susceptible to various modifications and alternative forms. It should be further understood that the claims are not intended to be limited to the particular forms disclosed, but rather to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure.

[0043] In one example embodiment, a method for performing a calculation is disclosed. The method may comprise providing a neural network and accepting a data set into the neural network. The method may also comprise performing calculations with the neural network using the data set, wherein the calculations use a loss function and provide an aleatoric and epistemic uncertainty that is correlated to a value and displaying results of the calculations performed.

[0044] In another example embodiment, the method may be performed wherein the neural network incorporates a machine learning model.

[0045] In another example embodiment, the method may be performed wherein the loss function is of a form of:

[0046] $L\left(x,y,\theta \right)=\frac{1}{N}{\sum }_{i=1}^N\frac{1}{{\sigma }_i^2}{f_{\theta }\left(x_i\right)-y_i}^2+\log \left[{\sigma }_i^2\right]+{\alpha \left({\sigma }_i-❘f_{\theta }\left(x_i\right)-y_i❘\right)}^2.$

[0047] In another example embodiment, the method may be performed wherein the value is a prediction error.

[0048] In another example embodiment, the method may be performed wherein the value is a prediction bias.

[0049] In another example embodiment, the method may be performed wherein the neural network has three hidden layers.

[0050] In another example embodiment, the method may be performed wherein a first term of the loss function relates to an accuracy.

[0051] In another example embodiment, the method may be performed wherein a second term of the loss function prevents an assignment of uncertainty values for all data points.

[0052] In another example embodiment, the method may be performed wherein a third term of the loss function represents an aleatoric uncertainty.

[0053] In one example embodiment, a non-transitory computer readable medium storing a program causing a computer to execute a process is disclosed. The process performed may comprise accepting a data set into a neural network. The process may further comprise performing calculations with the neural network using the data set, wherein the calculations use a loss function and provide an aleatoric and epistemic uncertainty that is correlated to a value and displaying results of the calculations performed.

[0054] In another embodiment, the neural network incorporates a machine learning model.

[0055] In another embodiment, the loss function is of a form of:

[0056] $L\left(x,y,\theta \right)=\frac{1}{N}{\sum }_{i=1}^N\frac{1}{{\sigma }_i^2}{f_{\theta }\left(x_i\right)-y_i}^2+\log \left[{\sigma }_i^2\right]+{\alpha \left({\sigma }_i-❘f_{\theta }\left(x_i\right)-y_i❘\right)}^2.$

[0057] The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.

[0058] While embodiments have been described herein, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments are envisioned that do not depart from the inventive scope. Accordingly, the scope of the present claims or any subsequent claims shall not be unduly limited by the description of the embodiments described herein.