Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Framework for Strategic Discovery of Credible Neural Network Surrogate Models under Uncertainty

Pratyush Kumar Singh, Kathryn A. Farrell-Maupin, Danial Faghihi

TL;DR

The paper tackles the challenge of building credible neural network surrogates for high-fidelity physics simulations under uncertainty. It introduces OPAL-surrogate, a hierarchical Bayesian framework that searches the space of BayesNN models, uses model evidence and validation tests, and adaptively adjusts model complexity through a sparsification-driven process. By coupling Laplace-based posterior approximations with Kronecker-factored Hessians, it achieves scalable credible inference and model discovery across datasets spanning porous elasticity and turbulent combustion. The approach demonstrates improved predictive reliability and forward uncertainty quantification, with implications for robust decision-making in complex multiphysics contexts.

Abstract

The widespread integration of deep neural networks in developing data-driven surrogate models for high-fidelity simulations of complex physical systems highlights the critical necessity for robust uncertainty quantification techniques and credibility assessment methodologies, ensuring the reliable deployment of surrogate models in consequential decision-making. This study presents the Occam Plausibility Algorithm for surrogate models (OPAL-surrogate), providing a systematic framework to uncover predictive neural network-based surrogate models within the large space of potential models, including various neural network classes and choices of architecture and hyperparameters. The framework is grounded in hierarchical Bayesian inferences and employs model validation tests to evaluate the credibility and prediction reliability of the surrogate models under uncertainty. Leveraging these principles, OPAL-surrogate introduces a systematic and efficient strategy for balancing the trade-off between model complexity, accuracy, and prediction uncertainty. The effectiveness of OPAL-surrogate is demonstrated through two modeling problems, including the deformation of porous materials for building insulation and turbulent combustion flow for the ablation of solid fuels within hybrid rocket motors.

A Framework for Strategic Discovery of Credible Neural Network Surrogate Models under Uncertainty

TL;DR

The paper tackles the challenge of building credible neural network surrogates for high-fidelity physics simulations under uncertainty. It introduces OPAL-surrogate, a hierarchical Bayesian framework that searches the space of BayesNN models, uses model evidence and validation tests, and adaptively adjusts model complexity through a sparsification-driven process. By coupling Laplace-based posterior approximations with Kronecker-factored Hessians, it achieves scalable credible inference and model discovery across datasets spanning porous elasticity and turbulent combustion. The approach demonstrates improved predictive reliability and forward uncertainty quantification, with implications for robust decision-making in complex multiphysics contexts.

Abstract

The widespread integration of deep neural networks in developing data-driven surrogate models for high-fidelity simulations of complex physical systems highlights the critical necessity for robust uncertainty quantification techniques and credibility assessment methodologies, ensuring the reliable deployment of surrogate models in consequential decision-making. This study presents the Occam Plausibility Algorithm for surrogate models (OPAL-surrogate), providing a systematic framework to uncover predictive neural network-based surrogate models within the large space of potential models, including various neural network classes and choices of architecture and hyperparameters. The framework is grounded in hierarchical Bayesian inferences and employs model validation tests to evaluate the credibility and prediction reliability of the surrogate models under uncertainty. Leveraging these principles, OPAL-surrogate introduces a systematic and efficient strategy for balancing the trade-off between model complexity, accuracy, and prediction uncertainty. The effectiveness of OPAL-surrogate is demonstrated through two modeling problems, including the deformation of porous materials for building insulation and turbulent combustion flow for the ablation of solid fuels within hybrid rocket motors.
Paper Structure (34 sections, 41 equations, 18 figures, 3 tables)

This paper contains 34 sections, 41 equations, 18 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Neural Networks Learning as Probabilistic Inference
  3. Bayesian solution: Laplace approximation to the posterior
  4. Scalable algorithm via Kronecker-factored Laplace approximation
  5. Hierarchical Bayesian inferences and model plausibility
  6. Bayesian neural networks model
  7. 1) Network parameters $\boldsymbol \theta \in \mathbb{R}^P$
  8. 2) Inference hyper-parameters $\boldsymbol \sigma \in \mathbb{R}^H$
  9. 3) Architecture hyper-parameters $\boldsymbol \xi \in \Xi$
  10. 4) Solution hyper-parameters $\chi \in \mathbb{R}^X$
  11. Model evidence
  12. Hierarchical Bayesian inferences
  13. Remark 1
  14. Numerical solution
  15. A Framework for Surrogate Model Discovery
  16. ...and 19 more sections

Figures (18)

  • Figure 2: Illustrations of the hierarchy of scenarios and high-fidelity data considered for the surrogate modeling of the elasticity problem: The pre-training scenario $\mathcal{S}_c$ consists of smaller domain sizes and under uniaxial loading conditions, while the training scenario $\mathcal{S}$ captures the mechanical features of the target prediction over domain sizes affordable for the high-fidelity model. The prediction scenario $\mathcal{S}_p$ involves an aerogel insulation component with a size of $L = 297.5 \mu m$, and unobservable QoI is the strain energy.
  • Figure 3: Illustrative 1D example: The log-evidence $\ln \pi_{evid} (\boldsymbol D| \boldsymbol \xi_i)$, corresponding to posterior model plausibilities $\rho_i$ under the uniform prior $\pi_{pr}(\boldsymbol \xi_i)$ assumption, for fully connected BayesNN models with varying number of layers $D$ and neurons in each layer $W$, and activation functions: (a) Tanh, (b) Leaky ReLU, (c) Sigmoid, and (d) ReLU.
  • Figure 4: Illustrative 1D example: The mean and uncertainty predictions for different fully connected BayesNN models with Tanh activation functions in Figure \ref{['fig:1d_evid_grid']}(a) compared to the training data: (a) $D=3$, $W=720$, $\ln \pi_{evid} (\boldsymbol D| \boldsymbol \xi) = -3354$ (highest posterior model plausibility); (b) $D=3$, $W=800$, $\ln \pi_{evid} (\boldsymbol D| \boldsymbol \xi) = -3468$; (c) $D=2$, $W=1450$, $\ln \pi_{evid} (\boldsymbol D| \boldsymbol \xi) = -3877$; (d) $D=1$, $W=1840$, $\ln \pi_{evid} (\boldsymbol D| \boldsymbol \xi) = -3926$.
  • Figure 5: Illustrative 1D example: Network sparsification results for the BayesNN models. The mean and uncertainty predictions for (a) the fully connected network with $D=4$, $W=1600$, and $\ln \pi_{evid} (\boldsymbol D| \boldsymbol \xi) = -3821$, and (b) the corresponding sparsified network using $TOL_{\boldsymbol\theta} = 0.025$, resulting in the elimination of 22% of the parameters and yielding $\ln \pi_{evid} (\boldsymbol D| \boldsymbol \xi) = -3610$. (c) The fully connected network with $D=6$, $W=1700$, and $\ln \pi_{evid} (\boldsymbol D| \boldsymbol \xi) = -3874$, and (d) the corresponding sparsified network using $TOL_{\boldsymbol\theta} = 0.05$, resulting in the elimination of 32% of the parameters and yielding $\ln \pi_{evid} (\boldsymbol D| \boldsymbol \xi) = -3545$.
  • Figure 6: Elasticity problem: The log-evidence (corresponding to model plausibility $\rho$) for BayesNN models with single layer ($D=1$) with different widths and activation functions.
  • ...and 13 more figures