Quantifying and Attributing Submodel Uncertainty in Stochastic Simulation Models and Digital Twins

TL;DR

This paper develops a framework for quantifying submodel uncertainty in stochastic simulation models and extends the framework to digital-twin settings and proposes a tree-based method that decomposes total output variability and attributes uncertainty to individual submodels in the form of importance scores.

Abstract

Stochastic simulation is widely used to study complex systems composed of various interconnected subprocesses, such as input processes, routing and control logic, optimization routines, and data-driven decision modules. In practice, these subprocesses may be inherently unknown or too computationally intensive to directly embed in the simulation model. Replacing these elements with estimated or learned approximations introduces a form of epistemic uncertainty that we refer to as submodel uncertainty. This paper investigates how submodel uncertainty affects the estimation of system performance metrics. We develop a framework for quantifying submodel uncertainty in stochastic simulation models and extend the framework to digital-twin settings, where simulation experiments are repeatedly conducted with the model initialized from observed system states. Building on approaches from input uncertainty analysis, we leverage bootstrapping and Bayesian model averaging to construct quantile-based confidence or credible intervals for key performance indicators. We propose a tree-based method that decomposes total output variability and attributes uncertainty to individual submodels in the form of importance scores. The proposed framework is model-agnostic and accommodates both parametric and nonparametric submodels under frequentist and Bayesian modeling paradigms. A synthetic numerical experiment and a more realistic digital-twin simulation of a contact center illustrate the importance of understanding how and how much individual submodels contribute to overall uncertainty.

Figures (6)

• Figure 1: Confidence intervals over 100 macro-replications when accounting for no epistemic uncertainty, only input uncertainty, or submodel uncertainty (SU).
• Figure 2: Bias and variance across combinations. The elements in each tuple correspond to $X_1$, $X_2$, $p$, and $q$, respectively; $1$ indicates that the true model is used, and $0$ indicates that the estimated model is used.
• Figure 3: Response grid plots for a $2^4$ full factorial design of the four submodels (true vs estimated). The radius of each circle indicates the relative magnitude. Blue (red) circles show positive (negative) values.
• Figure 4: Feature importance plot attributing uncertainty to each of the estimated submodels and aleatoric uncertainty.
• Figure 5: 90% confidence and credible intervals when considering submodel uncertainty and not considering submodel uncertainty for (a) frequentist and (b) Bayesian submodels.