Automated Importance Sampling via Optimal Control for Stochastic Reaction Networks: A Markovian Projection-based Approach

TL;DR

The paper addresses efficient estimation of rare-event probabilities in stochastic reaction networks by marrying an IS path-change strategy with an explicit tau-leap scheme and introducing Markovian projection to overcome dimensionality. By formulating IS controls through a stochastic optimal control framework and deriving a continuous-time HJB system, the method yields time-continuous IS controls; solving these on a reduced MP-SRN makes high-dimensional problems tractable. The authors show that projecting to a low-dimensional surrogate preserves marginals, enabling near-optimal IS controls that, when mapped back to the full system, produce unbiased estimates with dramatically reduced variance. Numerical experiments on enzyme kinetics and transcription models demonstrate orders-of-magnitude variance reductions and substantial computational savings, highlighting the practical impact for rare-event SRN simulations.

Abstract

We propose a novel alternative approach to our previous work (Ben Hammouda et al., 2023) to improve the efficiency of Monte Carlo (MC) estimators for rare event probabilities for stochastic reaction networks (SRNs). In the same spirit of (Ben Hammouda et al., 2023), an efficient path-dependent measure change is derived based on a connection between determining optimal importance sampling (IS) parameters within a class of probability measures and a stochastic optimal control formulation, corresponding to solving a variance minimization problem. In this work, we propose a novel approach to address the encountered curse of dimensionality by mapping the problem to a significantly lower-dimensional space via a Markovian projection (MP) idea. The output of this model reduction technique is a low-dimensional SRN (potentially even one dimensional) that preserves the marginal distribution of the original high-dimensional SRN system. The dynamics of the projected process are obtained by solving a related optimization problem via a discrete $L^2$ regression. By solving the resulting projected Hamilton-Jacobi-Bellman (HJB) equations for the reduced-dimensional SRN, we obtain projected IS parameters, which are then mapped back to the original full-dimensional SRN system, resulting in an efficient IS-MC estimator for rare events probabilities of the full-dimensional SRN. Our analysis and numerical experiments reveal that the proposed MP-HJB-IS approach substantially reduces the MC estimator variance, resulting in a lower computational complexity in the rare event regime than standard MC estimators.

Figures (7)

• Figure 1.1: Schematic diagram of the learning-based approach in ben2023learning.
• Figure 1.2: Schematic diagram of the MP-IS approach.
• Figure 4.1: Schematic diagram MP-IS-MC. The costs of the operations in the first line (blue boxes) are off-line.
• Figure 5.1: Sample paths of TL and MP-TL for a step size of $\Delta t=2^{-8}$.
• Figure 5.2: Relative occurrences of states at final time $T$ in $M_{test}=10^4$ sample paths, comparing the TL estimate of $P(\mathbf{X}(t))\mid_{\{\mathbf{X}_0=\mathbf{x}_0\}}$ and the MP estimate of $\overline{\boldsymbol{S}}(T)\mid_{\{\mathbf{X}_0=\mathbf{x}_0\}}$. We set the step size to $\Delta t=2^{-8}$ for the sample paths. The MP is based on $M=10^4$ TL paths with a step size of $\Delta t=2^{-8}$ and uses the orthogonal basis of polynomials described in Remark \ref{['rem:orth_basis']}, where $\Lambda = \{0,1,2\} \times \{0,1,2\}$.

Theorems & Definitions (18)

• Definition 2.1: Value function
• Theorem 2.2: Dynamic programming for IS parameters ben2023learning
• Corollary 2.3: HJB equations for IS parameters
• proof
• Remark 2.4: Continuous-time IS controls
• Theorem 3.1: MP for SRNs
• proof
• Remark 3.2: Sufficient conditions for assumption \ref{['eq:assumpnoninfty']}
• Remark 3.3: Orthonormal basis approach via empirical inner product
• Remark 3.4: Propensity functions