Learning-Based Importance Sampling via Stochastic Optimal Control for Stochastic Reaction Networks

TL;DR

The analysis and numerical experiments verify that the proposed learning-based IS approach substantially reduces MC estimator variance, resulting in a lower computational complexity in the rare event regime, compared with standard tau-leap MC estimators.

Abstract

We explore efficient estimation of statistical quantities, particularly rare event probabilities, for stochastic reaction networks. Consequently, we propose an importance sampling (IS) approach to improve the Monte Carlo (MC) estimator efficiency based on an approximate tau-leap scheme. The crucial step in the IS framework is choosing an appropriate change of probability measure to achieve substantial variance reduction. This task is typically challenging and often requires insights into the underlying problem. Therefore, we propose an automated approach to obtain a highly efficient path-dependent measure change based on an original connection in the stochastic reaction network context between finding optimal IS parameters within a class of probability measures and a stochastic optimal control formulation. Optimal IS parameters are obtained by solving a variance minimization problem. First, we derive an associated dynamic programming equation. Analytically solving this backward equation is challenging, hence we propose an approximate dynamic programming formulation to find near-optimal control parameters. To mitigate the curse of dimensionality, we propose a learning-based method to approximate the value function using a neural network, where the parameters are determined via a stochastic optimization algorithm. Our analysis and numerical experiments verify that the proposed learning-based IS approach substantially reduces MC estimator variance, resulting in a lower computational complexity in the rare event regime, compared with standard tau-leap MC estimators.

Figures (4)

• Figure 3.1: Example \ref{['exp:decay']} with step size $\Delta t_{pl}=\Delta t_f=1/2^4$ for the proposed IS-MC estimator: (a) sample mean; (b) squared coefficient of variation; (c) parameters; (d) kurtosis for each optimizer step. Adam optimizer gradient, sample variance, and kurtosis were estimated using $M_0=10^4$ samples. The reference value for the standard MC-TL approach was derived from a single run with $M=10^6$ samples and with step size $\Delta t=1/2^4$.
• Figure 3.2: Example \ref{['exp:mm']} with step size $\Delta t_{ pl}=\Delta t_f=1/2^4$ for the proposed IS-MC estimator: (a) sample mean; (b) squared coefficient of variation; (c) parameters; (d) kurtosis for each optimizer step. The gradient for the Adam optimization, the sample variance, and the kurtosis were estimated using $M_0=10^5$ samples. Standard MC-TL with step size $\Delta t=1/2^4$ and $M=10^7$ samples was used for comparison.
• Figure 3.3: Example \ref{['exp:6d']} with step size $\Delta t_{ pl}=\Delta t_f=1/2^4$ for the proposed IS-MC estimators: (a) sample mean; (b) squared coefficient of variation; (c) parameters; (d) kurtosis for each optimizer step. The gradient for Adam optimization, the sample variance, and the kurtosis were estimated using $M_0=10^5$ samples. Standard MC-TL with $M=10^6$ samples and step size $\Delta t=1/2^4$ was used for comparison.
• Figure 3.4: Example \ref{['exp:mm']}, parameters $\boldsymbol{\beta}^{space}$ and $\beta^{time}$ learned with $\Delta t_{pl} =1/2^4$ (see final optimizer step in Figure \ref{['fig:4d']}) and applied to forward runs with different $\Delta t_f$ values. The squared coefficient of variation was estimated with $M=10^6$ sample paths. The standard MC-TL approach is used as reference (dashed red line).

Theorems & Definitions (14)

• Definition 2.1: Second moment for the proposed importance sampling estimator
• Remark 2.2: Structure of the cost function
• Definition 2.3: Value function
• Theorem 2.4: Dynamic programming for importance sampling parameters
• Remark 2.6: Assumption \ref{['eq:condition of standard case']}
• Remark 2.7: Computational cost for dynamic programming
• Remark 2.8: Choosing the ansatz function
• Lemma 2.9
• Remark 2.10
• Example 3.1: Pure decay