A Two-fold Randomization Framework for Impulse Control Problems
Haoyang Cao, Yuchao Dong, Zhouhao Yang
TL;DR
The paper introduces a two-fold entropy-regularized randomization framework for impulse control, coupling a randomized nonlocal jump operator M^λ and a randomized stopping operator T^λ to form a fixed-point problem ψ^{λ}=T^{λ1}[M^{λ2}ψ^{λ}]. It derives a semi-linear HJB equation and proves a verification theorem guaranteeing uniqueness, with an iterative scheme that establishes existence and C^{2,α}_{loc} regularity, and shows convergence to the classical impulse control value as λ→0. The authors develop an offline reinforcement learning algorithm based on this framework, proving geometric convergence of policy improvement and providing a TD-based model-free implementation; numerical experiments on a linear model validate convergence toward the classical solution and illustrate the exploration-exploitation tradeoff via the volatility parameter σ. The framework thus offers a principled, learnable approximation for classical impulse control problems and lays the groundwork for learning algorithms in high-dimensional settings.
Abstract
We propose and analyze a randomization scheme for a general class of impulse control problems. The solution to this randomized problem is characterized as the fixed point of a compound operator which consists of a regularized nonlocal operator and a regularized stopping operator. This approach allows us to derive a semi-linear Hamilton-Jacobi-Bellman (HJB) equation. Through an equivalent randomization scheme with a Poisson compound measure, we establish a verification theorem that implies the uniqueness of the solution. Via an iterative approach, we prove the existence of the solution. The existence-and-uniqueness result ensures the randomized problem is well-defined. We then demonstrate that our randomized impulse control problem converges to its classical counterpart as the randomization parameter $\pmb λ$ vanishes. This convergence, combined with the value function's $C^{2,α}_{loc}$ regularity, confirms our framework provides a robust approximation and a foundation for developing learning algorithms. Under this framework, we propose an offline reinforcement learning (RL) algorithm. Its policy improvement step is naturally derived from the iterative approach from the existence proof, which enjoys a geometric convergence rate. We implement a model-free version of the algorithm and numerically demonstrate its effectiveness using a widely-studied example. The results show that our RL algorithm can learn the randomized solution, which accurately approximates its classical counterpart. A sensitivity analysis with respect to the volatility parameter $σ$ in the state process effectively demonstrates the exploration-exploitation tradeoff.