Continuous-time reinforcement learning for optimal switching over multiple regimes

TL;DR

This work develops a continuous-time RL framework for optimal switching across multiple regimes using entropy-regularized exploration driven by a CTMC generator. It establishes the well-posedness of the exploratory HJB system, proves policy improvement with a convergent policy-iteration scheme, and shows convergence to the classical switching problem as the regularization vanishes. A martingale-based policy evaluation RL algorithm is derived and analyzed with explicit error bounds, and neural-network-based experiments validate the approach on two regime-switching problems. The results highlight a principled link between exploratory continuous-time control and classical regime-switching, with practical algorithms for learning switching strategies from data.

Abstract

This paper studies the continuous-time reinforcement learning (RL) for optimal switching problems across multiple regimes. We consider a type of exploratory formulation under entropy regularization where the agent randomizes both the timing of switches and the selection of regimes through the generator matrix of an associated continuous-time finite-state Markov chain. We establish the well-posedness of the associated system of Hamilton-Jacobi-Bellman (HJB) equations and provide a characterization of the optimal policy. The policy improvement and the convergence of the policy iterations are rigorously established by analyzing the system of equations. We also show the convergence of the value function in the exploratory formulation towards the value function in the classical formulation as the temperature parameter vanishes. Finally, a reinforcement learning algorithm is devised and implemented by invoking the policy evaluation based on the martingale characterization. Our numerical examples with the aid of neural networks illustrate the effectiveness of the proposed RL algorithm.

Figures (5)

• Figure 1: (a): Convergence of the training loss for the bounded regulator problem with $\lambda=0.2$. (b): Learned value functions and switching probabilities at $t=0.5$ for $\lambda=0.2$.
• Figure 2: Training convergence for different temperature parameters $\lambda$.
• Figure 3: Evolution of the switching probability from regime 0 to 1 as $\lambda$ decreases.
• Figure 4: The training loss for the put option selection problem.
• Figure 5: The optimal asset allocation policy at $t=0.5$ as a function of stock prices $S^A$ and $S^B$.

Theorems & Definitions (21)

• Definition 2.1
• Lemma 2.3: Comparison Principle
• Theorem 2.4
• proof
• Lemma 3.2
• proof
• Proposition 3.3: Verification Theorem
• proof
• Proposition 4.1
• proof