Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control

Rene Carmona, Mathieu Lauriere

TL;DR

This work analyzes the relationship between discrete-time mixed policies used in RL and continuous-time relaxed controls from stochastic control, focusing on when RL-style randomization converges to the relaxed framework and where the analogy breaks down. It proves a strong convergence result for drift-only control: as the time mesh $T/N$ shrinks, the discrete-time mixed-policy dynamics converge in trajectory to the continuous-time relaxed model with rate $\mathcal{O}(N^{-1/2})$, and it shows that cost functionals converge as well. When diffusion (volatility) is allowed to depend on the control, however, this convergence can become sublinear and the naive randomization fails to reproduce the relaxed model, motivating a martingale-problem based reformulation. The paper also provides numerical evidence supporting the convergence rate in the drift-controlled case and discusses how relaxing the dynamics via the martingale framework provides a principled way to handle controlled diffusion, with practical implications for RL-inspired algorithms and continuous-time control theory.

Abstract

Reinforcement learning (RL) is currently one of the most prominent methods for optimizing dynamical systems, with breakthrough results across various fields. The framework is based on the concept of a Markov decision process (MDP), leading to a discrete-time optimal control problem. In the RL literature, such problems are typically formulated and solved using mixed policies, from which random actions are sampled at each time step. Recently, part of the optimal control community has begun investigating continuous-time versions of RL algorithms, replacing MDPs with continuous-time stochastic processes governed by relaxed controls, and asserting a full analogy between the two formulations. In this work, we examine the limitations of this analogy and rigorously establish a connection between the two problems in the case where only the drift term of the continuous-time model is controlled. We prove strong convergence of the RL implementation of mixed strategies as the time discretization mesh tends to zero. We also discuss the technical challenges posed by the possible presence of control in the diffusion component of the state.

Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control

TL;DR

This work analyzes the relationship between discrete-time mixed policies used in RL and continuous-time relaxed controls from stochastic control, focusing on when RL-style randomization converges to the relaxed framework and where the analogy breaks down. It proves a strong convergence result for drift-only control: as the time mesh shrinks, the discrete-time mixed-policy dynamics converge in trajectory to the continuous-time relaxed model with rate , and it shows that cost functionals converge as well. When diffusion (volatility) is allowed to depend on the control, however, this convergence can become sublinear and the naive randomization fails to reproduce the relaxed model, motivating a martingale-problem based reformulation. The paper also provides numerical evidence supporting the convergence rate in the drift-controlled case and discusses how relaxing the dynamics via the martingale framework provides a principled way to handle controlled diffusion, with practical implications for RL-inspired algorithms and continuous-time control theory.

Abstract

Reinforcement learning (RL) is currently one of the most prominent methods for optimizing dynamical systems, with breakthrough results across various fields. The framework is based on the concept of a Markov decision process (MDP), leading to a discrete-time optimal control problem. In the RL literature, such problems are typically formulated and solved using mixed policies, from which random actions are sampled at each time step. Recently, part of the optimal control community has begun investigating continuous-time versions of RL algorithms, replacing MDPs with continuous-time stochastic processes governed by relaxed controls, and asserting a full analogy between the two formulations. In this work, we examine the limitations of this analogy and rigorously establish a connection between the two problems in the case where only the drift term of the continuous-time model is controlled. We prove strong convergence of the RL implementation of mixed strategies as the time discretization mesh tends to zero. We also discuss the technical challenges posed by the possible presence of control in the diffusion component of the state.
Paper Structure (13 sections, 3 theorems, 55 equations, 4 figures)

This paper contains 13 sections, 3 theorems, 55 equations, 4 figures.

Table of Contents

  1. Introduction
  2. The State of Affairs with Pure Controls
  3. Optimization of MDPs with Pure Control Policies
  4. Optimal Control in Continuous Time
  5. Compactifications in the Search for Optimal Controls
  6. Controlling MDPs with Mixed Controls
  7. Continuous Time Control with Relaxed Controls
  8. First Attempt at the Randomization of a Continuous Time Model
  9. What Could a Randomization with Mixed Controls Look Like?
  10. Warm-Up with a Simple Example
  11. Continuous Time Relaxed Models as Limits of Mixed MDP Models
  12. Numerical experiments
  13. What is so Different when the Volatility is Controlled?

Key Result

Proposition 4

For $\;\Omega=C([0,T]; {\mathbb R} ^d)$ equipped with its Borel $\sigma$-field ${\mathcal{F}}$ and Wiener measure ${\mathbb P}$, there exists a Fubini extension $([0,T]\times\Omega,{\mathcal{B}}_{[0,T]}\boxtimes{\mathcal{F}},\lambda\boxtimes {\mathbb P} )$ with the Lebesgue measure $\lambda$ such th

Figures (4)

  • Figure 1: Setting 1. Convergence rate.
  • Figure 2: Setting 1. $4$ samples of couples of trajectories with $N=100$ (left) and $N=1000$ (right).
  • Figure 3: Setting 2. Left: Convergence rate. Right: 4 sample trajectories with $N=1000$.
  • Figure 4: Setting 3. Comparison of convergence rate for several values of $c^a_\sigma$ using formula \ref{['fo:naive_randomization']} (left) and formula \ref{['fo:last_attempt']} (right).

Theorems & Definitions (9)

  • Remark 1
  • Remark 2
  • Definition 3
  • Proposition 4
  • Remark 5: Important Remark
  • Theorem 6
  • Remark 7
  • proof : Proof of Theorem \ref{['thm:main-X']}
  • Corollary 8