Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation

TL;DR

The paper addresses instance-dependent adaptivity in continuous-time reinforcement learning (CTRL) by proposing CT-MLE, a model-based algorithm that learns the marginal state density via maximum likelihood estimation rather than explicit dynamics. It introduces a randomized measurement scheme to estimate reward integrals and proves a variance-aware, instance-dependent regret bound that scales with total reward variance and measurement gaps, showing robustness to the measurement schedule when adaptation aligns with problem difficulty. The analysis leverages a continuous-time Bellman-like identity, eluder-dimension-based complexity, and bracketing numbers to obtain near-horizon-free regret guarantees without bespoke variance estimators. Empirical-style discussion and theoretical insights indicate CT-MLE can adaptively balance measurement effort with problem hardness, yielding improved sample efficiency in stochastic CTRL settings compared to baselines.

Abstract

Continuous-time reinforcement learning (CTRL) provides a natural framework for sequential decision-making in dynamic environments where interactions evolve continuously over time. While CTRL has shown growing empirical success, its ability to adapt to varying levels of problem difficulty remains poorly understood. In this work, we investigate the instance-dependent behavior of CTRL and introduce a simple, model-based algorithm built on maximum likelihood estimation (MLE) with a general function approximator. Unlike existing approaches that estimate system dynamics directly, our method estimates the state marginal density to guide learning. We establish instance-dependent performance guarantees by deriving a regret bound that scales with the total reward variance and measurement resolution. Notably, the regret becomes independent of the specific measurement strategy when the observation frequency adapts appropriately to the problem's complexity. To further improve performance, our algorithm incorporates a randomized measurement schedule that enhances sample efficiency without increasing measurement cost. These results highlight a new direction for designing CTRL algorithms that automatically adjust their learning behavior based on the underlying difficulty of the environment.

Figures (5)

• Figure 1: We depict the state trajectory $x_n(t)$ over $t\in[0,T]$ in episode $n$, with $x_n(0)=x_{\text{ini}}$ and $x_n(T)$ at the endpoints. Observation times $t_n^k$ are marked by black dots. Each measurement interval $\Delta_{n,k}=t_n^{k+1}-t_n^k$ is overlaid by a brown rectangle of width $\Delta_{n,k}$ and height proportional to $\Delta_{n,k}$, so that its area encodes $\Delta_{n,k}^2$ in our regret bound. The green shading illustrates the total variance $\mathrm{Var}^{u_n}$. Proper measurement gap should be selected in accordance with policy variance $\mathrm{Var}^{u_n}$ to achieve an optimal instance‐dependent performance.
• Figure 2: Performance comparison of Algorithm \ref{['alg:imp-friendly']}, ENODE yildiz2021continuous, and SAC-TaCoS treven2024sense across three environments with noise $\sigma=2.0$ ($\pm1$ standard error).
• Figure 3: Optimal measurement gap $\Delta_\sigma$ and mean episodes to success ($\pm1$ standard error) under varying environment stochasticity $\sigma$. Results averaged over 10 random seeds for Pendulum and 5 seeds for Cart Pole and Acrobot environments.
• Figure 4: Ablation on Neural Network Width in the Dynamics Model
• Figure 5: Reward error convergence follows the theoretical $1/\sqrt{N}$ rate (Pendulum, $\sigma=2.0$).

Theorems & Definitions (32)

• Remark 5.2
• Definition 5.3
• Proposition 5.4
• Definition 5.5
• Definition 5.6
• Remark 5.7
• Remark 5.8
• Proposition 5.9
• Proposition 5.10
• Theorem 5.11