Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces

TL;DR

This paper introduces a stability-based framework for reinforcement learning in continuous state-action spaces and proves fast convergence rates for both offline and online settings by linking value sub-optimality to squared Bellman residuals. Central to the approach are two stability properties—Bellman stability and occupation-measure stability—that control how perturbations in value functions and policies affect downstream updates, enabling $O(1/n)$ offline rates and $O( ext{log }T)$ online regret under suitable conditions. The authors develop a concrete, curvature-driven analysis for linear function approximation, derive fast-rate results for FQI with ridge penalties, and discuss how these results inform the roles of pessimism and optimism, as well as connections to transfer learning under covariate shift. These insights offer a principled perspective on when pessimistic/optimistic strategies are necessary and demonstrate substantial practical implications for data-efficient RL in continuous domains.

Abstract

We introduce a novel framework for analyzing reinforcement learning (RL) in continuous state-action spaces, and use it to prove fast rates of convergence in both off-line and on-line settings. Our analysis highlights two key stability properties, relating to how changes in value functions and/or policies affect the Bellman operator and occupation measures. We argue that these properties are satisfied in many continuous state-action Markov decision processes, and demonstrate how they arise naturally when using linear function approximation methods. Our analysis offers fresh perspectives on the roles of pessimism and optimism in off-line and on-line RL, and highlights the connection between off-line RL and transfer learning.

Figures (2)

• Figure 1: Illustration of the "fast rate" phenomenon for fitted $Q$-iteration (FQI) applied to the Mountain Car problem. (a) The Mountain Car problem is a canonical continuous state-action space control problem, in which the goal is to drive the car to the flag. See \ref{['sec:mountain_car']} for further details. (b) We used off-line FQI with linear function approximation to learn approximately optimal policies $\widehat{\pi}_{n}$ over a range of sample sizes $n$. Log-log plot of the value sub-optimality $J(\pi^{\star}_{})- J(\widehat{\pi}_{n})$ over sample sizes $n \in \{ \lfloor e^{k} \rfloor \bigm| k = 10.5, 10.75, 11, \ldots, 13 \} = \{ 36315 , 46630, 59874, \ldots, 442413 \}$. In the plot, each red point represents the average value sub-optimality $J(\pi^{\star}_{})- J(\widehat{\pi}_{n})$ estimated from $T = 80$ Monte Carlo trials. The shaded area represents twice the standard errors. The blue dashed line represents the least-squares fit to the last $6$ data points. This regression leads to the $95\%$ confidence interval $(-1.084, -0.905)$ for the underlying slope, indicative of a decay rate much faster than the typical $-0.5$ "slow rate".
• Figure 2: An example with feature mapping $\boldsymbol{\phi}$ defined in $\mathds{R}^2$. (a) The relation between $\boldsymbol{\phi} - \boldsymbol{\phi}^{\star}$ and ${\boldsymbol{w}} - {\boldsymbol{w}}^{\star}$. The feature vectors $\boldsymbol{\phi}^{\star} \equiv \boldsymbol{\phi}(s, \pi^{\star}_{}(s))$ and $\boldsymbol{\phi} \equiv \boldsymbol{\phi}(s, \pi_{}(s))$ at the greedy policies $\pi^{\star}_{}$ and $\pi_{}$ are marked by stars. The figure shows that the Euclidean norm of the deviation $\boldsymbol{\phi} - \boldsymbol{\phi}^{\star}$ is approximately $\varrho \; \angle({\boldsymbol{w}}^{\star}, {\boldsymbol{w}})$. Furthermore, when measured along the direction of ${\boldsymbol{w}}$, the deviation $\Pi_{{\boldsymbol{w}}}(\boldsymbol{\phi} - \boldsymbol{\phi}^{\star})$ is rather small and, in fact, is of second order with respect to the angle $\angle({\boldsymbol{w}}^{\star}, {\boldsymbol{w}})$. (b) The relation between the difference in vectors ${\boldsymbol{w}} - {\boldsymbol{w}}^{\star}$ and the angle $\angle( {\boldsymbol{w}}^{\star}, {\boldsymbol{w}})$. A key observation is that $\angle({\boldsymbol{w}}^{\star}, {\boldsymbol{w}}) \leq \arcsin \{\norm{{\boldsymbol{w}} - {\boldsymbol{w}}^{\star}}_2 / \norm{{\boldsymbol{w}}^{\star}}_2\}$.

Theorems & Definitions (11)

• Theorem 1
• example 1: An illustration of curvature property
• Proposition 1
• Lemma 1
• Corollary 1: Fast rates for ridge-based FQI
• Corollary 2
• Lemma 2
• Lemma 3
• Lemma 4
• Proposition 2