Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees

TL;DR

The article provides a comprehensive theory of offline reinforcement learning in large state spaces, highlighting how data coverage and function-approximation expressivity shape learnability and guarantees. It surveys three core strands: (i) value-function based methods (FQE/BRM) with Bellman-completeness and state-action coverage, (ii) pessimistic algorithms (PSPI, PEVI) that relax all-policy coverage to single-policy coverage via uncertainty quantification, and (iii) realizability-driven approaches (BVFT, LSTDQ) and MIS-based density-ratio methods for OPE and policy optimization. A unifying theme is translating Bellman errors across data distributions using coverage notions (Cπ, C^sqπ, C^avgπ) and developing oracle-efficient or computation-friendly schemes that yield non-vacuous guarantees even with offline data. The paper also links model-based offline RL, online-offline hybrids, and partial observability directions, offering a roadmap for robust, scalable offline learning in real-world large-state problems. Overall, it clarifies when and how offline RL achieves reliable policy learning under limited or biased data, and points to open problems in deep, structured, and multi-agent settings.

Abstract

This article introduces the theory of offline reinforcement learning in large state spaces, where good policies are learned from historical data without online interactions with the environment. Key concepts introduced include expressivity assumptions on function approximation (e.g., Bellman completeness vs. realizability) and data coverage (e.g., all-policy vs. single-policy coverage). A rich landscape of algorithms and results is described, depending on the assumptions one is willing to make and the sample and computational complexity guarantees one wishes to achieve. We also discuss open questions and connections to adjacent areas.

Figures (3)

• Figure 1: Figurative illustration of different expressivity assumptions on the value-function class $\mathcal{F}$. Left: Only realizability $Q^\pi \in \mathcal{F}$ is assumed, and Bellman operator ("$\to$") can generally take functions in $\mathcal{F}$ out of the class. Middle: Bellman-completeness, where $\mathcal{F}$ is closed under $\mathcal{T}^\pi$. Right: All functions, including those not in $\mathcal{F}$, have their image in $\mathcal{F}$ (Section \ref{['sec:pevi']}).
• Figure 2: Figurative illustration of different coverage assumptions, adapted from xie2022role. Left: All-policy coverage. Right: Data only covers $\pi_1$ and $\pi_2$, and pessimistic algorithms in Section \ref{['sec:pess']} can compete with the best among them.
• Figure 3: Uncertainty in Multi-armed bandits (MABs). "O" is true mean and "X" is point estimate. Left: Example where greedy w.r.t. point estimate chooses arm2 and suffers large loss. Instead, pessimism chooses arm3 with suboptimality bounded by the uncertainty of the best arm (arm1). Right: Example where return optimization chooses a different pessimistic policy (arm1) than regret minimization. Both arms have the same regret (height of double-headed arrows), and randomizing between them only incurs half of the regret.

Theorems & Definitions (11)

• Proposition 1: tsitsiklis1996feature
• Example 1: Low-rank MDP; barreto2011reinforcementbarreto2014policyjiang2017contextual
• Lemma 2: Bellman error telescoping
• Lemma 3: Error translation under coverage
• Lemma 4
• Theorem 5
• Lemma 6
• Theorem 7
• Proposition 8
• Theorem 9: Guarantee of MQL uehara2019minimax