Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation

TL;DR

The paper investigates offline reinforcement learning with function approximation and proves that realizability together with concentrability do not guarantee sample-efficient learning in large state spaces. By constructing hard MDP families that exploit planted-subset structures and strong over-coverage, the authors show information-theoretic lower bounds that require either stronger data coverage or richer representations beyond supervised-learning guarantees. They provide two main results: a lower bound under general concentrability and a stronger admissible-data bound illustrating a trade-off between sample complexity and problem parameters like the state space size and effective horizon. The findings imply a fundamental gap between online and offline RL and guide future work toward data-collection strategies or function classes that enable practical offline RL in complex environments.

Abstract

We consider the offline reinforcement learning problem, where the aim is to learn a decision making policy from logged data. Offline RL -- particularly when coupled with (value) function approximation to allow for generalization in large or continuous state spaces -- is becoming increasingly relevant in practice, because it avoids costly and time-consuming online data collection and is well suited to safety-critical domains. Existing sample complexity guarantees for offline value function approximation methods typically require both (1) distributional assumptions (i.e., good coverage) and (2) representational assumptions (i.e., ability to represent some or all $Q$-value functions) stronger than what is required for supervised learning. However, the necessity of these conditions and the fundamental limits of offline RL are not well understood in spite of decades of research. This led Chen and Jiang (2019) to conjecture that concentrability (the most standard notion of coverage) and realizability (the weakest representation condition) alone are not sufficient for sample-efficient offline RL. We resolve this conjecture in the positive by proving that in general, even if both concentrability and realizability are satisfied, any algorithm requires sample complexity polynomial in the size of the state space to learn a non-trivial policy. Our results show that sample-efficient offline reinforcement learning requires either restrictive coverage conditions or representation conditions that go beyond supervised learning, and highlight a phenomenon called over-coverage which serves as a fundamental barrier for offline value function approximation methods. A consequence of our results for reinforcement learning with linear function approximation is that the separation between online and offline RL can be arbitrarily large, even in constant dimension.

Figures (3)

• Figure 1: The MDPs in $\mathcal{M}$ are parametrized by three scalars $\alpha,\beta,w$ and a subset of states $I$. The state space consists of an initial state$\mathfrak{s}$, a large number of intermediate states$\mathcal{S}^1$, and four self-looping terminal states$\{W,X,Y,Z\}$. From the initial state $\mathfrak{s}$, action $1$ (in red) transitions to state $W$, while action $2$ (in blue) transitions to a subset of intermediate states $I\subset\mathcal{S}^1$ with equal probability. In all intermediate states and terminal states, actions 1 and 2 have the same effect, with transitions denoted in black. Among the intermediate states, $I \subset \mathcal{S}^1$ (the gray ones) are the planted states which transition with probability $\alpha$ to state $X$ and $1-\alpha$ to state $Y$, and the remaining $\mathcal{S}^1 \setminus I$ (the striped ones) are the unplanted states which transition with probability $\beta$ to $Z$ and $(1-\beta)$ to $Y$. There are combinatorially many choices for $I$. Only terminal states can generate non-zero rewards: the rewards of the states $W$, $X$, $Y$ and $Z$ are $w$, $1$, $0$ and $\alpha/\beta$, respectively.
• Figure 2: Illustration of the MDP family used to prove \ref{['thm:admissible']}, with $L = 3$ layers of the planted subset structure (note that in general, we take $L>3$). The rewards of the states $W$, $X$, $Y$ and $Z$ are $w$, $1$, $0$ and $\alpha/(1-L\alpha)$, respectively, where $w$ and $\alpha$ are parameters of the MDP family.
• Figure 3: Illustration of the average MDP with $L=3$.

Theorems & Definitions (20)

• Theorem 1: Main theorem
• Theorem 2: Lower bound for admissible data
• Proposition 1
• Lemma 1
• Lemma 2
• Lemma 3
• Lemma 4
• Lemma 5
• Lemma 6
• Lemma 7: Monotonicity of $g_{\theta,\alpha,\beta}$