Settling the Sample Complexity of Model-Based Offline Reinforcement Learning

TL;DR

The paper addresses offline RL under distribution shift and limited data coverage, focusing on tabular MDPs. It introduces VI-LCB, a pessimistic model-based algorithm using Bernstein-style penalties to learn from fixed datasets without additional exploration. The authors prove minimax-optimal sample complexities for both discounted infinite-horizon and episodic finite-horizon MDPs, with upper bounds scaling as $\widetilde{O}(\frac{S C^{\star}_{\mathsf{clipped}}}{(1-\gamma)^3 \varepsilon^2})$ and $\widetilde{O}(\frac{H^4 S C^{\star}_{\mathsf{clipped}}}{\varepsilon^2})$ respectively, and matching lower bounds up to log factors. They also show that no burn-in is needed to achieve optimal accuracy in the full $\varepsilon$-range, and provide minimax lower bounds, instance-dependent analyses, and numerical experiments on a gambler’s problem to validate the theory. The work demonstrates that simple, pessimistic, model-based planning can attain data-efficient offline RL without variance-reduction tricks, with practical implications for settings where data are scarce and exploration is infeasible.

Abstract

This paper is concerned with offline reinforcement learning (RL), which learns using pre-collected data without further exploration. Effective offline RL would be able to accommodate distribution shift and limited data coverage. However, prior algorithms or analyses either suffer from suboptimal sample complexities or incur high burn-in cost to reach sample optimality, thus posing an impediment to efficient offline RL in sample-starved applications. We demonstrate that the model-based (or "plug-in") approach achieves minimax-optimal sample complexity without burn-in cost for tabular Markov decision processes (MDPs). Concretely, consider a finite-horizon (resp. $γ$-discounted infinite-horizon) MDP with $S$ states and horizon $H$ (resp. effective horizon $\frac{1}{1-γ}$), and suppose the distribution shift of data is reflected by some single-policy clipped concentrability coefficient $C^{\star}_{\text{clipped}}$. We prove that model-based offline RL yields $\varepsilon$-accuracy with a sample complexity of \[ \begin{cases} \frac{H^{4}SC_{\text{clipped}}^{\star}}{\varepsilon^{2}} & (\text{finite-horizon MDPs}) \frac{SC_{\text{clipped}}^{\star}}{(1-γ)^{3}\varepsilon^{2}} & (\text{infinite-horizon MDPs}) \end{cases} \] up to log factor, which is minimax optimal for the entire $\varepsilon$-range. The proposed algorithms are "pessimistic" variants of value iteration with Bernstein-style penalties, and do not require sophisticated variance reduction. Our analysis framework is established upon delicate leave-one-out decoupling arguments in conjunction with careful self-bounding techniques tailored to MDPs.

Figures (5)

• Figure 1: An illustration of prior works, where (a) is about discounted infinite-horizon MDPs and (b) is about finite-horizon MDPs. To facilitate comparisons, we replace $C^{\star}_{\mathsf{clipped}}$ with $C^{\star}$ in our results when drawing the plots given that $C^{\star}_{\mathsf{clipped}}\leq C^{\star}$. The shaded regions indicate the state-of-the-art achievability results. Our work manages to close the gaps between the prior achievable regions and the minimax lower bounds.
• Figure 2: Offline value iteration with LCB (VI-LCB) for discounted infinite-horizon MDPs
• Figure 3: Offline value iteration with LCB (VI-LCB) for finite-horizon MDPs.
• Figure 4: Subsampled VI-LCB for episodic finite-horizon MDPs
• Figure 5: The performances of the proposed method VI-LCB and the baseline value iteration (VI) in the gambler's problem. It shows that VI-LCB outperforms VI by taking advantage of the pessimism principle and achieves approximately $1/\sqrt{N}$ sample complexity dependency w.r.t. the sample size $N$.

Theorems & Definitions (33)

• Definition 1: Single-policy concentrability for infinite-horizon MDPs
• Definition 2: Single-policy clipped concentrability for infinite-horizon MDPs
• Remark 1
• Lemma 1
• Lemma 2
• Theorem 1
• Remark 2
• Theorem 2
• Remark 3
• Definition 3: Single-policy concentrability for finite-horizon MDPs