Sample Efficient Active Algorithms for Offline Reinforcement Learning

TL;DR

This work addresses the data-efficiency challenge in offline reinforcement learning by introducing a GP-based Active Offline RL framework that selectively queries online transitions in regions of high epistemic uncertainty. It develops a PAC-style analysis showing that ${ ilde{O}}(1/ ext{ε}^2)$ active samples suffice to achieve $ ext{ε}$-optimality, with a horizon dependence improved from $(1- ext{γ})^{-4}$ to $(1- ext{γ})^{-2}$ relative to purely offline methods, via GP concentration and information-gain arguments combined with Bellman contraction. The key contributions include modeling epistemic uncertainty with a GP prior over the value function, deriving a universal sample-complexity bound that scales with the maximum information gain $oldsymbol{ extgamma_T}$, and validating the theory empirically on D4RL benchmarks using SVGP to manage large datasets. The results demonstrate significant data efficiency gains from uncertainty-guided exploration and provide a principled theoretical foundation linking Bayesian nonparametric regression to reinforced learning in a hybrid offline-online setting. This work offers a practical and theoretically grounded pathway to safer, more efficient policy learning when online interactions are costly or limited.

Abstract

Offline reinforcement learning (RL) enables policy learning from static data but often suffers from poor coverage of the state-action space and distributional shift problems. This problem can be addressed by allowing limited online interactions to selectively refine uncertain regions of the learned value function, which is referred to as Active Reinforcement Learning (ActiveRL). While there has been good empirical success, no theoretical analysis is available in the literature. We fill this gap by developing a rigorous sample-complexity analysis of ActiveRL through the lens of Gaussian Process (GP) uncertainty modeling. In this respect, we propose an algorithm and using GP concentration inequalities and information-gain bounds, we derive high-probability guarantees showing that an $ε$-optimal policy can be learned with ${\mathcal{O}}(1/ε^2)$ active transitions, improving upon the $Ω(1/ε^2(1-γ)^4)$ rate of purely offline methods. Our results reveal that ActiveRL achieves near-optimal information efficiency, that is, guided uncertainty reduction leads to accelerated value-function convergence with minimal online data. Our analysis builds on GP concentration inequalities and information-gain bounds, bridging Bayesian nonparametric regression and reinforcement learning theories. We conduct several experiments to validate the algorithm and theoretical findings.

Figures (7)

• Figure 1: Learning performance and uncertainty reduction across different environments. Top: HalfCheetah-Random. Middle: Maze2D-Medium-Hard. Bottom: Maze2D-Large-Easy.
• Figure 2: Uncertainty-driven exploration and value estimation in maze2d-medium-hard. For each timestep $t$, the exploration map (left) and the corresponding to the value function estimate (right). Value Estimates are calculated using the full offline dataset.
• Figure 3: Uncertainty decay vs. information gain (halfcheetah-random-v2 ). The mean posterior variance decreases, while the cumulative information gain increases with active steps $T$, consistent with the GP concentration bounds $\sigma_T = O(\gamma_T / T)$.
• Figure 4: Learning dynamics in different environments. Each row corresponds to a different environment, and the columns show (a) the learning curve under active data collection, (b) the reduction of GP epistemic uncertainty over active steps, and (c) the relationship between posterior uncertainty and cumulative information gain.
• Figure 5: State visitation distributions before (top) and after (bottom) dataset pruning for four Maze2D environments: maze2d-umaze-hard, maze2d-medium-hard, maze2d-large-medium, and maze2d-large-hard. Although pruning retains only a small fraction of trajectories, it preserves coverage of task-relevant regions while removing redundant and low-utility states.

Theorems & Definitions (22)

• Theorem 4.1: Sample Complexity of \ref{['alg:gp_active_rl']}
• Corollary 4.2: Squared-Exponential Kernel
• Corollary 4.3: Comparison with Purely Offline RL
• Definition 2.1: Offline Coverage Radius
• Definition 2.2: Information Gain
• Definition 2.3: Information Gain in Gaussian Process Regression
• Lemma 2.4: Bounded Information Gain for Smooth Kernels
• Lemma 3.1: GP Concentration Bound
• Lemma 3.2: Variance–Information Gain Relationship
• Lemma 3.3: Lipschitz Propagation of Bellman Error