Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning

TL;DR

This paper tackles distributional shift in offline RL by learning reliable uncertainty over Q-values with minimal ensembles. It introduces Diverse Randomized Value Functions (DRVF), which combine Bayesian last-layer neural networks with ensemble methods and a repulsive regularization to approximate the Q-posterior and produce a provably pessimistic LCB penalty, especially under linear MDP assumptions. The approach yields competitive or superior performance on D4RL benchmarks with markedly better parametric efficiency, and demonstrates robust uncertainty quantification that aligns higher uncertainty with OOD actions. Theoretical results connect the DRVF posterior sampling to efficient pessimism in linear settings, and empirical evidence shows DRVF's practicality for offline policy learning with reduced computational burden. Overall, DRVF offers a principled, scalable framework for uncertainty-aware offline RL that mitigates extrapolation errors while using far fewer ensembles than prior uncertainty-based methods.

Abstract

Offline Reinforcement Learning (RL) faces distributional shift and unreliable value estimation, especially for out-of-distribution (OOD) actions. To address this, existing uncertainty-based methods penalize the value function with uncertainty quantification and demand numerous ensemble networks, posing computational challenges and suboptimal outcomes. In this paper, we introduce a novel strategy employing diverse randomized value functions to estimate the posterior distribution of $Q$-values. It provides robust uncertainty quantification and estimates lower confidence bounds (LCB) of $Q$-values. By applying moderate value penalties for OOD actions, our method fosters a provably pessimistic approach. We also emphasize on diversity within randomized value functions and enhance efficiency by introducing a diversity regularization method, reducing the requisite number of networks. These modules lead to reliable value estimation and efficient policy learning from offline data. Theoretical analysis shows that our method recovers the provably efficient LCB-penalty under linear MDP assumptions. Extensive empirical results also demonstrate that our proposed method significantly outperforms baseline methods in terms of performance and parametric efficiency.

Figures (10)

• Figure 1: The architecture of DRVF, where we utilize approximate Bayesian inference in the last layer of the critic network. We perform OOD sampling to obtain $(s,a^{\rm ood})$ pairs based on $(s,a)\sim {\mathcal{D}}_m$. We input $(s,a)$ and $(s,a^{\rm ood})$ pairs to obtain the $Q$-samples and the repulsive term, respectively. Then $(s,a)$ is used for the pessimistic update of the $Q$-values, and $(s,a^{\rm ood})$ provides regularization.
• Figure 2: Visualization for the intuition behind the repulsive regularization term. Blank dots represent data points, while colorful lines denote possible estimates of the Q-function. The shaded area denotes the uncertainty of ensemble predictions. In panel (a), uncertainty quantification and estimates are obtained from a large number of ensembles, which provides reliable uncertainty estimation. In panel (b), the uncertainty and possible estimates come from a small number of ensembles, which fails to properly account for the uncertainty. In DRVF, we explicitly maximize the standard deviation of the samples from the ensemble BNNs to obtain reliable uncertainty quantification with fewer ensembles.
• Figure 3: Uncertainty estimation of the in-distribution samples (white) and OOD samples (orange). The results (a-c) are obtained from DRVF, while the results (d-f) are estimated by PBRL. The brighter area in the contour indicates higher uncertainty, while the darker area indicates lower uncertainty. DRVF demonstrates a more favorable alignment of the brighter areas with the OOD samples, suggesting an enhanced ability to quantify uncertainty.
• Figure 4: Minimum number of $Q$-ensembles ($M$) required to obtain the performance in Table \ref{['table1']}. DRVF needs much fewer ensembles than EDAC in most cases and reduces the required parameters.
• Figure 5: Aggregate metrics on D4RL with 95% CIs based on Gym Mujoco tasks and 5 random seeds for each task. DRVF shows a higher median, mean, and IQM, and a lower optimality gap than other methods.

Theorems & Definitions (16)

• Theorem 1
• Definition 1: $\xi$-Uncertainty Quantifier pevi-2021
• Proposition 1
• Corollary 1: Suboptimality Gap pevi-2021
• Proposition 2: Informal
• Theorem : Theorem 1 restate
• proof
• Definition : Definition 1 restate
• Proposition : Proposition 1 restate
• proof