Expert or not? assessing data quality in offline reinforcement learning

TL;DR

This work tackles the challenge of assessing offline RL data quality without interacting with the environment. It introduces Bellman-Wasserstein Distance (BWD), a value-aware optimal transport score with cost $c((s,a),(s,a'))=Q^\\beta_\\theta(s,a')-\\|a'-a\\|_2^2$ that measures how far the dataset's behavioral policy is from a random reference. BWD demonstrates strong correlation with oracle performance across D4RL MuJoCo tasks and can be used as a regularizer to improve policy returns, enabling data triage and more data-efficient training. Overall, distributional, value-aware signals like BWD offer a practical tool for offline RL data-curation and policy learning without costly extensive pretraining.

Abstract

Offline reinforcement learning (RL) learns exclusively from static datasets, without further interaction with the environment. In practice, such datasets vary widely in quality, often mixing expert, suboptimal, and even random trajectories. The choice of algorithm therefore depends on dataset fidelity. Behavior cloning can suffice on high-quality data, whereas mixed- or low-quality data typically benefits from offline RL methods that stitch useful behavior across trajectories. Yet in the wild it is difficult to assess dataset quality a priori because the data's provenance and skill composition are unknown. We address the problem of estimating offline dataset quality without training an agent. We study a spectrum of proxies from simple cumulative rewards to learned value based estimators, and introduce the Bellman Wasserstein distance (BWD), a value aware optimal transport score that measures how dissimilar a dataset's behavioral policy is from a random reference policy. BWD is computed from a behavioral critic and a state conditional OT formulation, requiring no environment interaction or full policy optimization. Across D4RL MuJoCo tasks, BWD strongly correlates with an oracle performance score that aggregates multiple offline RL algorithms, enabling efficient prediction of how well standard agents will perform on a given dataset. Beyond prediction, integrating BWD as a regularizer during policy optimization explicitly pushes the learned policy away from random behavior and improves returns. These results indicate that value aware, distributional signals such as BWD are practical tools for triaging offline RL datasets and policy optimization.

Figures (3)

• Figure 1: The impact of the hyperparameter $\alpha$ for TD3+BC fujimoto2021minimalist, which balances: RL and BC. 12 different values of the $\alpha$ parameter were tested. The average normalized scores are reported over the different training steps and three random seeds. As the value of $\alpha$ increases, the performance on high-quality datasets decreases or not changing, while the performance on lower-quality datasets improves.
• Figure 2: Pearson correlation between different metrics and the oracle score. The scores are reported over the self-crafted datasets from the different training steps and three random seeds.
• Figure 3: Results of including BWD distance into the training process of IQL. The average normalized scores are reported over the datasets from the different training steps. The performance results are averaged over the three random seeds.