Improving Offline RL by Blending Heuristics
Sinong Geng, Aldo Pacchiano, Andrey Kolobov, Ching-An Cheng
TL;DR
Offline RL often suffers from instability due to value bootstrapping on fixed datasets. HUBL mitigates this by relabeling data with a heuristic-based component, using ${\tilde{r}}= r + \gamma \lambda h$ and ${\tilde{\gamma}}= \gamma(1-\lambda)$, where $h$ is Monte-Carlo returns and $\lambda$ is trajectory-dependent. The approach yields a bias–regret trade-off explained via a reshaped MDP and a finite-sample analysis, and empirically improves four leading bootstrapping-based offline RL methods by about 9% on 27 D4RL/Meta-World datasets, with substantial gains on unstable cases. HUBL is simple to implement as data relabeling and is compatible with many offline RL pipelines, making it a practical, scalable improvement for real-world offline decision-making tasks.
Abstract
We propose Heuristic Blending (HUBL), a simple performance-improving technique for a broad class of offline RL algorithms based on value bootstrapping. HUBL modifies the Bellman operators used in these algorithms, partially replacing the bootstrapped values with heuristic ones that are estimated with Monte-Carlo returns. For trajectories with higher returns, HUBL relies more on the heuristic values and less on bootstrapping; otherwise, it leans more heavily on bootstrapping. HUBL is very easy to combine with many existing offline RL implementations by relabeling the offline datasets with adjusted rewards and discount factors. We derive a theory that explains HUBL's effect on offline RL as reducing offline RL's complexity and thus increasing its finite-sample performance. Furthermore, we empirically demonstrate that HUBL consistently improves the policy quality of four state-of-the-art bootstrapping-based offline RL algorithms (ATAC, CQL, TD3+BC, and IQL), by 9% on average over 27 datasets of the D4RL and Meta-World benchmarks.