Bellman Calibration for V-Learning in Offline Reinforcement Learning
Lars van der Laan, Nathan Kallus
TL;DR
The paper tackles calibration of long-horizon off-policy value predictions in offline reinforcement learning without relying on Bellman completeness. It introduces Iterated Bellman Calibration, a post-hoc procedure that learns a one-dimensional calibrator via regressing doubly robust Bellman targets onto an existing value predictor, using histogram, isotonic, or hybrid calibration strategies. The authors formalize weak and strong Bellman calibration, provide finite-sample guarantees for calibration and prediction, and demonstrate that calibration can improve or preserve predictive accuracy. Practically, the method enables reliable, computation-efficient post-hoc adjustment of calibrated value estimates, with strong performance gains for misspecified or under-trained estimators, especially neural networks.
Abstract
We introduce Iterated Bellman Calibration, a simple, model-agnostic, post-hoc procedure for calibrating off-policy value predictions in infinite-horizon Markov decision processes. Bellman calibration requires that states with similar predicted long-term returns exhibit one-step returns consistent with the Bellman equation under the target policy. We adapt classical histogram and isotonic calibration to the dynamic, counterfactual setting by repeatedly regressing fitted Bellman targets onto a model's predictions, using a doubly robust pseudo-outcome to handle off-policy data. This yields a one-dimensional fitted value iteration scheme that can be applied to any value estimator. Our analysis provides finite-sample guarantees for both calibration and prediction under weak assumptions, and critically, without requiring Bellman completeness or realizability.