UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning

TL;DR

This work tackles the problem of overestimation in offline reinforcement learning caused by MSE-based value-function estimation and the max-operator backups. It introduces UDQL, an offline RL framework that uses an underestimated Bellman backup implemented via expectile regression, together with a diffusion-policy model to learn robust policies. The authors prove contraction of the proposed underestimated operator and provide theoretical bounds on overestimation under a Gumbel-based error model, then demonstrate strong empirical performance on the D4RL benchmark, often achieving state-of-the-art results. The approach reduces reliance on behavior cloning and highlights that controlled underestimation coupled with expressive diffusion policies can improve stability and performance in offline RL.

Abstract

The Mean Square Error (MSE) is commonly utilized to estimate the solution of the optimal value function in the vast majority of offline reinforcement learning (RL) models and has achieved outstanding performance. However, we find that its principle can lead to overestimation phenomenon for the value function. In this paper, we first theoretically analyze overestimation phenomenon led by MSE and provide the theoretical upper bound of the overestimated error. Furthermore, to address it, we propose a novel Bellman underestimated operator to counteract overestimation phenomenon and then prove its contraction characteristics. At last, we propose the offline RL algorithm based on underestimated operator and diffusion policy model. Extensive experimental results on D4RL tasks show that our method can outperform state-of-the-art offline RL algorithms, which demonstrates that our theoretical analysis and underestimation way are effective for offline RL tasks.