ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
Kun Wu, Yinuo Zhao, Zhiyuan Xu, Zhengping Che, Chengxiang Yin, Chi Harold Liu, Feiferi Feng, Jian Tang
TL;DR
The paper tackles offline reinforcement learning challenges arising from distribution shift and Q-value overestimation by proposing Adaptive Conservative Level in Q-Learning (ACL-QL). It introduces two learnable weight functions to modulate conservatism per state-action pair, ensuring Q-values lie in a mild range between the ordinary Q-function and the CQL estimate, with per-transition adjustments guided by transition-quality signals and monotonicity constraints. The authors provide theoretical conditions linking weight functions to conservatism levels and offer a practical algorithm that combines monotonicity losses, surrogate hinge losses, and a BC-like behavioral policy term, then validate it on the D4RL suite showing state-of-the-art results across multiple domains. The approach reduces the need for hand-tuned global hyperparameters and improves generalization across datasets by enabling fine-grained control over conservatism, which has practical implications for robust offline RL deployment.
Abstract
Offline Reinforcement Learning (RL), which operates solely on static datasets without further interactions with the environment, provides an appealing alternative to learning a safe and promising control policy. The prevailing methods typically learn a conservative policy to mitigate the problem of Q-value overestimation, but it is prone to overdo it, leading to an overly conservative policy. Moreover, they optimize all samples equally with fixed constraints, lacking the nuanced ability to control conservative levels in a fine-grained manner. Consequently, this limitation results in a performance decline. To address the above two challenges in a united way, we propose a framework, Adaptive Conservative Level in Q-Learning (ACL-QL), which limits the Q-values in a mild range and enables adaptive control on the conservative level over each state-action pair, i.e., lifting the Q-values more for good transitions and less for bad transitions. We theoretically analyze the conditions under which the conservative level of the learned Q-function can be limited in a mild range and how to optimize each transition adaptively. Motivated by the theoretical analysis, we propose a novel algorithm, ACL-QL, which uses two learnable adaptive weight functions to control the conservative level over each transition. Subsequently, we design a monotonicity loss and surrogate losses to train the adaptive weight functions, Q-function, and policy network alternatively. We evaluate ACL-QL on the commonly used D4RL benchmark and conduct extensive ablation studies to illustrate the effectiveness and state-of-the-art performance compared to existing offline DRL baselines.