Imagination-Limited Q-Learning for Offline Reinforcement Learning

TL;DR

Offline reinforcement learning must cope with distributional shifts that inflate OOD action values. Imagination-Limited Q-learning (ILQ) uses a one-step imagined backup from a learned dynamics model and caps it with the maximum in-distribution value, controlled by an offset $\delta$, yielding the Imagination-Limited Bellman operator which is a $\gamma$-contraction. Theoretical results show the OOD action-value error matches the in-distribution error in order, and empirical results on the D4RL suite demonstrate state-of-the-art performance across MuJoCo, Maze2D, and Adroit tasks. This approach provides a practical, convergence-guaranteed way to retain useful optimism for OOD actions while avoiding excessive bias, improving offline policy learning.

Abstract

Offline reinforcement learning seeks to derive improved policies entirely from historical data but often struggles with over-optimistic value estimates for out-of-distribution (OOD) actions. This issue is typically mitigated via policy constraint or conservative value regularization methods. However, these approaches may impose overly constraints or biased value estimates, potentially limiting performance improvements. To balance exploitation and restriction, we propose an Imagination-Limited Q-learning (ILQ) method, which aims to maintain the optimism that OOD actions deserve within appropriate limits. Specifically, we utilize the dynamics model to imagine OOD action-values, and then clip the imagined values with the maximum behavior values. Such design maintains reasonable evaluation of OOD actions to the furthest extent, while avoiding its over-optimism. Theoretically, we prove the convergence of the proposed ILQ under tabular Markov decision processes. Particularly, we demonstrate that the error bound between estimated values and optimality values of OOD state-actions possesses the same magnitude as that of in-distribution ones, thereby indicating that the bias in value estimates is effectively mitigated. Empirically, our method achieves state-of-the-art performance on a wide range of tasks in the D4RL benchmark.

Figures (10)

• Figure 1: (a) illustrates the fundamental principle of value regularization methods. While effectively suppressing OOD action-values, it may introduce uncontrolled bias in estimations. In contrast, instead of indiscriminately suppressing OOD action-values, ILQ, depicted in (b), envisions reasonable values (purple line) and then appropriately limits potential over-estimations using the maximum behavior value $Q^{\beta}_{\rm max}$ (black dashed line), resulting in more appropriate policy evaluation (cyan line). (c) demonstrates that Q-value estimations of CQL across MuJoCo "-v2" tasks are notably compromised, falling well below maximum returns (black line) in datasets. Conversely, ILQ maintains reasonably optimistic Q-value estimations in anticipation of superior policies. Finally, (d) shows that ILQ's ultimate performance is significantly improved, particularly in medium tasks.
• Figure 2: Performances of ILQ under different values of offset parameter $\delta$.
• Figure 3: Performances of ILQ under different values of trade-off factor $\eta$.
• Figure 4: Performance comparison of the ILQ algorithm with and without the imagined value $y_{\rm img}^Q$ in the target value.
• Figure 5: Performance comparison of the ILQ algorithm with and without the limitation value $y_{\rm lmt}^Q$ in the target value.

Theorems & Definitions (11)

• Definition 1
• Theorem 1: Convergence
• Theorem 2
• Theorem 3
• Theorem 4: Action-value gap
• proof
• Lemma 1
• proof
• proof
• proof