Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach

TL;DR

ADAC tackles extrapolation error in offline RL by introducing a discriminative, advantage-based evaluation that uses a dataset-optimal value function $V_\mu^*$ and next-state values to modulate TD targets. The core idea is to assess OOD actions via an advantage $A(a|s)$ computed from $V(s')$ relative to a behavior-policy-based threshold, and to augment the Bellman backup with this advantage through a contractive operator $\mathcal{T}_A^{\pi_\theta}$. The method combines a diffusion-based policy with a Q-function that is guided by the advantage signal, while a fixed, offline-derived $A(a|s)$ stabilizes learning. Empirically, ADAC achieves state-of-the-art performance on most D4RL tasks, visualizes effective OOD action selection in PointMaze, and demonstrates strong transferability across locomotion, maze, manipulation, and kitchen domains, along with notable training and inference efficiency gains. This approach offers a principled balance between conservatism and generalization in offline reinforcement learning.

Abstract

Offline reinforcement learning (RL) learns policies from fixed datasets without online interactions, but suffers from distribution shift, causing inaccurate evaluation and overestimation of out-of-distribution (OOD) actions. Existing methods counter this by conservatively discouraging all OOD actions, which limits generalization. We propose Advantage-based Diffusion Actor-Critic (ADAC), which evaluates OOD actions via an advantage-like function and uses it to modulate the Q-function update discriminatively. Our key insight is that the (state) value function is generally learned more reliably than the action-value function; we thus use the next-state value to indirectly assess each action. We develop a PointMaze environment to clearly visualize that advantage modulation effectively selects superior OOD actions while discouraging inferior ones. Moreover, extensive experiments on the D4RL benchmark show that ADAC achieves state-of-the-art performance, with especially strong gains on challenging tasks.

Figures (10)

• Figure 1: ADAC Architecture: We use an approximated optimal value function (learned with a batch data) as the measure to evaluate OOD actions. We use an approximate optimal value function (learned from batch data) to evaluate OOD actions. Its relative advantage over in-distribution actions is then used to modulate critic update.
• Figure 2: Comparison of prior evaluation methods against our advantage-based evaluation. We visualize the Q-function for a fixed state. The thick blue solid line denotes the optimal value function. The solid line denotes the Q-function learned via advantage-based evaluation. The dashed line denotes the Q-function learned via conservative evaluation. The dotted line denotes the Q-function learned via standard Bellman backup evaluation.
• Figure 3: Performance comparison of DQL and ADAC on three AntMaze tasks: Umaze, Medium Diverse, and Large Play. Each method was trained with 4 random seeds, and the reward curves were smoothed with a running average ($n=10$). In this figure, the solid lines correspond to the mean and the shaded regions correspond to the standard deviation.
• Figure 4: Sparse reward PointMaze: dataset and method performance.Top: 853 sub-optimal trajectories are gathered with only terminal reward. Three trajectory patterns—Left (33%), Middle (22%), Right (45%)—span lengths of 200 to 1000 steps (the optimal length is 142). Bottom: The first two subfigures illustrate the trajectories generated by DQL and ADAC after training on the dataset, respectively. The last subfigure summarizes the distribution of trajectory lengths for the offline dataset, DQL, and ADAC.
• Figure 5: Ablation of the advantage component across four domains. Bars show domain-level mean normalized scores.

Theorems & Definitions (8)

• Proposition 3.1
• Proposition 3.2
• Proposition 3.3
• Proposition 3.4
• proof
• proof
• proof
• proof