Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learning

TL;DR

This paper tackles offline reinforcement learning where out-of-distribution actions cause Q-value overestimation. It introduces DIVO, a diffusion-based framework that models the behavior policy with a diffusion process and uses value conditioning to bias training toward high-advantage actions, via a binary-weighted mechanism (PAD). Policy improvement is guided by adaptive diffusion-based optimization (ADPO), which selects high-advantage diffusion actions as targets to constrain the learned policy without backpropagating through time in the Q-value loss. Built on TD3+BC, DIVO achieves state-of-the-art results on the D4RL benchmark, with notable gains in locomotion tasks and particularly strong performance in the sparse AntMaze domain. The approach offers a robust, efficient balance between conservatism and explorability by leveraging diffusion expressiveness while focusing updates on high-value regions, making it practical for diverse offline-RL applications.

Abstract

In offline reinforcement learning, value overestimation caused by out-of-distribution (OOD) actions significantly limits policy performance. Recently, diffusion models have been leveraged for their strong distribution-matching capabilities, enforcing conservatism through behavior policy constraints. However, existing methods often apply indiscriminate regularization to redundant actions in low-quality datasets, resulting in excessive conservatism and an imbalance between the expressiveness and efficiency of diffusion modeling. To address these issues, we propose DIffusion policies with Value-conditional Optimization (DIVO), a novel approach that leverages diffusion models to generate high-quality, broadly covered in-distribution state-action samples while facilitating efficient policy improvement. Specifically, DIVO introduces a binary-weighted mechanism that utilizes the advantage values of actions in the offline dataset to guide diffusion model training. This enables a more precise alignment with the dataset's distribution while selectively expanding the boundaries of high-advantage actions. During policy improvement, DIVO dynamically filters high-return-potential actions from the diffusion model, effectively guiding the learned policy toward better performance. This approach achieves a critical balance between conservatism and explorability in offline RL. We evaluate DIVO on the D4RL benchmark and compare it against state-of-the-art baselines. Empirical results demonstrate that DIVO achieves superior performance, delivering significant improvements in average returns across locomotion tasks and outperforming existing methods in the challenging AntMaze domain, where sparse rewards pose a major difficulty.

Figures (5)

• Figure 1: The robot continuous control offline RL benchmarks including ant, halfcheetah, hopper and walker2d.
• Figure 2: Performance comparison outcomes for nine original tasks within the D4RL dataset. The lines and shaded regions represent the mean values and standard deviations, computed across 5 different random seeds respectively.
• Figure 3: Robust assessment of statistical uncertainty on D4RL using 95% confidence intervals derived from 18 tasks, with 5 random seeds per task.
• Figure 4: The hyperparameter sensitivity analysis examines the effect of the weight $\eta$ in the optimization objective of the diffusion model and the weight $\beta$ in the optimization objective of policy improvement.
• Figure 5: To ensure reliable comparison, we evaluate performance on D4RL across 18 tasks, each with 5 random seeds.