Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learning

Hanye Zhao, Xiaoshen Han, Zhengbang Zhu, Minghuan Liu, Yong Yu, Weinan Zhang

TL;DR

This paper addresses the challenge of generating long-horizon, policy-consistent rollouts in offline RL by decoupling diffusion models from the learning policy. It introduces Dynamics Diffusion (DyDiff), which uses diffusion models as long-horizon rollout synthesizers initialized by a policy-consistent, single-step dynamics rollout and then iteratively refines trajectories with the learning policy, controlled by reward-based filtering. The authors provide theoretical bounds showing non-autoregressive DM rollouts bound the return error more tightly than autoregressive single-step models, and they demonstrate empirical improvements across D4RL MuJoCo and Maze2d benchmarks when DyDiff is used as an add-on to existing model-free offline RL methods. The work highlights the importance of policy-mismatch-aware data generation and offers a practical, scalable approach to leverage diffusion models for offline policy improvement.

Abstract

With the great success of diffusion models (DMs) in generating realistic synthetic vision data, many researchers have investigated their potential in decision-making and control. Most of these works utilized DMs to sample directly from the trajectory space, where DMs can be viewed as a combination of dynamics models and policies. In this work, we explore how to decouple DMs' ability as dynamics models in fully offline settings, allowing the learning policy to roll out trajectories. As DMs learn the data distribution from the dataset, their intrinsic policy is actually the behavior policy induced from the dataset, which results in a mismatch between the behavior policy and the learning policy. We propose Dynamics Diffusion, short as DyDiff, which can inject information from the learning policy to DMs iteratively. DyDiff ensures long-horizon rollout accuracy while maintaining policy consistency and can be easily deployed on model-free algorithms. We provide theoretical analysis to show the advantage of DMs on long-horizon rollout over models and demonstrate the effectiveness of DyDiff in the context of offline reinforcement learning, where the rollout dataset is provided but no online environment for interaction. Our code is at https://github.com/FineArtz/DyDiff.

Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learning

TL;DR

This paper addresses the challenge of generating long-horizon, policy-consistent rollouts in offline RL by decoupling diffusion models from the learning policy. It introduces Dynamics Diffusion (DyDiff), which uses diffusion models as long-horizon rollout synthesizers initialized by a policy-consistent, single-step dynamics rollout and then iteratively refines trajectories with the learning policy, controlled by reward-based filtering. The authors provide theoretical bounds showing non-autoregressive DM rollouts bound the return error more tightly than autoregressive single-step models, and they demonstrate empirical improvements across D4RL MuJoCo and Maze2d benchmarks when DyDiff is used as an add-on to existing model-free offline RL methods. The work highlights the importance of policy-mismatch-aware data generation and offers a practical, scalable approach to leverage diffusion models for offline policy improvement.

Abstract

With the great success of diffusion models (DMs) in generating realistic synthetic vision data, many researchers have investigated their potential in decision-making and control. Most of these works utilized DMs to sample directly from the trajectory space, where DMs can be viewed as a combination of dynamics models and policies. In this work, we explore how to decouple DMs' ability as dynamics models in fully offline settings, allowing the learning policy to roll out trajectories. As DMs learn the data distribution from the dataset, their intrinsic policy is actually the behavior policy induced from the dataset, which results in a mismatch between the behavior policy and the learning policy. We propose Dynamics Diffusion, short as DyDiff, which can inject information from the learning policy to DMs iteratively. DyDiff ensures long-horizon rollout accuracy while maintaining policy consistency and can be easily deployed on model-free algorithms. We provide theoretical analysis to show the advantage of DMs on long-horizon rollout over models and demonstrate the effectiveness of DyDiff in the context of offline reinforcement learning, where the rollout dataset is provided but no online environment for interaction. Our code is at https://github.com/FineArtz/DyDiff.
Paper Structure (30 sections, 5 theorems, 29 equations, 4 figures, 4 tables, 2 algorithms)

This paper contains 30 sections, 5 theorems, 29 equations, 4 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion Models in offline RL.
  4. Offline model-based RL.
  5. Preliminaries
  6. Diffusion model.
  7. Offline RL.
  8. Dynamics Diffusion (DyDiff)
  9. Diffusion Models as Rollout Synthesizer
  10. Correct Rollouts with Diffusion Models
  11. Theoretical Analysis
  12. Experiments
  13. Experiment settings
  14. Results
  15. Ablation Studies
  16. ...and 15 more sections

Key Result

Lemma 1

(Lemma B.3 of MBPO). Suppose the error of a single-step dynamics model $T_m(s'|s,a)$ can be bounded as $\max_t \mathbb{E}_{a\sim \pi} [D_{\mathrm{KL}}(T_m(s'|s,a) \| T(s'|s,a))] \le \epsilon_m$. Then after executing the same policy $\pi$ from the same initial state $s_0$ in $T_m$ and the real dynami

Figures (4)

  • Figure 1: Training the policy on a part of hopper-medium-replay dataset under different settings. (a) During training, we train a diffusion model to generate and gradually augment on-policy data and dataset behavior data, compared with no extra data augmented. (b) Augment model generated on-policy rollouts with different lengths. (c) Use single-step dynamics models and our DyDiff to generate rollouts. The detailed setting is described in Appendix \ref{['app:motivation']}.
  • Figure 2: The sketch process of DyDiff. It mainly consists of three parts: (1) Sampling start states from $\mathcal{D}$ to generate initial trajectories as conditions. (2) Synthesizing rollout trajectories by iteratively sampling from the DM and the learning policy. (3) Filtering synthesized data and adding high-reward trajectories to $\mathcal{D}_\mathrm{syn}$.
  • Figure 3: The mean-squared error of DMs and single-step models for rollout. As the rollout length increases, the MSE of single-step models surpasses DMs.
  • Figure 4: Ablation studies on various hyperparameters. Experiments on iteration times and rollout length validate our theory analysis, whereas those on filter proportion and real ratio prove the robustness of DyDiff.

Theorems & Definitions (5)

  • Lemma 1
  • Theorem 1
  • Lemma 2
  • Lemma 1
  • Theorem 1