Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Reach Goals via Diffusion

Vineet Jain, Siamak Ravanbakhsh

TL;DR

A novel perspective on goal-conditioned reinforcement learning by framing it within the context of denoising diffusion models, which stands out as the first method to perform diffusion in the state space, requiring only one ``denoising"iteration per environment step.

Abstract

We present a novel perspective on goal-conditioned reinforcement learning by framing it within the context of denoising diffusion models. Analogous to the diffusion process, where Gaussian noise is used to create random trajectories that walk away from the data manifold, we construct trajectories that move away from potential goal states. We then learn a goal-conditioned policy to reverse these deviations, analogous to the score function. This approach, which we call Merlin, can reach specified goals from arbitrary initial states without learning a separate value function. In contrast to recent works utilizing diffusion models in offline RL, Merlin stands out as the first method to perform diffusion in the state space, requiring only one ``denoising" iteration per environment step. We experimentally validate our approach in various offline goal-reaching tasks, demonstrating substantial performance enhancements compared to state-of-the-art methods while improving computational efficiency over other diffusion-based RL methods by an order of magnitude. Our results suggest that this perspective on diffusion for RL is a simple and scalable approach for sequential decision making.

Learning to Reach Goals via Diffusion

TL;DR

A novel perspective on goal-conditioned reinforcement learning by framing it within the context of denoising diffusion models, which stands out as the first method to perform diffusion in the state space, requiring only one ``denoising"iteration per environment step.

Abstract

We present a novel perspective on goal-conditioned reinforcement learning by framing it within the context of denoising diffusion models. Analogous to the diffusion process, where Gaussian noise is used to create random trajectories that walk away from the data manifold, we construct trajectories that move away from potential goal states. We then learn a goal-conditioned policy to reverse these deviations, analogous to the score function. This approach, which we call Merlin, can reach specified goals from arbitrary initial states without learning a separate value function. In contrast to recent works utilizing diffusion models in offline RL, Merlin stands out as the first method to perform diffusion in the state space, requiring only one ``denoising" iteration per environment step. We experimentally validate our approach in various offline goal-reaching tasks, demonstrating substantial performance enhancements compared to state-of-the-art methods while improving computational efficiency over other diffusion-based RL methods by an order of magnitude. Our results suggest that this perspective on diffusion for RL is a simple and scalable approach for sequential decision making.
Paper Structure (66 sections, 2 theorems, 36 equations, 19 figures, 11 tables, 2 algorithms)

This paper contains 66 sections, 2 theorems, 36 equations, 19 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Diffusion Probabilistic Models
  4. Goal-conditioned Reinforcement Learning
  5. Reaching Goals via Diffusion
  6. Forward and Reverse Process
  7. Optimization Objective
  8. An Illustrative Example
  9. Generating Reverse Trajectories
  10. Reverse Parametric Model
  11. Reverse policy
  12. Reverse Dynamics Model
  13. Reverse Non-Parametric Model
  14. Theoretical Implications
  15. Experiments
  16. ...and 51 more sections

Key Result

Theorem 3.1

Consider a dataset $\mathcal{D}(g)$ collected by an unknown behavior policy $\pi_\beta$, consisting of trajectories ending in states $S_T \coloneqq \{s_T \mid g=\phi(s_T)\}$ with $q(s_T|g)$ denoting the distribution of final states corresponding to $g$. Then, behavior cloning given by $\theta^* = \m

Figures (19)

  • Figure 1: Reverse diffusion policy.
  • Figure 1: Discounted returns for state-space input, averaged over 10 seeds.
  • Figure 2: Forward and reverse diffusion process for GCRL using 2D navigation as an example. Star represents the goal state, the red dotted arrows denote the forward process transitions $q(s_t|s_{t+1})$, and the green arrows denote the reverse process transitions $\mathcal{P}^{\pi_\theta}(s_{t+1}|s_t)$.
  • Figure 2: Discounted returns for pixel-space input, averaged over 10 seeds.
  • Figure 3: (a) Visualization of trajectories starting from the goal X generated during the forward process, (b) Predicted actions from policy trained via diffusion, (c) Predicted actions from policy trained using GCSL.
  • ...and 14 more figures

Theorems & Definitions (2)

  • Theorem 3.1
  • Corollary 3.2