Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development

TL;DR

A dual-axis taxonomy is established that organizes the field along two orthogonal dimensions: a function-oriented taxonomy that clarifies the roles DMs play within the RL pipeline, and a technique-oriented taxonomy that situates implementations across online versus offline learning regimes.

Abstract

Diffusion Models (DMs), as a leading class of generative models, offer key advantages for reinforcement learning (RL), including multi-modal expressiveness, stable training, and trajectory-level planning. This survey delivers a comprehensive and up-to-date synthesis of diffusion-based RL. We first provide an overview of RL, highlighting its challenges, and then introduce the fundamental concepts of DMs, investigating how they are integrated into RL frameworks to address key challenges in this research field. We establish a dual-axis taxonomy that organizes the field along two orthogonal dimensions: a function-oriented taxonomy that clarifies the roles DMs play within the RL pipeline, and a technique-oriented taxonomy that situates implementations across online versus offline learning regimes. We also provide a comprehensive examination of this progression from single-agent to multi-agent domains, thereby forming several frameworks for DM-RL integration and highlighting their practical utility. Furthermore, we outline several categories of successful applications of diffusion-based RL across diverse domains, discuss open research issues of current methodologies, and highlight key directions for future research to advance the field. Finally, we summarize the survey to identify promising future development directions. We are actively maintaining a GitHub repository (https://github.com/ChangfuXu/D4RL-FTD) for papers and other related resources to apply DMs for RL.

Figures (12)

• Figure 1: The number of published papers by searching "Diffusion Model" in Web of Science (Access date: June 23, 2025). Diffusion models have garnered significant attention in both research and industry fields, particularly over the last two years.
• Figure 2: Overview of DMs for RL. DMs pretrained on broad data are adapted to accomplish specific tasks by making actions, interacting with external entities, and receiving feedback.
• Figure 3: The taxonomy of this survey. We provide a comprehensive overview of the current research landscape at the intersection of DMs and RL by systematically analyzing this emerging field's progress, challenges, solutions, and opportunities.
• Figure 4: Framework of Markov decision process. By observing the current system state $\textbf{S}_{t}$ and reward $r_{t-1}$,the agent generates an action $\textbf{A}_{t}$. Then, by performing the $\textbf{A}_{t}$, the reward $r_{t}$ and next system state $\textbf{S}_{t+1}$ will be obtained from the environment. Finally, the $\textbf{S}_{t+1}$ and $\textbf{S}_{t+1}$ are inputted to the agent to generate the next action $\textbf{A}_{t+1}$.
• Figure 5: The technical roadmap of DM development. There are technical routes: 1) From Autoencoder (AE), Vector Autoencoder (VAE), and DDPM to DDIM, CTDM, and GSDM; 2) From AE and Vector Quantised-Variational Autoencoder (VQ-VAE) to Latent Diffusion Model (LDM).