A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Control

TL;DR

A novel taxonomy is proposed that categorizes existing approaches into four distinct families -- Action-Gradient, Q-Weighting, Proximity-Based, and Backpropagation Through Time methods -- based on their policy improvement mechanisms, and provides concrete guidelines for algorithm selection tailored to specific operational constraints.

Abstract

Diffusion policies have emerged as a powerful approach for robotic control, demonstrating superior expressiveness in modeling multimodal action distributions compared to conventional policy networks. However, their integration with online reinforcement learning remains challenging due to fundamental incompatibilities between diffusion model training objectives and standard RL policy improvement mechanisms. This paper presents the first comprehensive review and empirical analysis of current Online Diffusion Policy Reinforcement Learning (Online DPRL) algorithms for scalable robotic control systems. We propose a novel taxonomy that categorizes existing approaches into four distinct families -- Action-Gradient, Q-Weighting, Proximity-Based, and Backpropagation Through Time (BPTT) methods -- based on their policy improvement mechanisms. Through extensive experiments on a unified NVIDIA Isaac Lab benchmark encompassing 12 diverse robotic tasks, we systematically evaluate representative algorithms across five critical dimensions: task diversity, parallelization capability, diffusion step scalability, cross-embodiment generalization, and environmental robustness. Our analysis identifies key findings regarding the fundamental trade-offs inherent in each algorithmic family, particularly concerning sample efficiency and scalability. Furthermore, we reveal critical computational and algorithmic bottlenecks that currently limit the practical deployment of online DPRL. Based on these findings, we provide concrete guidelines for algorithm selection tailored to specific operational constraints and outline promising future research directions to advance the field toward more general and scalable robotic learning systems.

Figures (7)

• Figure 1: Comparison of training methodologies for Diffusion Policies. (a) Imitation Learning (IL): Most conventional diffusion policies are trained in a supervised manner using expert demonstrations, focusing on behavior cloning. (b) Offline Diffusion Policy Reinforcement Learning (Offline DPRL): Recent approaches leverage additional reward labeling to learn reward signals and optimize policies within a static offline dataset. (c) Online Diffusion Policy RL (Online DPRL): Unlike previous methods, online DPRL enables real-time interaction with the environment, facilitating a more explorative learning process through direct feedback and continuous policy refinement.
• Figure 2: Overview of the paper’s organization and core themes. This review offers a holistic perspective on the intersection of diffusion models and online RL paradigms. By synthesizing empirical analyses and theoretical frameworks, we classify the field's emerging progress into a structured taxonomy, bridging the gap between current technical challenges and their respective solutions.
• Figure 3: Taxonomy of online Diffusion Policy Reinforcement Learning Paradigm. We categorize existing online DPRL approaches into four distinct families based on their policy training mechanisms: Action-Gradient, $Q$-weighting, Proximity-based, and BPTT-based methods.
• Figure 4: Training curves of online RL and DPRL methods across various Isaac Lab tasks. The solid lines represent the mean episodic reward averaged over five independent runs, with the shaded regions indicating the variance. All experiments were conducted using 1,024 parallelized environments to ensure high-throughput data collection.
• Figure 5: Training curves of online RL and DPRL methods across the number of simulation environments on the Ant and Unitree Go2 environments. The solid lines represent the mean episodic reward averaged over five independent runs, with the shaded regions indicating the variance.