Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Diffusion Policy Optimization for Robotic Manipulation

Huiyun Jiang, Zhuang Yang

TL;DR

This work tackles the challenge of fast and stable optimization of diffusion-based policies in robotic reinforcement learning. It proposes ADPO, an Adam-based Diffusion Policy Optimization framework that employs adaptive gradient updates with a discount factor to interpolate among optimizers and Katyusha-style momentum to reduce noise-induced oscillations. The approach is validated by embedding ADAPG into six diffusion-based RL methods (DPPO, DIPO, IDQL, DAWR, QSM, DQL) and testing on OpenAI Gym and ROBOMIMIC robotic manipulation tasks, where ADPO variants achieve superior or comparable performance and improved stability. The paper also conducts a thorough hyperparameter study, revealing robust settings for $\varepsilon$ and task-dependent settings for $\omega$, providing actionable guidance for practitioners.

Abstract

Recent studies have shown the great potential of diffusion models in improving reinforcement learning (RL) by modeling complex policies, expressing a high degree of multi-modality, and efficiently handling high-dimensional continuous control tasks. However, there is currently limited research on how to optimize diffusion-based polices (e.g., Diffusion Policy) fast and stably. In this paper, we propose an Adam-based Diffusion Policy Optimization (ADPO), a fast algorithmic framework containing best practices for fine-tuning diffusion-based polices in robotic control tasks using the adaptive gradient descent method in RL. Adaptive gradient method is less studied in training RL, let alone diffusion-based policies. We confirm that ADPO outperforms other diffusion-based RL methods in terms of overall effectiveness for fine-tuning on standard robotic tasks. Concretely, we conduct extensive experiments on standard robotic control tasks to test ADPO, where, particularly, six popular diffusion-based RL methods are provided as benchmark methods. Experimental results show that ADPO acquires better or comparable performance than the baseline methods. Finally, we systematically analyze the sensitivity of multiple hyperparameters in standard robotics tasks, providing guidance for subsequent practical applications. Our video demonstrations are released in https://github.com/Timeless-lab/ADPO.git.

Adaptive Diffusion Policy Optimization for Robotic Manipulation

TL;DR

This work tackles the challenge of fast and stable optimization of diffusion-based policies in robotic reinforcement learning. It proposes ADPO, an Adam-based Diffusion Policy Optimization framework that employs adaptive gradient updates with a discount factor to interpolate among optimizers and Katyusha-style momentum to reduce noise-induced oscillations. The approach is validated by embedding ADAPG into six diffusion-based RL methods (DPPO, DIPO, IDQL, DAWR, QSM, DQL) and testing on OpenAI Gym and ROBOMIMIC robotic manipulation tasks, where ADPO variants achieve superior or comparable performance and improved stability. The paper also conducts a thorough hyperparameter study, revealing robust settings for and task-dependent settings for , providing actionable guidance for practitioners.

Abstract

Recent studies have shown the great potential of diffusion models in improving reinforcement learning (RL) by modeling complex policies, expressing a high degree of multi-modality, and efficiently handling high-dimensional continuous control tasks. However, there is currently limited research on how to optimize diffusion-based polices (e.g., Diffusion Policy) fast and stably. In this paper, we propose an Adam-based Diffusion Policy Optimization (ADPO), a fast algorithmic framework containing best practices for fine-tuning diffusion-based polices in robotic control tasks using the adaptive gradient descent method in RL. Adaptive gradient method is less studied in training RL, let alone diffusion-based policies. We confirm that ADPO outperforms other diffusion-based RL methods in terms of overall effectiveness for fine-tuning on standard robotic tasks. Concretely, we conduct extensive experiments on standard robotic control tasks to test ADPO, where, particularly, six popular diffusion-based RL methods are provided as benchmark methods. Experimental results show that ADPO acquires better or comparable performance than the baseline methods. Finally, we systematically analyze the sensitivity of multiple hyperparameters in standard robotics tasks, providing guidance for subsequent practical applications. Our video demonstrations are released in https://github.com/Timeless-lab/ADPO.git.
Paper Structure (16 sections, 24 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 16 sections, 24 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion-based RL
  4. ADAptive Policy Gradient
  5. Preliminaries
  6. Reinforcement Learning
  7. Diffusion Model
  8. Diffusion model as policies
  9. Method
  10. Six popular Diffusion-based RL methods
  11. Adam-based Diffusion Policy Optimization (ADPO) framework
  12. Experiment
  13. Environments
  14. Comparison of ADPO diffusion-based RL methods with Baselines
  15. Performance of different $\varepsilon$ and $\omega$ on GYM tasks
  16. ...and 1 more sections

Figures (4)

  • Figure 1: The first line shows the case of GYM benchmark, and the second line shows the case of ROBOMIMIC benchmark.
  • Figure 2: Comparison of ADAPG diffusion-based RL methods with modern diffusion-based RL methods on the GYM tasks, with an average of 5 seeds.
  • Figure 3: Comparison of ADAPG diffusion-based RL methods with advanced diffusion-based RL methods on the ROBOMIMIC tasks, with an average of 3 seeds.
  • Figure 4: A comparison of the performance of ADPPO with different $\varepsilon$ (left) and different $\omega$ (right) on GYM tasks.