Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning

Norman Di Palo, Leonard Hasenclever, Jan Humplik, Arunkumar Byravan

TL;DR

DAAG addresses data scarcity in embodied RL by fusing large language models, vision-language models, and diffusion models into an autonomous lifelong learning framework. It introduces Hindsight Experience Augmentation to modify and re-use past experiences, and a diffusion pipeline that preserves geometrical and temporal consistency while editing observations. The approach enables autonomous reward detector finetuning for new tasks, efficient exploration through subgoal decomposition, and transfer via offline experience extraction and augmentation, leading to faster learning and improved robustness in simulated robotics tasks. Overall, DAAG demonstrates that diffusion-augmented observations and autonomous planning can substantially reduce rewards-labeling requirements and enhance lifelong transfer in embodied AI.

Abstract

We introduce Diffusion Augmented Agents (DAAG), a novel framework that leverages large language models, vision language models, and diffusion models to improve sample efficiency and transfer learning in reinforcement learning for embodied agents. DAAG hindsight relabels the agent's past experience by using diffusion models to transform videos in a temporally and geometrically consistent way to align with target instructions with a technique we call Hindsight Experience Augmentation. A large language model orchestrates this autonomous process without requiring human supervision, making it well-suited for lifelong learning scenarios. The framework reduces the amount of reward-labeled data needed to 1) finetune a vision language model that acts as a reward detector, and 2) train RL agents on new tasks. We demonstrate the sample efficiency gains of DAAG in simulated robotics environments involving manipulation and navigation. Our results show that DAAG improves learning of reward detectors, transferring past experience, and acquiring new tasks - key abilities for developing efficient lifelong learning agents. Supplementary material and visualizations are available on our website https://sites.google.com/view/diffusion-augmented-agents/

Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning

TL;DR

DAAG addresses data scarcity in embodied RL by fusing large language models, vision-language models, and diffusion models into an autonomous lifelong learning framework. It introduces Hindsight Experience Augmentation to modify and re-use past experiences, and a diffusion pipeline that preserves geometrical and temporal consistency while editing observations. The approach enables autonomous reward detector finetuning for new tasks, efficient exploration through subgoal decomposition, and transfer via offline experience extraction and augmentation, leading to faster learning and improved robustness in simulated robotics tasks. Overall, DAAG demonstrates that diffusion-augmented observations and autonomous planning can substantially reduce rewards-labeling requirements and enhance lifelong transfer in embodied AI.

Abstract

We introduce Diffusion Augmented Agents (DAAG), a novel framework that leverages large language models, vision language models, and diffusion models to improve sample efficiency and transfer learning in reinforcement learning for embodied agents. DAAG hindsight relabels the agent's past experience by using diffusion models to transform videos in a temporally and geometrically consistent way to align with target instructions with a technique we call Hindsight Experience Augmentation. A large language model orchestrates this autonomous process without requiring human supervision, making it well-suited for lifelong learning scenarios. The framework reduces the amount of reward-labeled data needed to 1) finetune a vision language model that acts as a reward detector, and 2) train RL agents on new tasks. We demonstrate the sample efficiency gains of DAAG in simulated robotics environments involving manipulation and navigation. Our results show that DAAG improves learning of reward detectors, transferring past experience, and acquiring new tasks - key abilities for developing efficient lifelong learning agents. Supplementary material and visualizations are available on our website https://sites.google.com/view/diffusion-augmented-agents/
Paper Structure (21 sections, 16 figures)

This paper contains 21 sections, 16 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminaries
  5. Finetune, Extract, Explore: The Diffusion Augmented Agent Framework
  6. Experiments
  7. Experimental Setup
  8. Can DAAG Finetune VLMs as Reward Detectors for Novel Tasks?
  9. Can DAAG Explore and Learn New Tasks More Efficiently?
  10. Can DAAG More Effectively Learn Tasks in Succession Transferring Experience from Past Tasks?
  11. Improving Robustness via Scene Visual Augmentation
  12. Conclusion
  13. Acknowledgments
  14. Appendix
  15. Backward Transfer with Hindsight Experience Augmentation
  16. ...and 6 more sections

Figures (16)

  • Figure 1: An illustration of our proposed framework.
  • Figure 2: An example of the Hindsight Experience Augmentation pipeline, with which new or past experience can be optionally modified and added to a new buffer.
  • Figure 3: Two example instances of tasks we use in our investigation.
  • Figure 4: An illustration of our diffusion pipeline, highlighting the geometrical and temporal consistency obtained by combining the methodologies in zhang2023adding and khachatryan2023text2videozero.
  • Figure 5: Outputs of the method proposed in ROSIE yu2023scaling and our diffusion pipeline when asked to "swap a green cup with a red cup".
  • ...and 11 more figures