So You Think You Can Scale Up Autonomous Robot Data Collection?

TL;DR

This work suggests a negative result: that scaling up autonomous data collection for learning robot policies for real-world tasks is more challenging and impractical than what is suggested in prior work.

Abstract

A long-standing goal in robot learning is to develop methods for robots to acquire new skills autonomously. While reinforcement learning (RL) comes with the promise of enabling autonomous data collection, it remains challenging to scale in the real-world partly due to the significant effort required for environment design and instrumentation, including the need for designing reset functions or accurate success detectors. On the other hand, imitation learning (IL) methods require little to no environment design effort, but instead require significant human supervision in the form of collected demonstrations. To address these shortcomings, recent works in autonomous IL start with an initial seed dataset of human demonstrations that an autonomous policy can bootstrap from. While autonomous IL approaches come with the promise of addressing the challenges of autonomous RL as well as pure IL strategies, in this work, we posit that such techniques do not deliver on this promise and are still unable to scale up autonomous data collection in the real world. Through a series of real-world experiments, we demonstrate that these approaches, when scaled up to realistic settings, face much of the same scaling challenges as prior attempts in RL in terms of environment design. Further, we perform a rigorous study of autonomous IL methods across different data scales and 7 simulation and real-world tasks, and demonstrate that while autonomous data collection can modestly improve performance, simply collecting more human data often provides significantly more improvement. Our work suggests a negative result: that scaling up autonomous data collection for learning robot policies for real-world tasks is more challenging and impractical than what is suggested in prior work. We hope these insights about the core challenges of scaling up data collection help inform future efforts in autonomous learning.

Figures (13)

• Figure 1: Conceptual diagram illustrating the expected vs. actual effort tradeoff for human supervision and environment design.
• Figure 2: Illustration of environment design challenges and necessary simplification in our real-world environments to satisfy pre-conditions of running and evaluating autonomous IL methods.
• Figure 3: Success rate on HangOvenMitt over 600 rollouts, with window size of 50 rollouts.
• Figure 4: Performance of filtered BC over difference scales of human and autonomous data with 50-50 co-training on various simulation and real environments. More autonomous data often helps (left to right within each line plot), but having more initial human data (labeled H) generally has a stronger effect (purple vs. orange).
• Figure 5: Multiple Rounds (in order, left to right) of filtered BC in simulation and real environments. We see either saturating increases or decreases in performance.