BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning

TL;DR

This work tackles the challenge of generalizing vision-based robotic manipulation to unseen tasks by proposing BC-Z, a large-scale interactive imitation learning framework that can condition policies on language or human video commands. It combines HG-DAgger-style interventions with multi-task training across 100 tasks to enable zero-shot and few-shot generalization, achieving 32% average success on 24 held-out tasks with language conditioning and notable improvements from interactive data collection. The authors demonstrate strong single-task baselines and provide insights into when language or video conditioning is most effective, while releasing a substantial real-robot dataset to the community. Overall, the study suggests that simple imitation learning, scaled up with flexible task conditioning and cooperative human-in-the-loop data collection, can substantially generalize to new tasks and lay groundwork for broader robotic versatility.

Abstract

In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning. We approach the challenge from an imitation learning perspective, aiming to study how scaling and broadening the data collected can facilitate such generalization. To that end, we develop an interactive and flexible imitation learning system that can learn from both demonstrations and interventions and can be conditioned on different forms of information that convey the task, including pre-trained embeddings of natural language or videos of humans performing the task. When scaling data collection on a real robot to more than 100 distinct tasks, we find that this system can perform 24 unseen manipulation tasks with an average success rate of 44%, without any robot demonstrations for those tasks.

Figures (13)

• Figure 1: Overview of BC-Z. We collect a large-scale dataset (25,877 episodes) of 100 diverse manipulation tasks, and train a 7-DoF multi-task policy that conditions on task language strings or human video. We show this system produces a policy that is capable of generalizing zero-shot to new unseen tasks.
• Figure 2: A subset of training tasks (top row), and a subset of held-out tasks (bottom two rows) used for evaluating zero shot task generalization. Top left: Given a pretrained task embedding computed from human videos or text, BC-Z acts as an "action decoder" for the task embedding.
• Figure 3: BC-Z network architecture. A monocular RGB image from the head-mounted camera is passed through a ResNet18 encoder, then through a two-layer MLP to predict each action modality (delta XYZ, delta axis-angle, and gripper angle). FiLM layers perez2018film condition the architecture on a task embedding $z$ computed from language $w_\ell$ or video $w_h$.
• Figure 4: Qualitative examples of BC-Z successfully performing held-out tasks.
• Figure 5: Mean number of interventions vs. task success rate. Each point represents a policy evaluated during HG-DAgger data collection. There is a clear correlation between the mean number of interventions and success rate, suggesting that interventions can be used as a live proxy for performance.