Demonstration-Bootstrapped Autonomous Practicing via Multi-Task Reinforcement Learning

TL;DR

This work tackles the challenge of enabling autonomous, long-horizon robot learning in unseen kitchens with minimal human resets. It introduces Demonstration-Bootstrapped Autonomous Practicing (DBAP), which bootstraps a multi-task goal-conditioned policy from demonstrations and constructs a high-level task-graph to sequence subgoals, both during training and testing. By combining offline RL bootstrapping for low-level policies with a model-based graph search for task sequencing, DBAP achieves high real-world success on complex kitchen tasks while substantially reducing human intervention. The results—both in simulation and on a real robotic system—demonstrate that graph-based planning and demonstration-driven bootstrapping can enable practical, reset-efficient continual learning for temporally extended manipulation tasks.

Abstract

Reinforcement learning systems have the potential to enable continuous improvement in unstructured environments, leveraging data collected autonomously. However, in practice these systems require significant amounts of instrumentation or human intervention to learn in the real world. In this work, we propose a system for reinforcement learning that leverages multi-task reinforcement learning bootstrapped with prior data to enable continuous autonomous practicing, minimizing the number of resets needed while being able to learn temporally extended behaviors. We show how appropriately provided prior data can help bootstrap both low-level multi-task policies and strategies for sequencing these tasks one after another to enable learning with minimal resets. This mechanism enables our robotic system to practice with minimal human intervention at training time while being able to solve long horizon tasks at test time. We show the efficacy of the proposed system on a challenging kitchen manipulation task both in simulation and in the real world, demonstrating the ability to practice autonomously in order to solve temporally extended problems.

Figures (12)

• Figure 1: Robot setup for real world training in a kitchen. The robot needs to manipulate multiple different elements to accomplish complex goals.
• Figure 2: Given human-provided unstructured demonstrations, the system bootstraps a multi-task RL policy via offline RL and builds a task graph that models transitions between different tasks. The system then practices the tasks autonomously with small number of resets, using the task graph to command the appropriate next task. The resulting multi-task policy and task graph are then used to solve long-horizon problems at test-time.
• Figure 3: Elements, tasks, and goals in the real-world kitchen environment. The agent is manipulating the cabinet, slider and knob to accomplish particular configurations as shown in goals $0$ to $7$. The dotted lines represent individual transitions, toggling one element at a time between its extreme positions. The goal of the agent is to learn a policy and a graph controller that is able to transition between goal states.
• Figure 4: Simulated tasks in kitchen. Tasks involve manipulating the kitchen cabinet and the sliding door to achieve various combinations of configurations.
• Figure 5: Multi-step behavior in the kitchen that transitions between having all elements closed to all open, by first having the graph search command the agent to open the cabinet, then turn the knob and open the slider.