Interactive Robot Learning from Verbal Correction

TL;DR

This work introduces OLAF, an LLM-based learning system that updates a robot's visuomotor policy from verbal corrections provided by regular users, enabling learning from mistakes rather than mere obedience to commands. OLAF relabels the pre-intervention portion of a stopped trajectory using an LLM as a critic, synthesizing data for policy updates via behavior cloning, and optionally incorporates physical interventions. The approach yields about a 20% improvement in policy success across simulation and real-robot tasks, and remains compatible with existing human-in-the-loop methods. By offloading LLM queries offline and grounding reasoning in robot observations, OLAF achieves smoother control and practical applicability in unstructured environments.

Abstract

The ability to learn and refine behavior after deployment has become ever more important for robots as we design them to operate in unstructured environments like households. In this work, we design a new learning system based on large language model (LLM), OLAF, that allows everyday users to teach a robot using verbal corrections when the robot makes mistakes, e.g., by saying "Stop what you're doing. You should move closer to the cup." A key feature of OLAF is its ability to update the robot's visuomotor neural policy based on the verbal feedback to avoid repeating mistakes in the future. This is in contrast to existing LLM-based robotic systems, which only follow verbal commands or corrections but not learn from them. We demonstrate the efficacy of our design in experiments where a user teaches a robot to perform long-horizon manipulation tasks both in simulation and on physical hardware, achieving on average 20.0% improvement in policy success rate. Videos and more results are at https://ut-austin-rpl.github.io/olaf/

Figures (20)

• Figure 1: Teaching the robot through verbal correction with OLAF. OLAF is a LLM-based learning system designed for updating a robot's visuomotor neural-network-based policy using verbal corrections given by regular non-expert users. To train the robot, the user simply needs to watch to robot performing a task, stop the robot when the user thinks the robot is not able to finish the task, and then provide an instruction in natural language on how the robot can do better.
• Figure 2: OLAF System. The OLAF pipeline consists of three steps: User Interaction, Data Synthesis, and Policy Update. In User Interaction, it collects pairs of $\langle \mathit{robot\ trajectory}, \mathit{verbal\ correction} \rangle$ of trajectories stopped by the user. In Data Synthesis, it uses the LLM as a critic to select the action (from a pool of action candidates) that best matches the user's verbal correction and relabels the pre-intervention trajectory segments (in red). In Policy Update, it updates the policy by performing behavior cloning on the newly synthesized data and the previously collected data.
• Figure 3: The left shows an example of a user interaction. The right shows how a trajectory is partitioned and the action in the pre-intervention region (abnormal state) is relabelled by the LLM.
• Figure 4: Prompts of an LLM as a critic for action relabeling. The system prompt specify system-level desired behavior, the context prompt describes the task level instruction, and the action relabeling prompt includes the trajectory-level information and the verbal correction. The black denotes the template and the blue denotes user- or sensor-dependent information. We highlight the action proposal in blue background.
• Figure 5: We evaluate OLAF on four tasks in simulation and two tasks on real robot. The tasks in simulation are fine-grained manipulation tasks while the tasks on real robot are long-horizon, multi-staged tasks.