Teaching Robots Like Dogs: Learning Agile Navigation from Luring, Gesture, and Speech

TL;DR

This work tackles enabling legged robots to interpret human social cues for agile navigation by introducing LURE, a multimodal, data-efficient framework that uses physical luring, gestural and verbal commands, and simulation-based scene reconstruction. It combines a high-level navigation policy with a low-level agile locomotion controller, augmented by data-aggregation (DAgger-style) and progressive goal cueing to mitigate distributional shifts from limited demonstrations, along with domain randomization and modality-specific data augmentation. Across six real-world scenarios, LURE achieves a 97.15% task-success rate with less than one hour of demonstrations, and it demonstrates rapid adaptation to novel users. The approach advances intuitive, data-efficient human-robot interaction for cluttered environments and lays groundwork for extending multimodal grounding to broader whole-body behaviors.

Abstract

In this work, we aim to enable legged robots to learn how to interpret human social cues and produce appropriate behaviors through physical human guidance. However, learning through physical engagement can place a heavy burden on users when the process requires large amounts of human-provided data. To address this, we propose a human-in-the-loop framework that enables robots to acquire navigational behaviors in a data-efficient manner and to be controlled via multimodal natural human inputs, specifically gestural and verbal commands. We reconstruct interaction scenes using a physics-based simulation and aggregate data to mitigate distributional shifts arising from limited demonstration data. Our progressive goal cueing strategy adaptively feeds appropriate commands and navigation goals during training, leading to more accurate navigation and stronger alignment between human input and robot behavior. We evaluate our framework across six real-world agile navigation scenarios, including jumping over or avoiding obstacles. Our experimental results show that our proposed method succeeds in almost all trials across these scenarios, achieving a 97.15% task success rate with less than 1 hour of demonstration data in total.

Figures (7)

• Figure 1: The proposed framework enables agile robot behaviors through verbal and gesture-based commands.
• Figure 2: The problem is formulated as an MDP with observations $(\mathbf{x}, \boldsymbol{\rho}, \mathbf{c})$ and actions representing the navigation goal $\mathbf{g}$. The transition dynamics include the pretrained locomotion controller $\pi_u$, the robot dynamics $f$, and the human model $h$, while the reward is defined as the alignment between the predicted and intended goals.
• Figure 3: Overview of the proposed framework. We first establish a locomotion controller that follows the navigation goal $\mathbf{g}$. In Stage 1, we collect interaction data $\mathcal{D}=\{\mathbf{v}, \mathbf{m}, \boldsymbol{\rho}, \mathbf{x}, \mathbf{g}^*\}$ through natural interactions between two participants. In Stage 2, we reconstruct the interaction scene from $\mathcal{D}$ and train the navigation module via data aggregation. The framework progressively provides the interaction and navigation goals only after the robot reaches its current goal, ensuring alignment between the command and behavior. Finally, in Stage 3, the user can control the robot through interaction.
• Figure 4: Illustration of data collection procedures for six interactive navigation scenarios. (a) Go there, (b) Come here, and (c) Follow me represent human-robot interactions in open space. (d) Come around and (e) Jump over illustrate interactions involving a box obstacle. (f) Zigzag demonstrates interaction with multiple tire obstacles.
• Figure 5: Analysis of collected data in terms of (a) total wall clock time and (b) number of episodes for each scenario