Speech-to-Trajectory: Learning Human-Like Verbal Guidance for Robot Motion

TL;DR

The paper tackles the challenge of translating natural spoken commands into low-level robot trajectories in $SE(2)$ space. It proposes the Directive Language Model (DLM), a speech-to-trajectory framework trained with Behavior Cloning on simulated demonstrations, augmented by GPT-4 paraphrases and a diffusion-policy trajectory generator conditioned on textual embeddings. Key contributions include direct speech-to-trajectory mapping without predefined vocabularies, robust linguistic generalization via semantic augmentation, and embodiment-agnostic applicability through trajectory-based learning. The results show superior trajectory accuracy, robustness to paraphrase variations, and successful real-time guidance on a quadruped, indicating substantial practical impact for intuitive human-robot interaction.

Abstract

Full integration of robots into real-life applications necessitates their ability to interpret and execute natural language directives from untrained users. Given the inherent variability in human language, equivalent directives may be phrased differently, yet require consistent robot behavior. While Large Language Models (LLMs) have advanced language understanding, they often falter in handling user phrasing variability, rely on predefined commands, and exhibit unpredictable outputs. This letter introduces the Directive Language Model (DLM), a novel speech-to-trajectory framework that directly maps verbal commands to executable motion trajectories, bypassing predefined phrases. DLM utilizes Behavior Cloning (BC) on simulated demonstrations of human-guided robot motion. To enhance generalization, GPT-based semantic augmentation generates diverse paraphrases of training commands, labeled with the same motion trajectory. DLM further incorporates a diffusion policy-based trajectory generation for adaptive motion refinement and stochastic sampling. In contrast to LLM-based methods, DLM ensures consistent, predictable motion without extensive prompt engineering, facilitating real-time robotic guidance. As DLM learns from trajectory data, it is embodiment-agnostic, enabling deployment across diverse robotic platforms. Experimental results demonstrate DLM's improved command generalization, reduced dependence on structured phrasing, and achievement of human-like motion.

Figures (6)

• Figure 1: A user guides a quadruped robot to a target position using natural verbal commands. The proposed Directive Language Model (DLM) interprets these commands and translates them into motion trajectories. The same behavior can be triggered by differently phrased instructions, demonstrating the model’s ability to generalize across linguistic variations.
• Figure 2: Illustration of the proposed Directive Language Model (DLM) framework.
• Figure 3: Simulation environments in NVidia IsaacSim simulator. In these example scenarios, robot motion responses to commands (a) Move forward four meters, (b) Go back three meters and (c) Go right five meters, are demonstrated by a human driver.
• Figure 4: Mean (black curve) and standard deviations (blue) of outputted trajectories to paraphrases of the commands (a) 'Move forward 5 meters', (b) 'Move 4 meters to the left' and (c) 'Turn slightly right', given in Table \ref{['tb:command_variations']}. The red arrows indicate the means of the required robot orientations along the paths.
• Figure 5: Sequential robot navigation towards a target square, achieved through four iterations of verbal command input processed by the DLM.