Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Movement Primitives: Grounding Language Models in Robot Motion

Yinlong Dai, Benjamin A. Christie, Daniel J. Evans, Dylan P. Losey, Simon Stepputtis

TL;DR

This work tackles the challenge of grounding natural-language instructions in embodied robot motion. It introduces Language Movement Primitives (LMPs), which ground Vision-Language Model reasoning in Dynamic Movement Primitives, enabling zero-shot manipulation by translating open-ended instructions into semantically meaningful DMP parameters that generate stable trajectories. The framework comprises a state-description extractor, a task decomposer, and a DMP parameter generator, plus a feedback loop using natural-language refinements; experiments on 20 real-world tabletop tasks show substantial improvements over baselines, with an 80% success rate and strong benefits from including both the decomposer and the judge. The approach offers a scalable path toward general-purpose robot control driven by everyday human language, reducing the need for demonstrations while maintaining precise, controllable motion. However, limitations include dependence on semantically interpretable parameters and challenges in dynamic environments, motivating future work on autonomous judgment and richer dynamics modeling.

Abstract

Enabling robots to perform novel manipulation tasks from natural language instructions remains a fundamental challenge in robotics, despite significant progress in generalized problem solving with foundational models. Large vision and language models (VLMs) are capable of processing high-dimensional input data for visual scene and language understanding, as well as decomposing tasks into a sequence of logical steps; however, they struggle to ground those steps in embodied robot motion. On the other hand, robotics foundation models output action commands, but require in-domain fine-tuning or experience before they are able to perform novel tasks successfully. At its core, there still remains the fundamental challenge of connecting abstract task reasoning with low-level motion control. To address this disconnect, we propose Language Movement Primitives (LMPs), a framework that grounds VLM reasoning in Dynamic Movement Primitive (DMP) parameterization. Our key insight is that DMPs provide a small number of interpretable parameters, and VLMs can set these parameters to specify diverse, continuous, and stable trajectories. Put another way: VLMs can reason over free-form natural language task descriptions, and semantically ground their desired motions into DMPs -- bridging the gap between high-level task reasoning and low-level position and velocity control. Building on this combination of VLMs and DMPs, we formulate our LMP pipeline for zero-shot robot manipulation that effectively completes tabletop manipulation problems by generating a sequence of DMP motions. Across 20 real-world manipulation tasks, we show that LMP achieves 80% task success as compared to 31% for the best-performing baseline. See videos at our website: https://collab.me.vt.edu/lmp

Language Movement Primitives: Grounding Language Models in Robot Motion

TL;DR

This work tackles the challenge of grounding natural-language instructions in embodied robot motion. It introduces Language Movement Primitives (LMPs), which ground Vision-Language Model reasoning in Dynamic Movement Primitives, enabling zero-shot manipulation by translating open-ended instructions into semantically meaningful DMP parameters that generate stable trajectories. The framework comprises a state-description extractor, a task decomposer, and a DMP parameter generator, plus a feedback loop using natural-language refinements; experiments on 20 real-world tabletop tasks show substantial improvements over baselines, with an 80% success rate and strong benefits from including both the decomposer and the judge. The approach offers a scalable path toward general-purpose robot control driven by everyday human language, reducing the need for demonstrations while maintaining precise, controllable motion. However, limitations include dependence on semantically interpretable parameters and challenges in dynamic environments, motivating future work on autonomous judgment and richer dynamics modeling.

Abstract

Enabling robots to perform novel manipulation tasks from natural language instructions remains a fundamental challenge in robotics, despite significant progress in generalized problem solving with foundational models. Large vision and language models (VLMs) are capable of processing high-dimensional input data for visual scene and language understanding, as well as decomposing tasks into a sequence of logical steps; however, they struggle to ground those steps in embodied robot motion. On the other hand, robotics foundation models output action commands, but require in-domain fine-tuning or experience before they are able to perform novel tasks successfully. At its core, there still remains the fundamental challenge of connecting abstract task reasoning with low-level motion control. To address this disconnect, we propose Language Movement Primitives (LMPs), a framework that grounds VLM reasoning in Dynamic Movement Primitive (DMP) parameterization. Our key insight is that DMPs provide a small number of interpretable parameters, and VLMs can set these parameters to specify diverse, continuous, and stable trajectories. Put another way: VLMs can reason over free-form natural language task descriptions, and semantically ground their desired motions into DMPs -- bridging the gap between high-level task reasoning and low-level position and velocity control. Building on this combination of VLMs and DMPs, we formulate our LMP pipeline for zero-shot robot manipulation that effectively completes tabletop manipulation problems by generating a sequence of DMP motions. Across 20 real-world manipulation tasks, we show that LMP achieves 80% task success as compared to 31% for the best-performing baseline. See videos at our website: https://collab.me.vt.edu/lmp
Paper Structure (15 sections, 10 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 15 sections, 10 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Small Expert Models for Robot Learning
  4. Robotics Foundation Models
  5. Hybrid Models
  6. Problem Statement
  7. Language Movement Primitives (LMPs)
  8. From Observations to State Descriptions
  9. From State Descriptions to Decomposed Subtasks
  10. From Subtasks to Generating DMPs
  11. Using Feedback to Refine the Motion
  12. Experiments
  13. Baseline Comparison
  14. Ablation Study
  15. Conclusion

Figures (4)

  • Figure 1: Overview of Language Movement Primitives (LMP). Given a task description from the user, LMP first detects objects in the environment and composes a suitable sequence of high-level subtasks to achieve the overall goal. For each subtask, LMP then generates low-level parameters to define a Dynamic Movement Primitive (DMP). The robot tracks the continuous DMP trajectory, grounding its semantic reasoning for zero-shot robot manipulation.
  • Figure 2: LMP pipeline for a single subtask rollout. (a) The robot begins with a user-provided task description. The robot then collects an image capturing the current environment state, and remembers any previously performed subtask(s). (b) The decomposer $\pi_\mathcal{D}$ identifies scene objects and outputs a subtask for the next DMP to complete. An open-vocabulary classifier and depth sensing are used to estimate 3D object locations. The scene description and proposed subtask are then forwarded to the DMP weight generator $\pi_\mathcal{G}$. (c) The generator predicts DMP weights and auxiliary parameters to define the low-level reference trajectory. (d) The robot tracks the continuous trajectory generated from the predicted DMP parameters. Optionally, the user may observe the robot and provide natural-language feedback about any mistakes. If the user gives this refinement $r$, then the robot resets the rollout and the process repeats from (b).
  • Figure 3: Our experiments evaluate LMP's performance on tabletop-manipulation tasks, converting natural-language task descriptions into robot controllers. In our tests, we evaluate 20 diverse household tasks requiring semantic task understanding, awareness of obstacles, and spatial reasoning.
  • Figure 4: We identify five dominant failure modes and analyze the impact of the judge as well as the task decomposition on these failure modes across task executions. Particularly the addition of a grounded task-decomposition substantially reduces the errors in task planning and subsequently more effective weight generation. Furthermore, we find that adding feedback has a strong influence on successful DMP weight generation.