Generalizable Long-Horizon Manipulations with Large Language Models
Haoyu Zhou, Mingyu Ding, Weikun Peng, Masayoshi Tomizuka, Lin Shao, Chuang Gan
TL;DR
The paper addresses generalization in long-horizon robotic manipulation by using large language models to generate task conditions that connect primitive actions. It combines LLM-generated pre/post-conditions, semantic PointCloud constraints, and Dynamic Movement Primitives to produce robust end-effector trajectories, evaluated on a PyBullet-based benchmark and real robots. Key contributions include a modular learning-from-demonstration framework, a condition-driven task reproduction pipeline, and demonstrated improvements in unseen-object and unseen-task scenarios. The work highlights the potential of LLMs to enhance robotic adaptability and provide a scalable path toward more versatile manipulation systems.
Abstract
This work introduces a framework harnessing the capabilities of Large Language Models (LLMs) to generate primitive task conditions for generalizable long-horizon manipulations with novel objects and unseen tasks. These task conditions serve as guides for the generation and adjustment of Dynamic Movement Primitives (DMP) trajectories for long-horizon task execution. We further create a challenging robotic manipulation task suite based on Pybullet for long-horizon task evaluation. Extensive experiments in both simulated and real-world environments demonstrate the effectiveness of our framework on both familiar tasks involving new objects and novel but related tasks, highlighting the potential of LLMs in enhancing robotic system versatility and adaptability. Project website: https://object814.github.io/Task-Condition-With-LLM/