Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation

TL;DR

This work presents Instruction-Augmented Long-Horizon Planning (IALP), a framework that couples LLM-based planning with grounding mechanisms to produce feasible and optimal action sequences for long-horizon mobile manipulation in open-world environments. By augmenting user instructions into PDDL problems with both promptable and grounding predicates derived from embodied perception, IALP enables closed-loop, real-time planning and execution, validated in five real-world tasks with over 80% success. Ablation studies confirm the crucial role of feasibility feedback and action selection based on language-model token probabilities, while analyses of failure modes guide future improvements. The approach advances autonomy in unstructured settings by integrating multi-modal sensing (2D/3D data, voxel maps) with structured symbolic planning, offering practical impact for robot assistants in real environments.

Abstract

Enabling humanoid robots to perform long-horizon mobile manipulation planning in real-world environments based on embodied perception and comprehension abilities has been a longstanding challenge. With the recent rise of large language models (LLMs), there has been a notable increase in the development of LLM-based planners. These approaches either utilize human-provided textual representations of the real world or heavily depend on prompt engineering to extract such representations, lacking the capability to quantitatively understand the environment, such as determining the feasibility of manipulating objects. To address these limitations, we present the Instruction-Augmented Long-Horizon Planning (IALP) system, a novel framework that employs LLMs to generate feasible and optimal actions based on real-time sensor feedback, including grounded knowledge of the environment, in a closed-loop interaction. Distinct from prior works, our approach augments user instructions into PDDL problems by leveraging both the abstract reasoning capabilities of LLMs and grounding mechanisms. By conducting various real-world long-horizon tasks, each consisting of seven distinct manipulatory skills, our results demonstrate that the IALP system can efficiently solve these tasks with an average success rate exceeding 80%. Our proposed method can operate as a high-level planner, equipping robots with substantial autonomy in unstructured environments through the utilization of multi-modal sensor inputs.

Figures (11)

• Figure 1: The IALP system leverages the reasoning capabilities of LLMs and grounding mechanisms to enrich the task representation, enabling plan feasible and optimal actions in the real world.
• Figure 2: The proposed IALP system is designed to complete long-horizon mobile manipulation tasks in real world environment. Firstly, it constructs a PDDL problem for the task at current state by utilizing promptable and ground truth predicates derived from embodied perception and the skill library. Then, the robot executes the actions generated and selected by the LLM planner based on the constructed PDDL problem. The system operates in a closed-loop manner until the task is completed.
• Figure 3: The room environment we used in our tasks. The position of the yellow star is the initial state of the robot.
• Figure 4: The states, feasibility feedback, and actions during the execution of long-horizon mobile manipulation tasks.
• Figure 5: The token probability of action candidates generated by the LLM planner for five long-horizon tasks.