Leveraging Pre-trained Large Language Models with Refined Prompting for Online Task and Motion Planning

TL;DR

The paper tackles robust long-horizon robot task planning under uncertainty by integrating a pre-trained Large Language Model (LLM) with a traditional Task and Motion Planning (TAMP) framework. It introduces First Look Prompting (FLP) to focus the LLM on anomaly information and generate PDDL goals for replanning, while executing using Conditional SubTrees (CSubBTs) that explore constraint space. Empirical validation in both simulation (Gazebo) and real-world (DOBOT/Panda) demonstrates improved robustness and reduced initial planning time compared with a pure PDDLStream baseline. The results indicate that LLMs can effectively complement classical planners for handling unexpected environmental changes without fully replacing planner capabilities.

Abstract

With the rapid advancement of artificial intelligence, there is an increasing demand for intelligent robots capable of assisting humans in daily tasks and performing complex operations. Such robots not only require task planning capabilities but must also execute tasks with stability and robustness. In this paper, we present a closed-loop task planning and acting system, LLM-PAS, which is assisted by a pre-trained Large Language Model (LLM). While LLM-PAS plans long-horizon tasks in a manner similar to traditional task and motion planners, it also emphasizes the execution phase of the task. By transferring part of the constraint-checking process from the planning phase to the execution phase, LLM-PAS enables exploration of the constraint space and delivers more accurate feedback on environmental anomalies during execution. The reasoning capabilities of the LLM allow it to handle anomalies that cannot be addressed by the robust executor. To further enhance the system's ability to assist the planner during replanning, we propose the First Look Prompting (FLP) method, which induces LLM to generate effective PDDL goals. Through comparative prompting experiments and systematic experiments, we demonstrate the effectiveness and robustness of LLM-PAS in handling anomalous conditions during task execution.

Figures (6)

• Figure 1: Overview of LLM-PAS. PDDLStream planner generates TAMP solutions, which are subsequently converted into CSubBTs for execution. When CSubBTs encounter unsolvable anomalies, they provide feedback on the abnormal information. This anomaly information, along with other relevant context, is sent to the LLM module using FLP. The LLM module then processes this information and outputs new PDDL task goals to facilitate the replanning process.
• Figure 2: Two example actions of the domain.pddl file. The preconditions incorporate the static literals ( BaseMotion and Kin).
• Figure 3: Two example streams of the stream.pddl file. The streams generate the static literals ( BaseMotion and Kin).
• Figure 4: The initial planning time over 5 trials per problem size.
• Figure 5: The systematic experiments of LLM-PAS in simulation.