Grounding LLMs For Robot Task Planning Using Closed-loop State Feedback

TL;DR

This work addresses the fragility of LLM-based robotic task planning by grounding high-level reasoning in the robot’s environment through a two-LLM, brain–body architecture (Brain-LLM and Body-LLM) with closed-loop state feedback. The approach uses environmental state representations and runtime error feedback to iteratively revise plans, reducing hallucinations and increasing task success across VirtualHome simulations and Franka Arm experiments, with GPT-4 generally performing best. Key contributions include the BrainBody-LLM framework, a structured planning/execution/feedback prompting setup, and extensive evaluation showing improved SR and GCR over baselines, plus practical guidelines for adapting to diverse robotic settings. The results highlight the potential of learning from simulator/controller errors to achieve more reliable, autonomous robot task execution and pave the way for safer, grounded LLM-driven autonomy in real-world robotics.

Abstract

Planning algorithms decompose complex problems into intermediate steps that can be sequentially executed by robots to complete tasks. Recent works have employed Large Language Models (LLMs) for task planning, using natural language to generate robot policies in both simulation and real-world environments. LLMs like GPT-4 have shown promising results in generalizing to unseen tasks, but their applicability is limited due to hallucinations caused by insufficient grounding in the robot environment. The robustness of LLMs in task planning can be enhanced with environmental state information and feedback. In this paper, we introduce a novel approach to task planning that utilizes two separate LLMs for high-level planning and low-level control, improving task-related success rates and goal condition recall. Our algorithm, \textit{BrainBody-LLM}, draws inspiration from the human neural system, emulating its brain-body architecture by dividing planning across two LLMs in a structured, hierarchical manner. BrainBody-LLM implements a closed-loop feedback mechanism, enabling learning from simulator errors to resolve execution errors in complex settings. We demonstrate the successful application of BrainBody-LLM in the VirtualHome simulation environment, achieving a 29\% improvement in task-oriented success rates over competitive baselines with the GPT-4 backend. Additionally, we evaluate our algorithm on seven complex tasks using a realistic physics simulator and the Franka Research 3 robotic arm, comparing it with various state-of-the-art LLMs. Our results show advancements in the reasoning capabilities of recent LLMs, which enable them to learn from raw simulator/controller errors to correct plans, making them highly effective in robotic task planning.

Figures (7)

• Figure 1: Illustration of how two LLMs work together in the proposed algorithm: The Brain-LLM splits the given task, 'Eat chips on the sofa', into sequential steps using its real-world knowledge. The Body-LLM takes these steps one-by-one and determines executable actions. In instances where a corresponding action is not found in the environment, as demonstrated in the final step of this example, the Body-LLM outputs a $\textless$pass$\textgreater$ token.
• Figure 2: Format of the planning prompt used in our experiments. The planning prompt tunes LLM outputs to meet environmental constraints while generating step-by-step task execution plans. In-context learning examples of high-level tasks and their corresponding subtasks, along with a list of available objects and actions, are needed. This enables the LLM to learn patterns from the examples and create plans for unseen tasks based on the robot's current environment.
• Figure 3: Format of the execution prompt used in our experiments. The execution prompt tunes the Body-LLM to generate appropriate control statements in the required syntax for a given plan created by the Brain-LLM.
• Figure 4: Format of the feedback prompt used in our experiments. The feedback prompt informs the LLM of an execution error and provides examples of how similiar errors can be resolved. The LLM learns from these examples, and uses the given environmental conditions, available actions and creates a new updated plan to resolve the error, conditioned on already executed plans before the error step.
• Figure 5: GPT-4 planning with and without feedback: Example 1 and Example 2 shows refinement of plans with context cues in VH and Franka Arm Simulation, respectively.