HiCRISP: An LLM-based Hierarchical Closed-Loop Robotic Intelligent Self-Correction Planner

TL;DR

This work addresses the lack of stepwise self-correction in LLM-based robotic planning by introducing HiCRISP, a Hierarchical Closed-loop Robotic Intelligent Self-correction Planner. It frames task execution as a finite MDP, with the LLM decomposing instructions into a state-action path $igl\langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R} \bigr\rangle$ and using perception to detect high-level and low-level failures, while employing a stack-based, depth-limited error-correction mechanism. High-level feedback re-plans via rectified actions $a_{\text{correction}}$ to bridge from $s_{\text{error}}$ toward the desired $s_{i+1}$; low-level feedback relies on movement primitives with built-in error checks, invoking LLM-driven corrections only for novel issues. Across virtual (Virtual Home) and robot-simulation (Gazebo, Bullet) setups, and real-world AGV experiments, HiCRISP achieves higher execution and success rates than baselines like ProgPrompt, demonstrating improved robustness and adaptability in dynamic environments.

Abstract

The integration of Large Language Models (LLMs) into robotics has revolutionized human-robot interactions and autonomous task planning. However, these systems are often unable to self-correct during the task execution, which hinders their adaptability in dynamic real-world environments. To address this issue, we present a Hierarchical Closed-loop Robotic Intelligent Self-correction Planner (HiCRISP), an innovative framework that enables robots to correct errors within individual steps during the task execution. HiCRISP actively monitors and adapts the task execution process, addressing both high-level planning and low-level action errors. Extensive benchmark experiments, encompassing virtual and real-world scenarios, showcase HiCRISP's exceptional performance, positioning it as a promising solution for robotic task planning with LLMs.

Figures (5)

• Figure 1: Illustration of LLM-based robotic systems: (1) without correction, (2) correction after task completion, and (3) correction within steps. An error occurs during the execution of Action 3. Our proposed HiCRISP belongs to the system that performs corrections within individual steps.
• Figure 2: Overview of our proposed HiCRISP framework. The LLM dissects the user input into various actions using pertinent details from the actuator, scene, and other sources. We utilize predefined code to translate actions into movement primitives that are easily executable by the robot. Perception detects environment information and judges whether the system state changes. If a failure is detected, the system fixes plan failure and action failure through high-level feedback in \ref{['subsec: high-level']} and low-level feedback in \ref{['subsec: low-level']}, respectively.
• Figure 3: HiCRISP demonstrates superior performance in a VH simulator when compared to ProgPromptsingh2023progprompt. In situations where the agent loses its position on the sofa and is unable to execute the command "Sit sofa," HiCRISP exhibits the capability to learn from the error message provided by the simulator and take corrective action by executing the command "Find sofa." This corrective action not only addresses the error but also guides the agent to reach the desired final state.
• Figure 4: HiCRISP acts in Gazebo simulator. LLM breaks down the input: "Place a box around a cylinder" into four steps: (1) locate a box; (2) pick up the box; (3) find a cylinder; (4) place the box around the cylinder.
• Figure 5: HiCRISP operates on both the Gazebo simulator and real-world AGV platform. We command the vehicle to sequentially approach specific landmarks according to a predetermined order. Action failure arises when obstacles obstruct the intended trajectory. Planning failure occurs if the vehicle erroneously navigates towards an incorrect landmark. HiCRISP addresses these failures by providing corresponding corrective actions and rectifying the issues.