Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners

Zhi Zheng, Qian Feng, Hang Li, Alois Knoll, Jianxiang Feng

TL;DR

This paper addresses the fragility of open-loop LLM planning in robotic manipulation by introducing KnowLoop, a closed-loop planner equipped with an uncertainty-based failure detector that is model-agnostic. It systematically evaluates three uncertainty metrics—token probability, entropy, and self-explained confidence—across three prompting strategies (SSC, SRA, NAC) and uses a threshold $\delta$ to abstain uncertain predictions and request human input. Through a self-collected dataset and real hardware experiments with LLaVA and ChatGPT-4V, token probability and entropy emerge as reliable indicators of accuracy, enabling substantial improvements in task success over baselines that fully trust MLLMs. The results demonstrate the practical viability of uncertainty-guided failure detection for robust, generalizable closed-loop planning in robotic manipulation, with future directions toward resource-efficient planning and statistically-grounded thresholding.

Abstract

Recently, Large Language Models (LLMs) have witnessed remarkable performance as zero-shot task planners for robotic manipulation tasks. However, the open-loop nature of previous works makes LLM-based planning error-prone and fragile. On the other hand, failure detection approaches for closed-loop planning are often limited by task-specific heuristics or following an unrealistic assumption that the prediction is trustworthy all the time. As a general-purpose reasoning machine, LLMs or Multimodal Large Language Models (MLLMs) are promising for detecting failures. However, However, the appropriateness of the aforementioned assumption diminishes due to the notorious hullucination problem. In this work, we attempt to mitigate these issues by introducing a framework for closed-loop LLM-based planning called KnowLoop, backed by an uncertainty-based MLLMs failure detector, which is agnostic to any used MLLMs or LLMs. Specifically, we evaluate three different ways for quantifying the uncertainty of MLLMs, namely token probability, entropy, and self-explained confidence as primary metrics based on three carefully designed representative prompting strategies. With a self-collected dataset including various manipulation tasks and an LLM-based robot system, our experiments demonstrate that token probability and entropy are more reflective compared to self-explained confidence. By setting an appropriate threshold to filter out uncertain predictions and seek human help actively, the accuracy of failure detection can be significantly enhanced. This improvement boosts the effectiveness of closed-loop planning and the overall success rate of tasks.

Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners

TL;DR

This paper addresses the fragility of open-loop LLM planning in robotic manipulation by introducing KnowLoop, a closed-loop planner equipped with an uncertainty-based failure detector that is model-agnostic. It systematically evaluates three uncertainty metrics—token probability, entropy, and self-explained confidence—across three prompting strategies (SSC, SRA, NAC) and uses a threshold to abstain uncertain predictions and request human input. Through a self-collected dataset and real hardware experiments with LLaVA and ChatGPT-4V, token probability and entropy emerge as reliable indicators of accuracy, enabling substantial improvements in task success over baselines that fully trust MLLMs. The results demonstrate the practical viability of uncertainty-guided failure detection for robust, generalizable closed-loop planning in robotic manipulation, with future directions toward resource-efficient planning and statistically-grounded thresholding.

Abstract

Recently, Large Language Models (LLMs) have witnessed remarkable performance as zero-shot task planners for robotic manipulation tasks. However, the open-loop nature of previous works makes LLM-based planning error-prone and fragile. On the other hand, failure detection approaches for closed-loop planning are often limited by task-specific heuristics or following an unrealistic assumption that the prediction is trustworthy all the time. As a general-purpose reasoning machine, LLMs or Multimodal Large Language Models (MLLMs) are promising for detecting failures. However, However, the appropriateness of the aforementioned assumption diminishes due to the notorious hullucination problem. In this work, we attempt to mitigate these issues by introducing a framework for closed-loop LLM-based planning called KnowLoop, backed by an uncertainty-based MLLMs failure detector, which is agnostic to any used MLLMs or LLMs. Specifically, we evaluate three different ways for quantifying the uncertainty of MLLMs, namely token probability, entropy, and self-explained confidence as primary metrics based on three carefully designed representative prompting strategies. With a self-collected dataset including various manipulation tasks and an LLM-based robot system, our experiments demonstrate that token probability and entropy are more reflective compared to self-explained confidence. By setting an appropriate threshold to filter out uncertain predictions and seek human help actively, the accuracy of failure detection can be significantly enhanced. This improvement boosts the effectiveness of closed-loop planning and the overall success rate of tasks.
Paper Structure (18 sections, 4 figures, 3 tables, 2 algorithms)

This paper contains 18 sections, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Uncertainty Estimation in Robotics
  4. Uncertainty Estimation in LLMs
  5. Closed-loop planning with LLMs
  6. The Proposed Closed-Loop Planner
  7. Uncertainty-based Failure Detection
  8. Prompting Strategies
  9. Uncertainty Quantification
  10. Closed-loop Planning
  11. Experiment
  12. Dataset
  13. Evaluation Metrics
  14. System Setup
  15. Results
  16. ...and 3 more sections

Figures (4)

  • Figure 1: A Sketch of our proposed closed-loop planning framework, KnowLoop featured by an uncertainty-based failure detection module, allowing robots to know when to close the loop properly during planning.
  • Figure 2: A flow diagram of the proposed framework for the task: opening the drawer.
  • Figure 3: Example Images for different tasks in the self-collected dataset, from left to right: picking up a mouse, placing a sponge on the notebook, placing a smartphone in the drawer, opening the drawer, and closing the drawer.
  • Figure 4: Uncertainty Accuracy Curve (left column) and Selective Generation Curve (right column) for three prompting strategies based on entropy (top row) and token probability (bottom row).