Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Human Trust in LLM-Based Planners: A Preliminary Study

Shenghui Chen, Yunhao Yang, Kayla Boggess, Seongkook Heo, Lu Feng, Ufuk Topcu

TL;DR

This study investigates how humans trust LLM-based planners relative to classical PDDL solvers in a gripper-domain planning task. By pairing subjective trust judgments with objective evaluation accuracy in a within-subject design (n=30) across four planner conditions, the authors show that correctness predominantly drives both accuracy and trust. Explanations provided with LLM plans can boost evaluation accuracy, but exert limited influence on trust, while plan refinement may increase trust without enhancing correctness, signaling potential overtrust under RLHF-driven outputs. The findings suggest design priorities for human-centered AI planning: prioritize correctness and calibrate trust carefully when deploying LLM-based planners, and explore how explanations and refinement mechanisms interact with user trust in real-world settings.

Abstract

Large Language Models (LLMs) are increasingly used for planning tasks, offering unique capabilities not found in classical planners such as generating explanations and iterative refinement. However, trust--a critical factor in the adoption of planning systems--remains underexplored in the context of LLM-based planning tasks. This study bridges this gap by comparing human trust in LLM-based planners with classical planners through a user study in a Planning Domain Definition Language (PDDL) domain. Combining subjective measures, such as trust questionnaires, with objective metrics like evaluation accuracy, our findings reveal that correctness is the primary driver of trust and performance. Explanations provided by the LLM improved evaluation accuracy but had limited impact on trust, while plan refinement showed potential for increasing trust without significantly enhancing evaluation accuracy.

Evaluating Human Trust in LLM-Based Planners: A Preliminary Study

TL;DR

This study investigates how humans trust LLM-based planners relative to classical PDDL solvers in a gripper-domain planning task. By pairing subjective trust judgments with objective evaluation accuracy in a within-subject design (n=30) across four planner conditions, the authors show that correctness predominantly drives both accuracy and trust. Explanations provided with LLM plans can boost evaluation accuracy, but exert limited influence on trust, while plan refinement may increase trust without enhancing correctness, signaling potential overtrust under RLHF-driven outputs. The findings suggest design priorities for human-centered AI planning: prioritize correctness and calibrate trust carefully when deploying LLM-based planners, and explore how explanations and refinement mechanisms interact with user trust in real-world settings.

Abstract

Large Language Models (LLMs) are increasingly used for planning tasks, offering unique capabilities not found in classical planners such as generating explanations and iterative refinement. However, trust--a critical factor in the adoption of planning systems--remains underexplored in the context of LLM-based planning tasks. This study bridges this gap by comparing human trust in LLM-based planners with classical planners through a user study in a Planning Domain Definition Language (PDDL) domain. Combining subjective measures, such as trust questionnaires, with objective metrics like evaluation accuracy, our findings reveal that correctness is the primary driver of trust and performance. Explanations provided by the LLM improved evaluation accuracy but had limited impact on trust, while plan refinement showed potential for increasing trust without significantly enhancing evaluation accuracy.

Paper Structure

This paper contains 27 sections, 18 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Planning Problem
  4. PDDL Solver
  5. LLM Planner
  6. LLM Planner with Explanation (LLM+Expl)
  7. LLM Planner with Refinement (LLM+Refine)
  8. User Study Design
  9. Participants
  10. Procedure
  11. Independent Variables
  12. Dependent Measures
  13. Hypotheses
  14. Results & Analysis
  15. On Evaluation Accuracy
  16. ...and 12 more sections

Figures (18)

  • Figure 1: Examples where LLM Planner correctly generates a plan for the gripper planning problem.
  • Figure 2: User study procedure.
  • Figure 3: Trust scores on a 7-point Likert scale before and after.
  • Figure 4: Evaluation accuracy measured by the number of correctly evaluated tasks.
  • Figure 5: Propensity to Trust Scale: 4 out of 6 Questions with Statistically Significant Differences Highlighted (*$p<0.05$, **$p<0.01$).
  • ...and 13 more figures