Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant

Gaole He, Gianluca Demartini, Ujwal Gadiraju

TL;DR

The paper examines how user involvement in planning and executing with plan-then-execute LLM agents affects trust and task performance across six risk-diverse tasks. It uses a 2×2 factorial design (automatic vs. user-involved planning and execution) in a simulated environment (N=248) to assess calibrated trust and execution outcomes, finding that high-quality plans plus execution involvement can improve results, but plausibly correct-looking plans can mislead trust. Execution involvement shows more consistent performance benefits than planning involvement, though neither strategy universally calibrates trust. The work provides design insights for flexible, trust-aware human–AI workflows and highlights the risks of convincingly wrong LLM outputs in daily assistive settings.

Abstract

Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose (e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of 'LLM-modulo' setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study (N = 248) of LLM agents as daily assistants in six commonly occurring tasks with different levels of risk typically associated with them (e.g., flight ticket booking and credit card payments). To ensure user agency and control over the LLM agent, we adopted LLM agents in a plan-then-execute manner, wherein the agents conducted step-wise planning and step-by-step execution in a simulation environment. We analyzed how user involvement at each stage affects their trust and collaborative team performance. Our findings demonstrate that LLM agents can be a double-edged sword -- (1) they can work well when a high-quality plan and necessary user involvement in execution are available, and (2) users can easily mistrust the LLM agents with plans that seem plausible. We synthesized key insights for using LLM agents as daily assistants to calibrate user trust and achieve better overall task outcomes. Our work has important implications for the future design of daily assistants and human-AI collaboration with LLM agents.

Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant

TL;DR

The paper examines how user involvement in planning and executing with plan-then-execute LLM agents affects trust and task performance across six risk-diverse tasks. It uses a 2×2 factorial design (automatic vs. user-involved planning and execution) in a simulated environment (N=248) to assess calibrated trust and execution outcomes, finding that high-quality plans plus execution involvement can improve results, but plausibly correct-looking plans can mislead trust. Execution involvement shows more consistent performance benefits than planning involvement, though neither strategy universally calibrates trust. The work provides design insights for flexible, trust-aware human–AI workflows and highlights the risks of convincingly wrong LLM outputs in daily assistive settings.

Abstract

Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose (e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of 'LLM-modulo' setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study (N = 248) of LLM agents as daily assistants in six commonly occurring tasks with different levels of risk typically associated with them (e.g., flight ticket booking and credit card payments). To ensure user agency and control over the LLM agent, we adopted LLM agents in a plan-then-execute manner, wherein the agents conducted step-wise planning and step-by-step execution in a simulation environment. We analyzed how user involvement at each stage affects their trust and collaborative team performance. Our findings demonstrate that LLM agents can be a double-edged sword -- (1) they can work well when a high-quality plan and necessary user involvement in execution are available, and (2) users can easily mistrust the LLM agents with plans that seem plausible. We synthesized key insights for using LLM agents as daily assistants to calibrate user trust and achieve better overall task outcomes. Our work has important implications for the future design of daily assistants and human-AI collaboration with LLM agents.

Paper Structure

This paper contains 35 sections, 2 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Human-AI Collaboration
  4. Trust and Reliance on AI systems
  5. Task Support with LLMs and LLM Agents
  6. Method
  7. Overview of User Involvement in Plan-then-execute LLM Agents
  8. Planning
  9. Execution
  10. Hypotheses
  11. Study Design
  12. Experimental Conditions
  13. Tasks
  14. Measures and Variables
  15. Participants
  16. ...and 20 more sections

Figures (4)

  • Figure 1: Screenshot of user-involved planning interface.
  • Figure 2: User-involved execution flow chart and interface. Panel (a): a flow chart illustrating how each primary step is executed with two stages: action prediction and action execution. Panel (b): a screenshot of the conversation interface for user-involved execution.
  • Figure 3: Bar plot for cognitive load across all conditions. ** indicates significance ($p < 0.0125$) through post-hoc Tukey HSD test. The error bars represent the 95% confidence interval.
  • Figure 4: Bar plot for confidence dynamics, the x-axis denotes the task ordering index (shuffled for every participant). The error bars represent the 95% confidence interval.