Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations

TL;DR

The paper critiques the use of LLM-simulated users for scaling agentic evaluations, showing substantial robustness, validity, and fairness gaps across demographics. By conducting a cross-national user study (US, India, Kenya, Nigeria) on tau-Bench retail tasks with multiple user models, it demonstrates that agent performance can vary by up to about $9\%$ depending on the simulated user model, and that simulations exhibit systematic miscalibration relative to human users across difficulty levels. It also reveals demographic biases, with AAVE and Indian English participants experiencing worse outcomes and calibration, and demonstrates that simulated conversations introduce artifacts such as heightened politeness and more question-asking. The findings argue for evaluating across multiple simulation models, validating against diverse human data when possible, and adopting transparent reporting to avoid biased or misleading conclusions in real-world deployments.

Abstract

Agentic benchmarks increasingly rely on LLM-simulated users to scalably evaluate agent performance, yet the robustness, validity, and fairness of this approach remain unexamined. Through a user study with participants across the United States, India, Kenya, and Nigeria, we investigate whether LLM-simulated users serve as reliable proxies for real human users in evaluating agents on τ-Bench retail tasks. We find that user simulation lacks robustness, with agent success rates varying up to 9 percentage points across different user LLMs. Furthermore, evaluations using simulated users exhibit systematic miscalibration, underestimating agent performance on challenging tasks and overestimating it on moderately difficult ones. African American Vernacular English (AAVE) speakers experience consistently worse success rates and calibration errors than Standard American English (SAE) speakers, with disparities compounding significantly with age. We also find simulated users to be a differentially effective proxy for different populations, performing worst for AAVE and Indian English speakers. Additionally, simulated users introduce conversational artifacts and surface different failure patterns than human users. These findings demonstrate that current evaluation practices risk misrepresenting agent capabilities across diverse user populations and may obscure real-world deployment challenges.

Figures (5)

• Figure 1: Conversational snippets between an agent and simulated (top) vs. human user (bottom) on the same task. Simulated users exhibit increased question-asking and politeness compared to human users.
• Figure 2: Success rate with human users vs. LLM-simulated users for United States participants, with an $ECE_\text{Human--LLM}$ of $15.1$. Error bars indicate $\pm$1 SD.
• Figure 3: Success rate with human users vs. LLM-simulated users for SAE (top) and AAVE (bottom) participants across different age groups (18-34, 35-54, and 55+). Note: The x-axis (success rate with simulated users) remains the same for all groups.
• Figure 4: Success rate with human users vs. LLM-simulated users across dialect and country groups. All participants are in the 18--34 age group.
• Figure 5: User study chat interface where participants interact with the agent (GPT-4o) to complete tasks.