SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?

TL;DR

SimulatorArena presents a benchmark for evaluating user simulators in multi-turn AI interactions, built on 909 authentic human–LLM conversations across math tutoring and document creation. It formalizes the evaluation framework with a three-party dialogue (user simulator, assistant, rater) and investigates prompting strategies, including profile-conditioned simulations, to improve realism and alignment with human judgments. Empirical results show that detailed user profiles significantly enhance correlation with human assessments (up to 0.7 Spearman's ρ) at a fraction of human evaluation cost. By benchmarking 18 assistants, including GPT-5 and Claude variants, the work demonstrates that carefully designed simulators can serve as scalable proxies for human evaluation, informing model development and selection in practical, real-world settings.

Abstract

Large language models (LLMs) are increasingly used in interactive applications, and human evaluation remains the gold standard for assessing their performance in multi-turn conversations. Since human studies are costly, time-consuming, and hard to reproduce, recent work explores using LLMs to simulate users for automatic assistant evaluation. However, there is no benchmark or systematic study to evaluate whether these simulated users are reliable stand-ins for real users. To address this, we introduce SimulatorArena, a benchmark of 909 annotated human-LLM conversations on two interactive tasks -- math tutoring and document creation. SimulatorArena evaluates simulators based on how closely their messages match human behavior and how well their assistant ratings align with human judgments. Experiments on various simulator methods show that simulators conditioned on user profiles, capturing traits like background and message styles, align closely with human judgments. They reach Spearman's $ρ$ of 0.7 on both tasks, providing a practical, scalable alternative to human evaluation. Using the best simulator for each task, we benchmark 18 assistants, including the latest LLMs such as GPT-5, Claude 4.1 Opus, and Gemini 2.5 Pro.

Figures (36)

• Figure 1: SimulatorArena systematically evaluates user simulators by comparing their behavior to humans'. User profiles improve simulator quality, offering an efficient, scalable alternative to human evaluation.
• Figure 2: Left: Example math tutoring conversations with a zero-shot-cot simulator, two profile-based simulators with different user profiles, and a real human user. Assistant responses are summarized for space. Takeaways: (1) The profile-based simulator produces messages that better resemble human users than the zero-shot-cot baseline; (2) Different user profiles lead to different conversation flows and outcomes. Right: Correlation between simulator and human ratings of assistant performance, computed over 27 grouped data points (model × difficulty level), shown as scatter plots. Takeaway: User profile-based simulator significantly improves correlation with human judgments from 0.61 Spearman’s $\rho$ to 0.77. Full conversations and document creation examples are in Appendix \ref{['app:examples']}.
• Figure 3: Annotation Process. To elicit authentic human-AI conversations, our interface follows a three‑step workflow: (1) We curate a bank of hundreds of math problems and common document topics. (2) Annotators select the problem or topic that interests them and answer brief pre‑chat questions to familiarize themselves with the task, an especially important step for document creation, where initial content ideas and context guide writing. (3) They then converse with the AI assistant and, upon finishing the dialogue, evaluate its performance.
• Figure 4: Evaluation of different LLMs as raters $\pi_r$ based on their alignment with human ratings for final documents in the document creation task.
• Figure 5: Evaluation of different LLMs as raters $\pi_r$ for self-bias. All models show no evidence of self-bias.