MirrorBench: An Extensible Framework to Evaluate User-Proxy Agents for Human-Likeness

TL;DR

MirrorBench proposes a reproducible framework to evaluate the human-likeness of user-proxy agents independently from task success. It establishes a six-layer architecture with typed interfaces, registries, and a planner to enable reproducible, scalable evaluation across multi-turn dialogues. The evaluation combines lexical-diversity metrics (MATTR, Yule's K, HD-D) and judge-based realism metrics (GTEval, PI, RNR), with calibration controls to account for judge biases, applied to four diverse open datasets. Results reveal dataset- and judge-dependent variations in realism and diversity, highlighting the necessity of calibration and multi-judge reporting for robust conclusions. Overall, MirrorBench provides a practical, extensible benchmark to standardize assessment of human-likeness in user proxies for conversational AI.

Abstract

Large language models (LLMs) are increasingly used as human simulators, both for evaluating conversational systems and for generating fine-tuning data. However, naive "act-as-a-user" prompting often yields verbose, unrealistic utterances, underscoring the need for principled evaluation of so-called user proxy agents. We present MIRRORBENCH, a reproducible, extensible benchmarking framework that evaluates user proxies solely on their ability to produce human-like user utterances across diverse conversational tasks, explicitly decoupled from downstream task success. MIRRORBENCH features a modular execution engine with typed interfaces, metadata-driven registries, multi-backend support, caching, and robust observability. The system supports pluggable user proxies, datasets, tasks, and metrics, enabling researchers to evaluate arbitrary simulators under a uniform, variance-aware harness. We include three lexical-diversity metrics (MATTR, YULE'S K, and HD-D) and three LLM-judge-based metrics (GTEval, Pairwise Indistinguishability, and Rubric-and-Reason). Across four open datasets, MIRRORBENCH yields variance-aware results and reveals systematic gaps between user proxies and real human users. The framework is open source and includes a simple command-line interface for running experiments, managing configurations and caching, and generating reports. The framework can be accessed at https://github.com/SAP/mirrorbench.

Figures (17)

• Figure 1: MirrorBench Architecture: Six-layer stack from low-level execution backends & persistence up through the core engine, plugin components, CLI & reporting, and task drivers. Top layers are user-facing; bottom layers are low-level infrastructure abstractions.
• Figure 2: MirrorBench execution flow. The framework decomposes an evaluation job into units$\{U_1,\ldots,U_p\}$; each unit $U_i$ iterates over episodes$\{e_1,\ldots,e_n\}$ produced by the dataset adapter and executed via task drivers. Metrics are computed per episode and aggregated within each unit with confidence intervals.
• Figure 3: Human-likeness of five user-proxy LLMs across four datasets. Higher is better for judge-based metrics (GTEval, PI, RNR). Lexical-diversity metrics are $z$-scored to human baselines (0 is best). We fix the judge to Claude-4-Sonnet and the assistant to GPT-4o.
• Figure 4: Judge sensitivity of judge-realism metrics on ChatbotArena with assistant & user-proxy both set to GPT-4o; bars vary the judge model. Error bars are 95% CIs.
• Figure 5: Judge–human correlation on ChatbotArena: Correlation of Claude-4-Sonnet judge scores & human scores for GTEval and PI, evaluated on Gemini-2.5-Pro user-proxy outputs (N=100 per metric). Solid line: linear fit; dashed: identity. Point size encodes local sample density. All correlations p $<$ 0.001.