Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM Personas as a Substitute for Field Experiments in Method Benchmarking

Enoch Hyunwook Kang

TL;DR

<3-5 sentence high-level summary> The paper tackles the efficiency bottleneck of field experiments for benchmarking societal-methods by proposing LLM-based persona panels as a cheap substitute. It derives an exact identification result: swapping human evaluators with LLM personas is equivalent to a simple panel change, if and only if aggregate-only observation ($AO$) and algorithm-blind evaluation ($AB$) hold, formalized through reduced-form channels $Q_{ ext{hum}}$ and $Q_{ ext{pers}}$. Beyond this, it distinguishes validity from usefulness by introducing discriminability $oldsymbol{ ext{κ}}_Q$ and a sample-size bound that ties the required number of persona evaluations to the information content of the induced channel. The framework provides concrete guidance on when persona benchmarking can reliably substitute field experiments and how large a persona panel must be to meaningfully distinguish and optimize different methods.

Abstract

Field experiments (A/B tests) are often the most credible benchmark for methods in societal systems, but their cost and latency create a major bottleneck for iterative method development. LLM-based persona simulation offers a cheap synthetic alternative, yet it is unclear whether replacing humans with personas preserves the benchmark interface that adaptive methods optimize against. We prove an if-and-only-if characterization: when (i) methods observe only the aggregate outcome (aggregate-only observation) and (ii) evaluation depends only on the submitted artifact and not on the algorithm's identity or provenance (algorithm-blind evaluation), swapping humans for personas is just panel change from the method's point of view, indistinguishable from changing the evaluation population (e.g., New York to Jakarta). Furthermore, we move from validity to usefulness: we define an information-theoretic discriminability of the induced aggregate channel and show that making persona benchmarking as decision-relevant as a field experiment is fundamentally a sample-size question, yielding explicit bounds on the number of independent persona evaluations required to reliably distinguish meaningfully different methods at a chosen resolution.

LLM Personas as a Substitute for Field Experiments in Method Benchmarking

TL;DR

<3-5 sentence high-level summary> The paper tackles the efficiency bottleneck of field experiments for benchmarking societal-methods by proposing LLM-based persona panels as a cheap substitute. It derives an exact identification result: swapping human evaluators with LLM personas is equivalent to a simple panel change, if and only if aggregate-only observation () and algorithm-blind evaluation () hold, formalized through reduced-form channels and . Beyond this, it distinguishes validity from usefulness by introducing discriminability and a sample-size bound that ties the required number of persona evaluations to the information content of the induced channel. The framework provides concrete guidance on when persona benchmarking can reliably substitute field experiments and how large a persona panel must be to meaningfully distinguish and optimize different methods.

Abstract

Field experiments (A/B tests) are often the most credible benchmark for methods in societal systems, but their cost and latency create a major bottleneck for iterative method development. LLM-based persona simulation offers a cheap synthetic alternative, yet it is unclear whether replacing humans with personas preserves the benchmark interface that adaptive methods optimize against. We prove an if-and-only-if characterization: when (i) methods observe only the aggregate outcome (aggregate-only observation) and (ii) evaluation depends only on the submitted artifact and not on the algorithm's identity or provenance (algorithm-blind evaluation), swapping humans for personas is just panel change from the method's point of view, indistinguishable from changing the evaluation population (e.g., New York to Jakarta). Furthermore, we move from validity to usefulness: we define an information-theoretic discriminability of the induced aggregate channel and show that making persona benchmarking as decision-relevant as a field experiment is fundamentally a sample-size question, yielding explicit bounds on the number of independent persona evaluations required to reliably distinguish meaningfully different methods at a chosen resolution.
Paper Structure (53 sections, 9 theorems, 82 equations)

This paper contains 53 sections, 9 theorems, 82 equations.

Table of Contents

  1. Introduction
  2. Related works
  3. Setup: Benchmarking experimentation
  4. knobs, artifacts, and algorithms
  5. Algorithm as an interactive learner.
  6. Evaluation as panel $\times$ instrument $\times$ aggregation
  7. Panel ($P$).
  8. Micro-instrument ($I$).
  9. Aggregation into observed feedback ($\Gamma$, $L$).
  10. What's observed by the method.
  11. Persona benchmark vs. human benchmark
  12. Human benchmark.
  13. Persona benchmark.
  14. What matters for the method.
  15. Key assumptions
  16. ...and 38 more sections

Key Result

Lemma 4.1

Assume Aggregate-only observation (AO), i.e., the algorithm observes only $o_t\in\mathcal{O}$ each round. Let $B=(P,I,\Gamma,L)$ and $B'=(P',I,\Gamma,L)$ differ by a literal panel change in the sense of Definition def:literal_panel_change. Define the reduced-form kernels as in eq:reducedform. Then the swap $B\leftrightarrow B'$ is JPC in the sense of Definition def:jpc_interface, with $Q:=Q_{P,I}

Theorems & Definitions (20)

  • Definition 1: Literal panel change
  • Definition 2: Just panel change (JPC)
  • Lemma 4.1: Literal panel change $\Rightarrow$ JPC
  • Lemma 4.2: JPC $\Rightarrow$ observational equivalence to a literal panel change
  • Lemma 4.3: (JPC) $\iff$ (AO)+(AB)
  • Lemma 5.1
  • Lemma 5.2: Pairwise comparison sample complexity from discriminability
  • proof : Proof of Lemma \ref{['lem:literal_implies_jpc']}
  • proof : Proof of Lemma \ref{['lem:jpc_implies_literal_representation']}
  • proof : Proof of Lemma \ref{['lem:main_new']}
  • ...and 10 more