Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

TL;DR

This work reframes LLM evaluation from static accuracy over large item pools to latent ability estimation via Item Response Theory, enabling Fisher-information-guided adaptive item selection and efficient termination. By calibrating a 3PL model, filtering for informative items, and applying common-person linking, ATLAS achieves substantial reductions in test length (about 90%) while maintaining precision and contamination resistance. Across five benchmarks, ATLAS reveals systematic rank shifts relative to accuracy-based scoring and identifies a minority of items with negative discrimination, underscoring the benefits of psychometric evaluation for robust comparisons. The approach offers scalable, form-invariant, and uncertainty-aware model assessment with practical implications for benchmark design and ongoing evaluation of evolving LLM capabilities.

Abstract

Large language model evaluation requires thousands of benchmark items, making evaluations expensive and slow. Existing methods compute average accuracy across fixed item sets, treating all items equally despite varying quality and informativeness. We present ATLAS an adaptive testing framework using Item Response Theory (IRT) to estimate model ability through Fisher information-guided item selection. Our analysis of five major benchmarks reveals that 3-6% of items exhibit negative discrimination, indicating annotation errors that corrupt static evaluation. ATLAS achieves 90% item reduction while maintaining measurement precision: on HellaSwag (5,608 items), we match full-benchmark estimates using only 42 items with 0.154 MAE. Our framework maintains item exposure rates below 10% and test overlap at 16-27%, compared to static benchmarks where every model sees all items (100% exposure). Among 4,000+ tested models, IRT ranks differ from accuracy ranks: models with the same accuracy get different IRT scores, and 23-31% of all models shift by more than 10 rank positions. Code and calibrated item banks are available at https://github.com/Peiyu-Georgia-Li/ATLAS.git.

Figures (8)

• Figure 1: Comparison of subset (predicted) ability estimates against whole-bank (reference) abilities across five benchmarks. Points along the diagonal indicate perfect agreement. ATLAS maintains the closest alignment overall, particularly on TruthfulQA, ARC, HellaSwag and in the low-ability regime of GSM8K, while static baselines such as TinyBenchmarks and random show greater variance and systematic deviation.
• Figure 2: Comparison of IRT ability estimates $\hat{\theta}_\ell^{\text{whole}}$ with raw accuracy. Left: Ability vs accuracy reveals strong correlation but critical differences at performance extremes where accuracy collapses. Right: Rank comparison shows systematic reordering, with 23% (GSM8K) and 31% (HellaSwag) of models shifting $>$ 10 positions. IRT separates models with identical accuracies by accounting for which items they solve correctly.
• Figure 3: Two models with identical accuracy (0.833) on WinoGrande receive different ability estimates ($\hat{\theta}_A = 1.2$ vs $\hat{\theta}_B = 0.6$). Model A succeeds on harder items (darker cells on right), while Model B answers easier items (darker cells on left). IRT captures these item difficulty patterns that raw accuracy cannot.
• Figure 4: Comparison of raw average scores and whole-bank ability estimates on TruthfulQA. (Left) While average scores compress performance at the extremes, whole-bank ability estimates reveal clearer separation among both low- and high-performing models, reflecting sensitivity to item difficulty and discrimination. (Right) Rank comparison shows strong consistency between the two measures (Spearman $\rho = 0.97$, Kendall $\tau = 0.87$), but ability-based ranking provides finer resolution, especially in distinguishing weaker and stronger models beyond what raw accuracy captures.
• Figure 5: Comparison of raw average scores and whole-bank ability estimates on WinoGrande. (Left) Whole-bank estimates show a non-linear relationship with average score and reveal clearer separation on high-performing models, highlighting that ability captures relative item difficulty and provides finer differentiation beyond raw accuracy. (Right) Rank comparison indicates strong but imperfect alignment (Spearman $\rho = 0.86$, Kendall $\tau = 0.70$), with deviations from the diagonal reflecting cases where ability-based ranking distinguishes models more effectively than accuracy alone.