Toward a unified framework for data-efficient evaluation of large language models

TL;DR

This work addresses the high cost of evaluating large language models by introducing LEGO-IRT, a unified Bayesian framework that extends item response theory to both binary and continuous evaluation metrics and explicitly models cross-metric and cross-benchmark structure. It deploys LEGO-CM for continuous metrics and two factorization variants (LEGO-MM and LEGO-MB) to capture general and task-specific abilities, using LKJ priors and MCMC for full posterior inference. Across 70 LLMs and 5 benchmarks, LEGO-IRT achieves stable ability estimates with only about 3% of the total items and yields up to 10% lower estimation error when incorporating structural information; its latent abilities also show stronger alignment with human judgments. The approach enables principled, data-efficient, and interpretable LLM assessment, with practical implications for scalable benchmarking and benchmark design.

Abstract

Evaluating large language models (LLMs) on comprehensive benchmarks is a cornerstone of their development, yet it's often computationally and financially prohibitive. While Item Response Theory (IRT) offers a promising path toward data-efficient evaluation by disentangling model capability from item difficulty, existing IRT-based methods are hampered by significant limitations. They are typically restricted to binary correctness metrics, failing to natively handle the continuous scores used in generative tasks, and they operate on single benchmarks, ignoring valuable structural knowledge like correlations across different metrics or benchmarks. To overcome these challenges, we introduce LEGO-IRT, a unified and flexible framework for data-efficient LLM evaluation. LEGO-IRT's novel design natively supports both binary and continuous evaluation metrics. Moreover, it introduces a factorized architecture to explicitly model and leverage structural knowledge, decomposing model ability estimates into a general component and structure-specific (e.g., per-metric or per-benchmark) components. Through extensive experiments involving $70$ LLMs across $5$ benchmarks, we show that LEGO-IRT achieves stable capability estimates using just $3\%$ of the total evaluation items. We demonstrate that incorporating structural knowledge reduces estimation error by up to $10\%$ and reveal that the latent abilities estimated by our framework may align more closely with human preferences.

Figures (19)

• Figure 1: PGM description of LEGO-CM
• Figure 2: PGM description of LEGO-MM
• Figure 3: PGM description of LEGO-MB
• Figure 4: LLM performance prediction over XSUM and WMT20 under varying training ratios.
• Figure 5: Power comparison between LEGO-IRT and model mean estimation (MME) over XSUM and WMT20 benchmarks.