Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring what Matters: Construct Validity in Large Language Model Benchmarks

Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, Oishi Deb, Emma Beharry, Cornelius Emde, Thomas Foster, Anna Gausen, María Grandury, Simeng Han, Valentin Hofmann, Lujain Ibrahim, Hazel Kim, Hannah Rose Kirk, Fangru Lin, Gabrielle Kaili-May Liu, Lennart Luettgau, Jabez Magomere, Jonathan Rystrøm, Anna Sotnikova, Yushi Yang, Yilun Zhao, Adel Bibi, Antoine Bosselut, Ronald Clark, Arman Cohan, Jakob Foerster, Yarin Gal, Scott A. Hale, Inioluwa Deborah Raji, Christopher Summerfield, Philip H. S. Torr, Cozmin Ududec, Luc Rocher, Adam Mahdi

TL;DR

The paper conducts a systematic review of 445 LLM benchmarks from major NLP/ML venues to assess construct validity, revealing widespread weaknesses in how phenomena are defined, how tasks are chosen, how metrics are scored, and how claims are made. It introduces an operational construct validity checklist with eight actionable recommendations to guide the design and interpretation of benchmarks, exemplified by GSM8K. The authors demonstrate that many benchmarks rely on ambiguous definitions, non-representative task spaces, and limited statistical validation, which undermines the reliability of conclusions about model capabilities, safety, and robustness. By providing structured guidance across the benchmark lifecycle, the work aims to improve the alignment between benchmark results and real-world phenomena, enabling more credible progress reporting and safer deployment decisions.

Abstract

Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as 'safety' and 'robustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.

Measuring what Matters: Construct Validity in Large Language Model Benchmarks

TL;DR

The paper conducts a systematic review of 445 LLM benchmarks from major NLP/ML venues to assess construct validity, revealing widespread weaknesses in how phenomena are defined, how tasks are chosen, how metrics are scored, and how claims are made. It introduces an operational construct validity checklist with eight actionable recommendations to guide the design and interpretation of benchmarks, exemplified by GSM8K. The authors demonstrate that many benchmarks rely on ambiguous definitions, non-representative task spaces, and limited statistical validation, which undermines the reliability of conclusions about model capabilities, safety, and robustness. By providing structured guidance across the benchmark lifecycle, the work aims to improve the alignment between benchmark results and real-world phenomena, enabling more credible progress reporting and safer deployment decisions.

Abstract

Evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment. Reliably measuring abstract and complex phenomena such as 'safety' and 'robustness' requires strong construct validity, that is, having measures that represent what matters to the phenomenon. With a team of 29 expert reviewers, we conduct a systematic review of 445 LLM benchmarks from leading conferences in natural language processing and machine learning. Across the reviewed articles, we find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims. To address these shortcomings, we provide eight key recommendations and detailed actionable guidance to researchers and practitioners in developing LLM benchmarks.

Paper Structure

This paper contains 33 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methods
  4. Study design
  5. Codebook and expert review
  6. Results
  7. Phenomenon
  8. Task
  9. Metric
  10. Claims
  11. Recommendations
  12. Define the phenomenon
  13. Measure the phenomenon and only the phenomenon
  14. Construct a representative dataset for the task
  15. Acknowledge limitations of reusing datasets
  16. ...and 18 more sections

Figures (4)

  • Figure 1: Systematic review process. (A) Identification and screening from relevant proceedings. (B) In-depth review and annotation of included benchmarks. A phenomenon is operationalised via a task, scored with a metric, to support a claim about this phenomenon. (C) Synthesis of best practices.
  • Figure 2: Summary of reviewed articles. (A) Three most common categories of benchmark phenomena, grouped into general capabilities, general applications, and specific applications. (B) Number of articles by publication year and number which discuss the construct validity of their benchmark.
  • Figure 3: Key codebook results. The distribution of codebook responses on selected items. In each column, the options are ordered from most to least preferred for high construct validity. The shaded area indicates the benchmarks that follow the best practices for all five items.
  • Figure 4: Flowchart of the systematic review process. Searching across EMNLP, NAACL, ACL, ICML, ICLR, and NeurIPS, we identified 2,189 papers matching the keyword search, and 445 which ultimately met the review criteria.