Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networks

Timo Freiesleben

Abstract

Recent work in machine learning increasingly attributes human-like capabilities such as reasoning or theory of mind to large language models (LLMs) on the basis of benchmark performance. This paper examines this practice through the lens of construct validity, understood as the problem of linking theoretical capabilities to their empirical measurements. It contrasts three influential frameworks: the nomological account developed by Cronbach and Meehl, the inferential account proposed by Messick and refined by Kane, and Borsboom's causal account. I argue that the nomological account provides the most suitable foundation for current LLM capability research. It avoids the strong ontological commitments of the causal account while offering a more substantive framework for articulating construct meaning than the inferential account. I explore the conceptual implications of adopting the nomological account for LLM research through a concrete case: the assessment of reasoning capabilities in LLMs.

Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networks

Abstract

Recent work in machine learning increasingly attributes human-like capabilities such as reasoning or theory of mind to large language models (LLMs) on the basis of benchmark performance. This paper examines this practice through the lens of construct validity, understood as the problem of linking theoretical capabilities to their empirical measurements. It contrasts three influential frameworks: the nomological account developed by Cronbach and Meehl, the inferential account proposed by Messick and refined by Kane, and Borsboom's causal account. I argue that the nomological account provides the most suitable foundation for current LLM capability research. It avoids the strong ontological commitments of the causal account while offering a more substantive framework for articulating construct meaning than the inferential account. I explore the conceptual implications of adopting the nomological account for LLM research through a concrete case: the assessment of reasoning capabilities in LLMs.
Paper Structure (11 sections, 1 figure)

This paper contains 11 sections, 1 figure.

Table of Contents

  1. Introduction
  2. The nomological account is the best available framework for conceptualizing LLM capabilities
  3. The causal account is ontologically demanding for LLMs
  4. The inferential account provides limited resources to theorize LLM capabilities
  5. Nomological Networks as a middle ground between causal realism and pure pragmatism
  6. Implications of the nomological account for LLM benchmark design
  7. Researchers should design nomological networks prior to benchmarks
  8. Task benchmarks require empirical and theoretical validation
  9. Capability benchmarks should undergo convergent and discriminant analysis
  10. Resolving conflicts between theory and measurement is difficult
  11. Conclusion

Figures (1)

  • Figure 1: The nomological account structures the benchmark design process into four steps. The right-hand side illustrates the framework using reasoning as an example, with the Cattell–Horn–Carroll theory carroll1993humanmcgrew2005cattell as the nomological network. To keep the example readable, only a subset of reasoning tasks is described and only two qualitative relationships are specified: ‘+’ indicates an expected positive relationship between reasoning and working memory kyllonen1990reasoning, and ‘ ’ denotes an expected independence between emotional intelligence and reasoning mayer2008emotional.