Medical Large Language Model Benchmarks Should Prioritize Construct Validity

TL;DR

Medical LLM benchmarks risk misrepresenting real-world capability when they rely on exam-style tasks. The authors import psychometrics to demand construct validity, using EHR data to empirically assess MedQA across criterion, content, and construct dimensions and finding only modest predictive value and content-domain gaps. They propose a Benchmark-Validation-First framework with hospital validators and validity-focused benchmarks to improve generalization to clinical practice. The work highlights the importance of aligning benchmarks with real-world tasks to support reliable claims about clinical performance and proposes an ecosystem that rewards rigorously validated benchmarks. Overall, the paper argues for principled benchmark design as a practical step toward trustworthy medical LLM deployment.

Abstract

Medical large language models (LLMs) research often makes bold claims, from encoding clinical knowledge to reasoning like a physician. These claims are usually backed by evaluation on competitive benchmarks; a tradition inherited from mainstream machine learning. But how do we separate real progress from a leaderboard flex? Medical LLM benchmarks, much like those in other fields, are arbitrarily constructed using medical licensing exam questions. For these benchmarks to truly measure progress, they must accurately capture the real-world tasks they aim to represent. In this position paper, we argue that medical LLM benchmarks should (and indeed can) be empirically evaluated for their construct validity. In the psychological testing literature, "construct validity" refers to the ability of a test to measure an underlying "construct", that is the actual conceptual target of evaluation. By drawing an analogy between LLM benchmarks and psychological tests, we explain how frameworks from this field can provide empirical foundations for validating benchmarks. To put these ideas into practice, we use real-world clinical data in proof-of-concept experiments to evaluate popular medical LLM benchmarks and report significant gaps in their construct validity. Finally, we outline a vision for a new ecosystem of medical LLM evaluation centered around the creation of valid benchmarks.

Figures (5)

• Figure 1: Overview of evaluation datasets for medical LLMs. We analyzed the evaluation datasets used in the 100 most cited papers on medical LLMs over the past 5 years. The majority (60%) of studies assess models on public benchmarks constructed based on medical exams, while 40% rely on (private or public access) real-world hospital data. There is no clear consensus on a standard benchmark—though MedQA is the most frequently used.
• Figure 2: Analogy between LLM benchmarks and psychological testing. Tests aim to evaluate latent constructs that are theoretically conceived but not directly observable. The validity of a test depends on how well inferences drawn from its scores align with the underlying construct being measured across test subjects.
• Figure 3: Outline of different types of test validity evidence. Beyond subjective face validity, the classical tripartite theory categorizes validity into criterion, content, and construct validity. Modern perspectives view construct validity as the overarching concept, with face, predictive, concurrent, convergent, and discriminant validity serving as distinct sources of evidence for construct validity.
• Figure 4: Empirical validation of the MedQA benchmark using EHR data. Each MedQA item consists of a clinical vignette, a question, and multiple-choice answers, while each EHR patient case includes a clinical note and a corresponding clinical decision. To assess the validity MedQA, we empirically test whether strong performance on benchmark items reflects the ability of an LLM to encode and apply medical knowledge in real-world practice.
• Figure 5: Left: Impact of number of answer choices in MedQA on accuracy. Middle: Distribution of clinical tasks and scenarios in MedQA and real-world data. Right: Distribution of the number of UMLS concepts in MedQA vignettes and real-world clinical notes.