Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight

TL;DR

This work reveals that a widely used clinical benchmark (MedCalc-Bench) relies on reference labels partly produced by LLMs, which can embed systematic misalignment with real clinical practice. The authors implement a phased physician-in-the-loop stewardship—comprising a tool-augmented automated audit, independent recomputation of labels, and targeted physician adjudication—to identify and repair label biases at scale. In a controlled RL experiment, replacing original labels with recomputed ones materially shifts downstream model behavior, underscoring how label bias can distort evaluation and alignment outcomes. The results advocate for living, versioned benchmarks with explicit abstention handling and a hybrid oversight framework to ensure safe deployment of LLMs in high-stakes medical decision support. Overall, the study demonstrates that maintaining clinically-grounded benchmarks is essential to prevent benchmarks from becoming biased teachers and to preserve the integrity of model evaluation and alignment in healthcare.

Abstract

We examine the reliability of a widely used clinical AI benchmark whose reference labels were partially generated by LLMs, and find that a substantial fraction are clinically misaligned. We introduce a phased stewardship procedure to amplify the positive impact of physician experts' feedback and then demonstrate, via a controlled RL experiment, how uncaught label bias can materially affect downstream LLM evaluation and alignment. Our results demonstrate that partially LLM-generated labels can embed systemic errors that distort not only evaluation but also downstream model alignment. By adopting a hybrid oversight system, we can prioritize scarce expert feedback to maintain benchmarks as living, clinically-grounded documents. Ensuring this alignment is a prerequisite for the safe deployment of LLMs in high-stakes medical decision support.

Figures (9)

• Figure 1: Our benchmark stewardship utilizes two distinct LLM agent workflows (Phases 1 & 2) to assure MedCalc-Bench’s label quality. Besides their prompt and output type (Yes/No verdict versus a recomputed label), the two workflows differ in what the LLM agent is given as context: the Phase 1 agent is shown the original reference label and its derivation metadata, whereas the Phase 2 agent is only provided the patient narrative and score question; the latter omission is intended to reduce any anchoring bias towards $\hat{y}^{\mathrm{original}}$. In Phase 3, three physician authors (DT, AG, NK) independently compute 50 instances where $\hat{y}^{\mathrm{original}}$ and $\hat{y}^{\mathrm{new}}$ show high disagreement. sMAPE stands for symmetric Mean Absolute Percentage Error (Eq. \ref{['eq:smape']}); it measures the relative disagreement in $\{(y^\star_i,\hat{y}^{\mathrm{new}}_i)\}_{i=1}^{50}$ and $\{(y^\star_i, \hat{y}^{\mathrm{original}}_i)\}_{i=1}^{50}$.
• Figure 2: Representative error types. (a) Feature extraction error: GPT-4 might have confused "hemoglobin" with "albumin", extracting a value that is physiologically impossible; (b) Incorrect aggregation logic: an incorrect Python code for Glasgow Coma Scale aggregation that double-counts a feature value, inflating $\hat{y}^{\mathrm{original}}$; (c) $q$ is not answerable given $C$: a Sodium correction for hyperglycemia inappropriately applied to a hypoglycemic patient.
• Figure 3: Label Instantiation Changes Alignment Interpretation. Test accuracy dynamics for Qwen3-8B trained via GRPO using the recomputed reference labels (green) versus the original MedCalc-Bench labels (grey). Shaded bands indicate $\pm 1\sigma$ smoothed over a 10-step window. Improving the reward signal's factual grounding effects a $+8.7\%$ absolute gain in the final moving averages of model test accuracy ($71.4\%$ vs. $62.6\%$).
• Figure 4: Prompts for the controlled RL experiment in §\ref{['sec:rl']}, which are identically used across the two comparison groups. These prompts are fed as context into a Qwen3-8B model checkpoint that parametrizes the RL policy.
• Figure 5: Prompts for the LLM audit agent executing the Phase 1 workflow (§\ref{['sec:phase1_audit']}.)