Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic

TL;DR

The paper argues that evaluating LLM judges through Balanced Accuracy, equivalently Youden’s J, yields prevalence-robust, symmetric judgments that reliably detect true prevalence differences between models. It mathematically connects J to classifier slope and ROC geometry, and demonstrates via empirical studies that BA outperforms traditional metrics (Accuracy, F1, Macro-F1) in selecting judges that preserve model-prevalence order. The results advocate adopting BA as a standard metric for judge evaluation to improve the reliability of downstream model comparisons and release decisions.

Abstract

Rigorous evaluation of large language models (LLMs) relies on comparing models by the prevalence of desirable or undesirable behaviors, such as task pass rates or policy violations. These prevalence estimates are produced by a classifier, either an LLM-as-a-judge or human annotators, making the choice of classifier central to trustworthy evaluation. Common metrics used for this choice, such as Accuracy, Precision, and F1, are sensitive to class imbalance and to arbitrary choices of positive class, and can favor judges that distort prevalence estimates. We show that Youden's $J$ statistic is theoretically aligned with choosing the best judge to compare models, and that Balanced Accuracy is an equivalent linear transformation of $J$. Through both analytical arguments and empirical examples and simulations, we demonstrate how selecting judges using Balanced Accuracy leads to better, more robust classifier selection.

Figures (5)

• Figure 1: Illustration of how an imperfect classifier affects prevalence estimation. The red line shows the relationship $y\!=\!\beta\!+\!(\alpha\!-\!\beta)x$ between actual prevalence $x$ and estimated prevalence $y$ for a classifier with sensitivity $\alpha$ and specificity $(1\!-\!\beta)$. The slope is $(\alpha\!-\!\beta)\!=\!J$, showing that Youden’s $J$ directly measures the classifier's ability to preserve true prevalence differences.
• Figure 2: The relationship between Youden’s $J$ and the ROC curve. J is the vertical distance between a point on the ROC curve for a given threshold and the diagonal chance line. The optimal threshold needs to balance sensitivity and specificity, and is represented by the point on the curve that maximizes this vertical distance.
• Figure 3: Balanced Accuracy is more robust to imbalance compared to other selection metrics across different prevalence regimes. Its ranking loss increases only slightly as the behavior becomes rarer.
• Figure 4: Going from 25 to a few thousand labeled examples reduces ranking loss for all metrics, but beyond about 1,000--2,000 labels the curves largely plateau. Balanced Accuracy is the clear winning metric for selecting a rank-optimal judge.
• Figure 5: Across all model evaluation sample sizes, Balanced Accuracy's selection probability curve is the highest, typically sitting 2–5 percentage points above F1/Macro-F1 and 10+ points above Accuracy at moderate to large $n_{\text{eval}}$.