When LLMs get significantly worse: A statistical approach to detect model degradations

TL;DR

This work addresses the challenge of distinguishing genuine degradation from statistical fluctuations when evaluating optimized LLMs, even under lossless or near-lossless changes. It develops a statistically principled framework based on an exact one-sided McNemar test applied to per-sample outcomes, with three aggregation strategies to combine evidence across benchmarks and a permutation-enabled extension for non-binary scores. The approach enables detection of very small accuracy degradations (e.g., as low as $0.3\%$) and provides practical guidance for dataset curation to improve testing efficiency, including a method to shrink datasets by excluding non-signal instances. The proposed tools and methodology offer robust, interpretable measures for regression testing in inference-stack optimizations, with broad applicability to real-world LLM deployment and optimization pipelines.

Abstract

Minimizing the inference cost and latency of foundation models has become a crucial area of research. Optimization approaches include theoretically lossless methods and others without accuracy guarantees like quantization. In all of these cases it is crucial to ensure that the model quality has not degraded. However, even at temperature zero, model generations are not necessarily robust even to theoretically lossless model optimizations due to numerical errors. We thus require statistical tools to decide whether a finite-sample accuracy deviation is an evidence of a model's degradation or whether it can be attributed to (harmless) noise in the evaluation. We propose a statistically sound hypothesis testing framework based on McNemar's test allowing to efficiently detect model degradations, while guaranteeing a controlled rate of false positives. The crucial insight is that we have to confront the model scores on each sample, rather than aggregated on the task level. Furthermore, we propose three approaches to aggregate accuracy estimates across multiple benchmarks into a single decision. We provide an implementation on top of the largely adopted open source LM Evaluation Harness and provide a case study illustrating that the method correctly flags degraded models, while not flagging model optimizations that are provably lossless. We find that with our tests even empirical accuracy degradations of 0.3% can be confidently attributed to actual degradations rather than noise.

Figures (5)

• Figure 1: Detecting accuracy degradation on Llama-3.1 8B Instruct based on empirical estimates and their uncertainties. The optimized model $\Tilde{M}$ quantizes the KV cache and the attention and results in an empirical accuracy drop of $0.79\%$ over different datasets (\ref{['sec:experiments']}). When naively treating the accuracy estimates as independent, the sampling error of the difference is overestimated, hindering the detection of actual accuracy differences (middle). Instead, we advocate to compare the per-sample scores. Our proposed exact one-sided McNemar's test focuses on the degradation probability. In cases where there is no accuracy difference, the degradation probability is $0.5$ and we can exactly characterize its empirical distribution. This allows to detect small accuracy degradations as statistically significant, here with a $p$-value of 1.69e-05 (right).
• Figure 2: Success histogram for MMLU-Pro.
• Figure 3: Rejection rates of the proposed aggregation schemes dependent on the number of tasks $T$. Error bars are left implicit and are given by the variance formula $\mathrm{Var}[p] = p(1-p)/1000$, since we estimate each probability from 1000 synthetic experiments.
• Figure 4: Asymptotic test power for $N=25,282$ and $\alpha=0.05$ as function of flip probability ${p_{\updownarrow}}$ and degradation probability ${q_{\downarrow}}$.
• Figure 5: $p$-values of the pooled test as a function of sample size $N$, empirical accuracy degradation $\hat{\delta}$ and different flip probabilities. Observations above the dotted line are rejected as significant degradations at $\alpha =5\%$.