Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals

Zihan Dong, Zhixian Zhang, Yang Zhou, Can Jin, Ruijia Wu, Linjun Zhang

TL;DR

The paper tackles the instability of evaluating mathematical reasoning in LLMs due to small benchmark sizes and stochastic outputs. It introduces a semiparametric framework that augments standard accuracy with auxiliary pairwise comparison signals and derives an Efficient Influence Function (EIF) to build a one-step estimator, achieving semiparametric efficiency and variance reduction. The method leverages a known conditional distribution of auxiliary signals and cross-fitting, with a practical semantic-regressor option for very small samples, and proves asymptotic normality of the estimator. Empirically, it demonstrates tighter accuracy estimates and more reliable model rankings on GPQA Diamond, AIME 2025, and GSM8K, outperforming naive estimators especially in small-sample regimes. This approach offers principled, uncertainty-quantified evaluation that can generalize to other performance metrics and auxiliary data sources, advancing robust AI model assessment.

Abstract

Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms. On difficult problems, an LLM may fail to produce a correct final answer, yet still provide reliable pairwise comparison signals indicating which of two candidate solutions is better. We leverage this observation to design a statistically efficient evaluation framework that combines standard labeled outcomes with pairwise comparison signals obtained by having models judge auxiliary reasoning chains. Treating these comparison signals as control variates, we develop a semiparametric estimator based on the efficient influence function (EIF) for the setting where auxiliary reasoning chains are observed. This yields a one-step estimator that achieves the semiparametric efficiency bound, guarantees strict variance reduction over naive sample averaging, and admits asymptotic normality for principled uncertainty quantification. Across simulations, our one-step estimator substantially improves ranking accuracy, with gains increasing as model output noise grows. Experiments on GPQA Diamond, AIME 2025, and GSM8K further demonstrate more precise performance estimation and more reliable model rankings, especially in small-sample regimes where conventional evaluation is pretty unstable.

Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals

TL;DR

The paper tackles the instability of evaluating mathematical reasoning in LLMs due to small benchmark sizes and stochastic outputs. It introduces a semiparametric framework that augments standard accuracy with auxiliary pairwise comparison signals and derives an Efficient Influence Function (EIF) to build a one-step estimator, achieving semiparametric efficiency and variance reduction. The method leverages a known conditional distribution of auxiliary signals and cross-fitting, with a practical semantic-regressor option for very small samples, and proves asymptotic normality of the estimator. Empirically, it demonstrates tighter accuracy estimates and more reliable model rankings on GPQA Diamond, AIME 2025, and GSM8K, outperforming naive estimators especially in small-sample regimes. This approach offers principled, uncertainty-quantified evaluation that can generalize to other performance metrics and auxiliary data sources, advancing robust AI model assessment.

Abstract

Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms. On difficult problems, an LLM may fail to produce a correct final answer, yet still provide reliable pairwise comparison signals indicating which of two candidate solutions is better. We leverage this observation to design a statistically efficient evaluation framework that combines standard labeled outcomes with pairwise comparison signals obtained by having models judge auxiliary reasoning chains. Treating these comparison signals as control variates, we develop a semiparametric estimator based on the efficient influence function (EIF) for the setting where auxiliary reasoning chains are observed. This yields a one-step estimator that achieves the semiparametric efficiency bound, guarantees strict variance reduction over naive sample averaging, and admits asymptotic normality for principled uncertainty quantification. Across simulations, our one-step estimator substantially improves ranking accuracy, with gains increasing as model output noise grows. Experiments on GPQA Diamond, AIME 2025, and GSM8K further demonstrate more precise performance estimation and more reliable model rankings, especially in small-sample regimes where conventional evaluation is pretty unstable.
Paper Structure (31 sections, 3 theorems, 11 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 31 sections, 3 theorems, 11 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Efficiency and Rigor in LLM Evaluation.
  4. Pairwise Comparisons and Alignment Signals.
  5. Semi-parametric Inference.
  6. Problem Set-up
  7. Method
  8. Efficient Influence Function
  9. One-Step Estimator
  10. Theoretical Analysis
  11. Asymptotic Normality
  12. Variance Reduction
  13. Experiments
  14. Simulation Study
  15. Data Generation
  16. ...and 16 more sections

Key Result

Proposition 3.1

The EIF for $\theta$ evaluated at a data point $(X, Y, G, Z)$ under Assumption asm:model_structure is given by: where the nuisance functions $\tau$ and $m$Nuisance function is a commonly used term in semiparametric inference. For self-completeness, we introduce its formal definition in Appendix 4. satisfy

Figures (2)

  • Figure 1: Overview of our Semiparametric Evaluation Framework. We augment standard accuracy evaluation with pairwise comparison signals and construct an EIF-based one-step estimator that achieves the semiparametric efficiency bound, yielding strict variance reduction.
  • Figure 2: Ranking accuracy vs. model-specific signal

Theorems & Definitions (9)

  • Remark 3.1
  • Proposition 3.1: Efficient Influence Function with Known Conditional Distribution
  • Remark 3.2
  • Remark 3.3: Semantic Regression for Small Samples
  • Remark 4.1
  • Theorem 4.1: Asymptotic Normality
  • Remark 4.2
  • Corollary 4.1: Strict Efficiency Gain
  • Remark 4.3