Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs

Paula Cordero-Encinar, Andrew B. Duncan

TL;DR

This work presents a principled, statistically grounded framework for certifiable reasoning in large language models by treating multiple rollouts as a stochastic process and certifying that majority voting recovers the model’s mode of the terminal distribution. It introduces the Martingale Majority Certificate (MMC), an anytime-valid stopping rule based on $e$-values that adaptively determines when enough rollouts have been collected to guarantee a small error probability. The authors connect test-time reinforcement learning to exponential tilting of the terminal distribution, providing theoretical justification for why KL-regularised TTRL and entropy-based rewards sharpen the mode and reduce sample requirements, while also proposing two label-free post-training objectives to optimize this trade-off. Empirically, the framework demonstrates tighter reliability guarantees and reduced compute with test-time adaptation on math reasoning benchmarks, and identifies the SNR of the margin as a label-free proxy for problem difficulty. Overall, the work unifies concentration bounds, martingale-based stopping, and test-time adaptation into a cohesive, certifiable approach to reliable reasoning in LLMs with no external supervision.

Abstract

Recent advances such as self-consistency and test-time reinforcement learning (TTRL) improve the reliability of large language models (LLMs) without additional supervision, yet their underlying mechanisms and statistical guarantees remain poorly understood. We present a unified framework for certifiable inference in LLMs, showing that majority voting provides a statistical certificate of self-consistency: under mild assumptions, the aggregated answer coincides with the mode of the model's terminal distribution with high probability. We derive finite-sample and anytime-valid concentration bounds that quantify this confidence, and introduce the Martingale Majority Certificate (MMC), a sequential stopping rule that adaptively determines when sufficient samples have been drawn. We further prove that label-free post-training methods such as TTRL implicitly sharpen the answer distribution by exponentially tilting it toward its mode, thereby reducing the number of samples required for certification. Building on this insight, we propose new post-training objectives that explicitly optimise this trade-off between sharpness and bias. Together, these results explain and connect two central test-time scaling strategies, self-consistency and TTRL, within a single statistical framework for label-free, certifiable reliability in reasoning LLMs.

Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs

TL;DR

This work presents a principled, statistically grounded framework for certifiable reasoning in large language models by treating multiple rollouts as a stochastic process and certifying that majority voting recovers the model’s mode of the terminal distribution. It introduces the Martingale Majority Certificate (MMC), an anytime-valid stopping rule based on -values that adaptively determines when enough rollouts have been collected to guarantee a small error probability. The authors connect test-time reinforcement learning to exponential tilting of the terminal distribution, providing theoretical justification for why KL-regularised TTRL and entropy-based rewards sharpen the mode and reduce sample requirements, while also proposing two label-free post-training objectives to optimize this trade-off. Empirically, the framework demonstrates tighter reliability guarantees and reduced compute with test-time adaptation on math reasoning benchmarks, and identifies the SNR of the margin as a label-free proxy for problem difficulty. Overall, the work unifies concentration bounds, martingale-based stopping, and test-time adaptation into a cohesive, certifiable approach to reliable reasoning in LLMs with no external supervision.

Abstract

Recent advances such as self-consistency and test-time reinforcement learning (TTRL) improve the reliability of large language models (LLMs) without additional supervision, yet their underlying mechanisms and statistical guarantees remain poorly understood. We present a unified framework for certifiable inference in LLMs, showing that majority voting provides a statistical certificate of self-consistency: under mild assumptions, the aggregated answer coincides with the mode of the model's terminal distribution with high probability. We derive finite-sample and anytime-valid concentration bounds that quantify this confidence, and introduce the Martingale Majority Certificate (MMC), a sequential stopping rule that adaptively determines when sufficient samples have been drawn. We further prove that label-free post-training methods such as TTRL implicitly sharpen the answer distribution by exponentially tilting it toward its mode, thereby reducing the number of samples required for certification. Building on this insight, we propose new post-training objectives that explicitly optimise this trade-off between sharpness and bias. Together, these results explain and connect two central test-time scaling strategies, self-consistency and TTRL, within a single statistical framework for label-free, certifiable reliability in reasoning LLMs.
Paper Structure (70 sections, 7 theorems, 185 equations, 9 figures, 5 tables, 3 algorithms)

This paper contains 70 sections, 7 theorems, 185 equations, 9 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Statistical certificates of self-consistency.
  3. Why test-time training helps.
  4. Emergent calibration in reasoning models.
  5. Our contributions.
  6. Statistical guarantees for majority voting
  7. Setting and scope.
  8. Exact error probability with oracle $\mathbf p$
  9. Finite-sample certificates
  10. Asymptotic consistency and the governing rate
  11. Martingale Majority Certificate: a practical stopping rule
  12. Anytime-valid $e$-processes
  13. Two practical priors: truncated Beta(a,b) and an updating point prior
  14. A. Truncated Beta(a,b) prior on $(\tfrac{1}{2},1]$.
  15. B. Updating plug-in point prior.
  16. ...and 55 more sections

Key Result

Theorem 2.1

For all $n\ge 1$,

Figures (9)

  • Figure 1: Overview of the proposed framework. Given a prompt, the model generates multiple reasoning rollouts from the reference distribution $\pi_{\mathrm{ref}}(\cdot|{pr})$. The resulting terminal answers are aggregated via majority voting, viewed as mode estimation under sampling uncertainty. The Martingale Majority Certificate (MMC) monitors the empirical margin and provides an anytime-valid stopping rule for certification. Test-time training with SNR or entropy-based adaptation sharpens the terminal distribution, thereby increasing the signal-to-noise ratio (SNR) and reducing the number of samples required for certification.
  • Figure 2: Distribution of the estimated lower bound of the probability $\mathbb{P}[\widehat{c}_n = c^\star]$ (computed via Beta approximations) and the signal-to-noise ratio $\text{SNR}(\Delta_{j^\star_n})$ when applying the MMC stopping rule with $\varepsilon = 0.1$ and $N_{\text{budget}}=100$. Results are obtained after test-time training with SNR-based rewards on the MATH-500 dataset.
  • Figure 3: Comparison of empirical and theoretical bounds on $P[\widehat{c}_n \neq c^\star]$ for the probability distribution $\mathbf{p} = (0.38, 0.35, 0.27)$.
  • Figure 4: Boxplots showing the distribution of the number of votes required until stopping under the MMC rule as a function of the probability gap $\delta = p_{c^\star}- p_{j^\star}$. Results are shown for $\varepsilon = 0.1$ with a maximum budget of 64 votes.
  • Figure 5: Evolution of different training and validation metrics on the MATH-500 dataset.
  • ...and 4 more figures

Theorems & Definitions (17)

  • Theorem 2.1: Exact small-sample probability
  • Theorem 2.2: Finite-sample certificate
  • Theorem 2.3: Asymptotic consistency
  • Theorem 3.1: Anytime validity
  • Corollary 3.2: Union null for stopping
  • Remark 3.3: Why $o_n$ excludes $B_{n-1}$
  • proof
  • proof
  • proof
  • proof
  • ...and 7 more