Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference

Yue Yu, Qiwei Di, Quanquan Gu, Dongruo Zhou

TL;DR

This work analyzes the limits of test-time compute under a mixture-of-reference-policies model and shows that Best-of-N (BoN) is inherently suboptimal in this setting. It introduces Reward-Filtered Sequential Best-of-n (RF-SeqBoN), a simple sequential TTC method that feeds back only high-reward generations, and proves it achieves stronger guarantees than parallel TTC under mild reward-model assumptions. The authors provide extensive experiments on math and science benchmarks with multiple backbone LLMs, demonstrating consistent improvements in budget efficiency over BoN and other sequential baselines. Theoretical results are complemented by ablations and case studies highlighting robustness and practical gains across difficulty levels and model sizes.

Abstract

Test-time compute (TTC) has become an increasingly prominent paradigm for enhancing large language models (LLMs). Despite the empirical success of methods such as best-of-$n$ (BoN) sampling and sequential revision, their fundamental limits remain unclear. We address this gap by analyzing a mixture-of-reference policy model and proving that standard BoN is inherently suboptimal. To move closer to the optimal frontier, we study reward-filtered sequential inference, a simple procedure that selectively incorporates only high-reward generations into the context. This mechanism concentrates computation on superior policy candidates and suppresses inferior ones. On the theoretical side, we show that reward-filtered sequential inference yields strictly stronger guarantees than standard TTC paradigms. On the empirical side, we evaluate such an inference strategy across diverse benchmarks and observe consistent improvements over widely used approaches, demonstrating the practical effectiveness of our framework.

On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference

TL;DR

This work analyzes the limits of test-time compute under a mixture-of-reference-policies model and shows that Best-of-N (BoN) is inherently suboptimal in this setting. It introduces Reward-Filtered Sequential Best-of-n (RF-SeqBoN), a simple sequential TTC method that feeds back only high-reward generations, and proves it achieves stronger guarantees than parallel TTC under mild reward-model assumptions. The authors provide extensive experiments on math and science benchmarks with multiple backbone LLMs, demonstrating consistent improvements in budget efficiency over BoN and other sequential baselines. Theoretical results are complemented by ablations and case studies highlighting robustness and practical gains across difficulty levels and model sizes.

Abstract

Test-time compute (TTC) has become an increasingly prominent paradigm for enhancing large language models (LLMs). Despite the empirical success of methods such as best-of- (BoN) sampling and sequential revision, their fundamental limits remain unclear. We address this gap by analyzing a mixture-of-reference policy model and proving that standard BoN is inherently suboptimal. To move closer to the optimal frontier, we study reward-filtered sequential inference, a simple procedure that selectively incorporates only high-reward generations into the context. This mechanism concentrates computation on superior policy candidates and suppresses inferior ones. On the theoretical side, we show that reward-filtered sequential inference yields strictly stronger guarantees than standard TTC paradigms. On the empirical side, we evaluate such an inference strategy across diverse benchmarks and observe consistent improvements over widely used approaches, demonstrating the practical effectiveness of our framework.

Paper Structure

This paper contains 44 sections, 9 theorems, 154 equations, 11 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Notation.
  3. Related Work
  4. Parallel TTC.
  5. Sequential TTC.
  6. Theory of TTC.
  7. Preliminaries
  8. Task Description.
  9. Pretraining of LLM.
  10. Fundamental limits of parallel test-time compute
  11. Better Test-Time Compute with Reward-Filtered Sequences
  12. Experiments
  13. Models and Datasets.
  14. Baselines and Implementation.
  15. Results.
  16. ...and 29 more sections

Key Result

Proposition 4.2

Given $\epsilon>0$, denote $M^{x, \epsilon}_\text{LLM}: = M^{\epsilon}_{\pi^\star(\cdot \mid x), \pi_\text{LLM}(\cdot\mid x)}$ and $C_\text{LLM}^\star(x):= C(\pi^\star(\cdot \mid x), \pi_{\text{LLM}}(\cdot \mid x))$. Then vanilla BoN (Algorithm alg:1 with $h_i = x$) takes $n = M^{x, \epsilon}_\text{

Figures (11)

  • Figure 1: Main results with Qwen3-4B-Instruct (top) and Qwen3-0.6B-Thinking (bottom) foundation models of values with generation budget $N$. Each column corresponds to one benchmark dataset. The points and error bars show the mean and standard deviation across five repeated experiments, respectively.
  • Figure 2: Breakdown of MATH500 performance when generation budget $N=128$ across five difficulty levels.
  • Figure 3: Number of MATH-500 questions that contribute $k=3$, $2$, $1$, or $0$ filtered answers to the LLM context, under RF-SeqBoN when $N=128$ and -history_budget$=3$, for different values of threshold $\gamma$.
  • Figure 4: Ablation study of prompt template on MATH500 with Qwen3-4B-Instruct foundation model. The points and error bars show the mean and standard deviation from five repeated experiments, respectively.
  • Figure 5: Accuracy comparison for BoN, PureSeq and RF-SeqBoN under the same test-time budget. The bar heights and error bars show the mean and standard deviation from five repeated experiments, respectively. The RF-SeqBoN still dominates the other two in all settings.
  • ...and 6 more figures

Theorems & Definitions (19)

  • Definition 3.2: Sequential sample-and-evaluate framework, generalized from huang2025best
  • Remark 3.3
  • Remark 3.4
  • Definition 4.1: $\mathcal{E}_M$-divergence and coverage, huang2025best
  • Proposition 4.2: Adapted from huang2025best
  • Theorem 4.3: Lower Bound of Sequential Sample-and Evaluate Algorithms
  • Theorem 5.1
  • Remark 5.2
  • Remark 5.3
  • Theorem 5.5
  • ...and 9 more