Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

More Test-Time Compute Can Hurt: Overestimation Bias in LLM Beam Search

Gal Dalal, Assaf Hallak, Gal Chechik, Yftah Ziser

Abstract

Wider beam search should improve LLM reasoning, but when should you stop widening? Prior work on beam width selection has focused on inference efficiency \citep{qin2025dsbd, freitag2017beam}, without analyzing whether wider search can \emph{hurt} output quality. We present an analysis, grounded in Extreme Value Theory, that answers this question. Beam selection over noisy scorer outputs introduces a systematic overestimation bias that grows with the candidate pool size, and we derive a maximum useful beam width $\hat{k}$ beyond which search degrades performance. This critical width depends on the signal-to-noise ratio of the scorer: $\hat{k}$ grows exponentially with $(Δ/σ)^2$, where $Δ> 0$ is the quality advantage of correct paths over incorrect ones and $σ$ is the scorer noise. We validate this theory by comparing perplexity-guided and PRM-guided beam search across three 7B-parameter models and ten domains on MR-BEN (5,975 questions). Perplexity scoring, with its high noise, yields $\hat{k} = 1$: search provides no benefit at any width tested. PRM scoring, with lower noise, yields $\hat{k} \geq 4$, with gains of up to 8.9 percentage points. The same model, the same algorithm, but different scorers place $\hat{k}$ at opposite ends of the beam width range. Our analysis identifies the scorer's signal-to-noise ratio as the key quantity governing beam width selection, and we propose diagnostic indicators for choosing the beam width in practice.

More Test-Time Compute Can Hurt: Overestimation Bias in LLM Beam Search

Abstract

Wider beam search should improve LLM reasoning, but when should you stop widening? Prior work on beam width selection has focused on inference efficiency \citep{qin2025dsbd, freitag2017beam}, without analyzing whether wider search can \emph{hurt} output quality. We present an analysis, grounded in Extreme Value Theory, that answers this question. Beam selection over noisy scorer outputs introduces a systematic overestimation bias that grows with the candidate pool size, and we derive a maximum useful beam width beyond which search degrades performance. This critical width depends on the signal-to-noise ratio of the scorer: grows exponentially with , where is the quality advantage of correct paths over incorrect ones and is the scorer noise. We validate this theory by comparing perplexity-guided and PRM-guided beam search across three 7B-parameter models and ten domains on MR-BEN (5,975 questions). Perplexity scoring, with its high noise, yields : search provides no benefit at any width tested. PRM scoring, with lower noise, yields , with gains of up to 8.9 percentage points. The same model, the same algorithm, but different scorers place at opposite ends of the beam width range. Our analysis identifies the scorer's signal-to-noise ratio as the key quantity governing beam width selection, and we propose diagnostic indicators for choosing the beam width in practice.
Paper Structure (36 sections, 4 theorems, 9 equations, 11 figures, 3 tables)

This paper contains 36 sections, 4 theorems, 9 equations, 11 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Beam Width Selection
  4. Process Supervision for LLM Reasoning
  5. Inference-Time Search
  6. Overestimation Bias and Reward Overoptimization
  7. Theoretical Framework: Overestimation Bias in Beam Selection
  8. Setup and Notation
  9. Assumptions
  10. Overestimation Bias from Beam Selection
  11. Proof outline.
  12. Sub-Optimal Selection Probability
  13. When Does Search Help? A Necessary Condition
  14. Experiments
  15. Setup
  16. ...and 21 more sections

Key Result

Lemma 3.1

Under Assumptions ass:noisy_scorer and ass:two_class, let $n$ denote the total number of candidates with $n - 1$ incorrect-type candidates. The expected score of the best incorrect candidate is well-approximated by the GEV limit: where the overestimation bias $B(\sigma, n-1)$ for $n \geq 3$ is given by with $\Phi^{-1}$ the standard normal quantile function, $\gamma_{\mathrm{EM}} \approx 0.5772$

Figures (11)

  • Figure 1: Per-model comparison of perplexity (gray) vs. PRM (green) scoring across beam widths. The two scorers place the predicted maximum useful beam width $\hat{k}$ at opposite ends of the range: perplexity curves are flat ($\hat{k} = 1$), while PRM curves rise through beam width 4 ($\hat{k} \geq 4$), illustrating how scorer quality determines the optimal beam width.
  • Figure 2: Per-model change from $k$=1 by scorer. Hatched bars indicate negative $\Delta$. The PRM unlocks search benefits where perplexity cannot.
  • Figure 3: Reward margin distribution for perplexity-scored beam selections. The peak near zero confirms that 44% of selections have negligible margins, signaling that the scorer cannot support wider search.
  • Figure 4: Per-subject success rate vs. beam width for Qwen2.5-7B-Instruct with perplexity scoring.
  • Figure 5: Per-subject success rate vs. beam width for Llama-3.1-8B-Instruct with perplexity scoring.
  • ...and 6 more figures

Theorems & Definitions (10)

  • Lemma 3.1: Overestimation Bias
  • Remark 1: Conservatism of the bound
  • Theorem 3.2: Sub-Optimal Selection Bound
  • proof
  • Corollary 3.3: Search Benefit Criterion
  • proof
  • Corollary 3.4: Maximum Useful Candidate Pool
  • proof
  • proof
  • proof