Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling

Xinglin Wang, Yiwei Li, Shaoxiong Feng, Peiwen Yuan, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li

TL;DR

This work rethinks test-time scaling by formulating parallel search as a fixed-budget resource-allocation problem and proving that traditional solution-level allocation biases can waste compute. It introduces DORA, a direction-oriented allocation method that uses semantic clustering to decouple direction quality from candidate count, achieving optimal direction-level rollout distribution. Theoretical analysis shows how priors and reliability shapes allocation decisions, and empirical results on math reasoning benchmarks (e.g., MATH500, AIME2024/2025) demonstrate that DORA consistently outperforms strong baselines while reducing compute costs by orders of magnitude in some settings. The findings advance practical TTS by delivering higher accuracy with lower latency and FLOPs, and suggest broader applicability to PRM-guided search and other reasoning tasks.

Abstract

Test-Time Scaling (TTS) improves the performance of Large Language Models (LLMs) by using additional inference-time computation to explore multiple reasoning paths through search. Yet how to allocate a fixed rollout budget most effectively during search remains underexplored, often resulting in inefficient use of compute at test time. To bridge this gap, we formulate test-time search as a resource allocation problem and derive the optimal allocation strategy that maximizes the probability of obtaining a correct solution under a fixed rollout budget. Within this formulation, we reveal a core limitation of existing search methods: solution-level allocation tends to favor reasoning directions with more candidates, leading to theoretically suboptimal and inefficient use of compute. To address this, we propose Direction-Oriented Resource Allocation (DORA), a provably optimal method that mitigates this bias by decoupling direction quality from candidate count and allocating resources at the direction level. To demonstrate DORA's effectiveness, we conduct extensive experiments on challenging mathematical reasoning benchmarks including MATH500, AIME2024, and AIME2025. The empirical results show that DORA consistently outperforms strong baselines with comparable computational cost, achieving state-of-the-art accuracy. We hope our findings contribute to a broader understanding of optimal TTS for LLMs.

Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling

TL;DR

This work rethinks test-time scaling by formulating parallel search as a fixed-budget resource-allocation problem and proving that traditional solution-level allocation biases can waste compute. It introduces DORA, a direction-oriented allocation method that uses semantic clustering to decouple direction quality from candidate count, achieving optimal direction-level rollout distribution. Theoretical analysis shows how priors and reliability shapes allocation decisions, and empirical results on math reasoning benchmarks (e.g., MATH500, AIME2024/2025) demonstrate that DORA consistently outperforms strong baselines while reducing compute costs by orders of magnitude in some settings. The findings advance practical TTS by delivering higher accuracy with lower latency and FLOPs, and suggest broader applicability to PRM-guided search and other reasoning tasks.

Abstract

Test-Time Scaling (TTS) improves the performance of Large Language Models (LLMs) by using additional inference-time computation to explore multiple reasoning paths through search. Yet how to allocate a fixed rollout budget most effectively during search remains underexplored, often resulting in inefficient use of compute at test time. To bridge this gap, we formulate test-time search as a resource allocation problem and derive the optimal allocation strategy that maximizes the probability of obtaining a correct solution under a fixed rollout budget. Within this formulation, we reveal a core limitation of existing search methods: solution-level allocation tends to favor reasoning directions with more candidates, leading to theoretically suboptimal and inefficient use of compute. To address this, we propose Direction-Oriented Resource Allocation (DORA), a provably optimal method that mitigates this bias by decoupling direction quality from candidate count and allocating resources at the direction level. To demonstrate DORA's effectiveness, we conduct extensive experiments on challenging mathematical reasoning benchmarks including MATH500, AIME2024, and AIME2025. The empirical results show that DORA consistently outperforms strong baselines with comparable computational cost, achieving state-of-the-art accuracy. We hope our findings contribute to a broader understanding of optimal TTS for LLMs.

Paper Structure

This paper contains 35 sections, 3 theorems, 45 equations, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Setup & Preliminaries
  3. Problem Formulation
  4. Parallel Search Method
  5. Temperature Sampling.
  6. Beam Search.
  7. DVTS.
  8. REBASE.
  9. Optimal Parallel Search for Test-Time Scaling
  10. Theoretical Formulation of Optimal Resource Allocation
  11. Suboptimality of Solution-Level Allocation
  12. Direction-Oriented Resource Allocation (DORA)
  13. Experiments
  14. Experimental Setup
  15. Main Results
  16. ...and 20 more sections

Key Result

Proposition 1

Let $O^\star(\bm{w})_i$ denote the optimal rollout allocation for candidate $i$, where $\bm{w} = \{w_1, \dots, w_k\}$ are the normalized PRM scores. Then:

Figures (5)

  • Figure 1: Comparison of different parallel Test-Time search strategies.
  • Figure 2: Comparison between Solution-Level Resource Allocation and proposed Direction-Oriented Resource Allocation (DORA).
  • Figure 3: Accuracy and Pass rate comparison under various rollout budgets on MATH500, AIME2024, and AIME2025.
  • Figure 4: Effect of the concentration parameter $\kappa$ on the Beta prior. All curves are plotted with fixed mean $w_i = 0.7$. Larger $\kappa$ yields a more concentrated prior around $w_i$, while smaller $\kappa$ reflects greater uncertainty.
  • Figure 5: Comparison of method accuracy on MATH500 across different difficulty levels.

Theorems & Definitions (3)

  • Proposition 1: Limiting Behavior of Optimal Allocation
  • Proposition 2: Suboptimality of Solution-Level Allocation
  • Theorem 1: Optimality of DORA