What If We Allocate Test-Time Compute Adaptively?

TL;DR

The paper addresses the inefficiency of uniform test-time compute in reasoning tasks by introducing a verifier-guided adaptive framework that treats reasoning as iterative trajectories. It implements a modular, prompt-driven agent with planning, tool and compute-strategy selection, and answer extraction, guided by a Process Reward Model (PRM) that scores steps locally and trajectories globally. Empirical results on math reasoning benchmarks ($\text{MATH-}500$, $\text{AIME24}$, $\text{AMO-Bench}$) show notable accuracy gains over direct inference, with compute efficiency quantified by theoretical FLOPs $F_{\text{theo}}$ and compute intensity $S_{\text{CI}}$, demonstrating concentrated utility-focused computation. The approach is training-free, model-agnostic, and extensible to other domains requiring reliable, compute-aware reasoning; limitations include PRM reliability on very hard problems and latency overhead, motivating future work on difficulty-aware verification and budget-aware policies.

Abstract

Test-time compute scaling allocates inference computation uniformly, uses fixed sampling strategies, and applies verification only for reranking. In contrast, we propose a verifier-guided adaptive framework treating reasoning as iterative trajectory generation and selection. For each problem, the agent runs multiple inference iterations. In each iteration, it optionally produces a high-level plan, selects a set of reasoning tools and a compute strategy together with an exploration parameter, and then generates a candidate reasoning trajectory. A process reward model (PRM) serves as a unified control signal: within each iteration, step-level PRM scores are aggregated to guide pruning and expansion during generation, and across iterations, aggregated trajectory rewards are used to select the final response. Across datasets, our dynamic, PRM-guided approach consistently outperforms direct test-time scaling, yielding large gains on MATH-500 and several-fold improvements on harder benchmarks such as AIME24 and AMO-Bench. We characterize efficiency using theoretical FLOPs and a compute intensity metric penalizing wasted generation and tool overhead, demonstrating that verification-guided allocation concentrates computation on high-utility reasoning paths.

Figures (5)

• Figure 1: Universal reasoning agent architecture. The system generates $K$ candidate trajectories $\{I_1,\ldots,I_K\}$ with answers $\{y_1,\ldots,y_K\}$, scored by a PRM to produce rewards $R$ for each trajectory $\tau_i$ for selecting the best response (top). Each iteration follows a four-stage workflow (bottom): planning ($\mathcal{A}_P$), tool selection ($\mathcal{A}_T$), compute strategy selection ($\mathcal{A}_C$), and answer extraction ($\mathcal{A}_F$).
• Figure 2: Per-level accuracy on MATH-500 across difficulty levels.
• Figure 3: Dynamic agent configuration statistics. Usage frequency (%) of reasoning tools and compute strategies selected by the adaptive controller across datasets. Dataset-specific patterns reflect differences in problem structure and reasoning difficulty.
• Figure 4: Compute cost scaling across datasets. Theoretical FLOPs and compute intensity ($S_{\text{CI}}$) for main experimental configurations on MATH-500, AIME24, and AMO-Bench.
• Figure 5: Accuracy-efficiency trade-offs on MATH-500 (Qwen-2.5-7B).