On the Power of (Approximate) Reward Models for Inference-Time Scaling
Youheng Zhu, Yiping Lu
TL;DR
This work analyzes inference-time scaling for large language models under Sequential Monte Carlo (SMC) sampling, focusing on the role of the reward-model quality measured by Bellman error. It shows information-theoretic lower bounds implying exponential horizon complexity unless the Bellman error is $O(1/T)$, and it derives end-to-end SMC guarantees that polynomially scale with horizon when the Bellman-error bound holds. The study contrasts naive and optimal proposals, investigates single-particle guidance (SP-gSMC), and introduces Metropolis-Hastings (MH) corrections and resampling-based MH variants to achieve provable TV-distance guarantees with favorable computational trade-offs. A chain-based perspective clarifies the relationship between particle-based and MH-corrected approaches, providing practical guidance on when to invest in reward-model accuracy versus inference-time compute. Overall, the results offer principled bounds and design principles for reward-models and inference strategies to achieve scalable, reliable reasoning with large-language models. $T$ appears as the horizon parameter, and $\varepsilon$ denotes Bellman error, with foundational bounds involving $O(1/T)$ and bases like $(1+\varepsilon)^{2T/3}$ in lower/upper bounds.
Abstract
Inference-time scaling has recently emerged as a powerful paradigm for improving the reasoning capability of large language models. Among various approaches, Sequential Monte Carlo (SMC) has become a particularly important framework, enabling iterative generation, evaluation, rejection, and resampling of intermediate reasoning trajectories. A central component in this process is the reward model, which evaluates partial solutions and guides the allocation of computation during inference. However, in practice, true reward models are never available. All deployed systems rely on approximate reward models, raising a fundamental question: Why and when do approximate reward models suffice for effective inference-time scaling? In this work, we provide a theoretical answer. We identify the Bellman error of the approximate reward model as the key quantity governing the effectiveness of SMC-based inference-time scaling. For a reasoning process of length $T$, we show that if the Bellman error of the approximate reward model is bounded by $O(1/T)$, then combining this reward model with SMC reduces the computational complexity of reasoning from exponential in $T$ to polynomial in $T$. This yields an exponential improvement in inference efficiency despite using only approximate rewards.