Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling

TL;DR

This work tackles premature exploitation in particle filtering used for inference-time scaling in language models by identifying overconfident process reward models as a primary trigger for particle impoverishment. It introduces Entropic Particle Filtering (ePF), combining Entropic Annealing (EA) to maintain diversity and Look-ahead Modulation (LaM) to bias sampling toward promising futures, enabling more robust exploration of long-horizon reasoning tasks. Across six math benchmarks and multiple model scales, ePF achieves substantial gains over standard PF and other ITS baselines, with particularly strong improvements at small particle budgets and on harder problems; LaM further boosts performance by providing non-myopic guidance. The approach demonstrates that preserving diversity and incorporating forward-looking signals can unlock higher-quality solutions in complex reasoning, making ITS more effective in constrained compute settings, albeit with added compute from look-ahead and reliance on reward-model calibration.

Abstract

Inference-Time Scaling (ITS) improves language models by allocating more computation at generation time. Particle Filtering (PF) has emerged as a strong ITS method for complex mathematical reasoning tasks, but it is vulnerable when guided by process reward models, which often assign overconfident scores early in the reasoning process. This causes PF to suffer from premature exploitation: it myopically commits to locally promising trajectories, prunes potentially correct hypotheses, and converges to suboptimal solutions. This failure mode, known as particle impoverishment, is especially severe under constrained computational budgets. To address this, we analyze the problem and identify two root causes: a lack of diversity in the particle set due to overconfident resampling and consequent inability to assess the potential of a reasoning path. We introduce Entropic Particle Filtering (ePF), an algorithm that integrates two new techniques to solve these issues. The first technique, Entropic Annealing (EA), directly mitigates particle impoverishment by monitoring search diversity via entropy; when diversity drops, it intervenes by dynamically annealing the resampling distribution to preserve exploration. The second, an enhancement called Look-ahead Modulation (LaM), adds a predictive guide to evaluate a state's potential based on its successors. On several challenging math benchmarks, ePF significantly outperforms strong baselines and achieves up to a 50 % relative improvement in task reward. Together, these methods improve PF's resilience by balancing the exploration of diverse solution spaces with the exploitation of high-reward regions, ultimately leading to higher-quality solutions.

Figures (27)

• Figure 1: Task reward comparison on AIME-2025 using Qwen3-1.7B. Our Entropic Particle Filtering (ePF) and its Look-ahead variant (ePF w/ LaM) significantly improve performance over standard Particle Filtering (PF) across all particle budgets. This demonstrates that mitigating premature exploitation leads to significant performance gains.
• Figure 2: PRM overconfidence causes particle impoverishment and harms performance. (a) Uncalibrated PRMs contribute to low diversity by assigning overly optimistic scores to partial solutions, causing Particle Filtering to converge prematurely (Fig. \ref{['fig:reward-entropy']}). (b) The variance of the resampling distribution increases with less particle diversity. (c) Task success is strongly correlated with high ESS.
• Figure 3: The Entropic Particle Filtering (ePF) pipeline and its core Mechanisms. Particle Filtering (left), Entropic Particle Filtering (center), and Entropic Particle Filtering w/ LaM (right). Each circle represents a particle at step $t$ and the size is proportional to the reward provided by the PRM. EA and LaM help the PF algorithm to mitigate early exploitation and myopic updates, greatly improving the diversity of the particles at step $t+1$. Standard PF often suffers from particle impoverishment, where diversity is lost after resampling. Our ePF pipeline incorporates the EA step to maintain particle diversity. ePF with LaM, further adds the LaM step to guide the search more effectively. Pipeline details in Appx \ref{['appx:pipeline']}.
• Figure 4: Adapting the resampling distribution temperature using Entropic Annealing.
• Figure 5: Non-myopic resampling distribution update using Look-ahead Modulation. We use the dynamics model at step $t$ to predict an intermediate next state $s$.