$φ$-Decoding: Adaptive Foresight Sampling for Balanced Inference-Time Exploration and Exploitation

TL;DR

φ-Decoding reframes inference-time decoding as foresight sampling to achieve a global-optimal balance between exploration and exploitation without requiring external rewards. It combines dynamic advantage estimation, alignment via clustering, and joint-distribution sampling to select steps, while employing in-width and in-depth pruning to allocate compute adaptively. Across seven benchmarks and multiple backbone LLMs, φ-Decoding consistently improves accuracy and reduces FLOPS compared with AR, ToT, MCTS, and predictive baselines, and scales to 70B-parameter models and competition-level tasks. The approach yields better step-value estimation accuracy, demonstrates robust generalization, and offers a practical, training-free pathway to more efficient non-myopic reasoning in large language models.

Abstract

Inference-time optimization scales computation to derive deliberate reasoning steps for effective performance. While previous search-based strategies address the short-sightedness of auto-regressive generation, the vast search space leads to excessive exploration and insufficient exploitation. To strike an efficient balance to derive the optimal step, we frame the decoding strategy as foresight sampling, leveraging simulated future steps to obtain globally optimal step estimation. Built on it, we propose a novel decoding strategy, named $φ$-Decoding. To provide a precise and expressive estimation of step value, $φ$-Decoding approximates two distributions via foresight and clustering. Sampling from the joint distribution, the optimal steps can be selected for exploitation. To support adaptive computation allocation, we propose in-width and in-depth pruning strategies, featuring a light-weight solution to achieve inference efficiency. Extensive experiments across seven benchmarks show $φ$-Decoding outperforms strong baselines in both performance and efficiency. Additional analysis demonstrates its generalization across various LLMs and scalability across a wide range of computing budgets. The code will be released at https://github.com/xufangzhi/phi-Decoding, and the open-source PyPI package is coming soon.

Figures (6)

• Figure 1: Comparisons of different decoding paradigms. (a) is auto-regressive decoding, which has high efficiency but lacks global awareness. (b) represents search-based methods, which requires huge search space with extensive time cost. (c) is the foresight sampling strategy. It leverages the simulated future steps to estimate the step value, which can strike a balanced inference-time exploration and exploitation.
• Figure 2: The overall framework of $\phi$-Decoding. We visualize the decoding process at the timestamp $t$. For simplicity, we set step beam size $M$ as 2, the number of rollouts $N$ as 3, and the number of clusters $K$ as 2.
• Figure 3: Inference-time scaling law on LLaMA3.1-8B-Instruct. The horizontal axis denotes the inference-time computational cost, while the vertical axis represents the average performances on 6 benchmarks.
• Figure 4: Analysis on the accuracy of step value estimation. The bar in light blue represents the accuracy of the step values, while the bar in dark blue denotes the averaged task performances.
• Figure 5: Visualization of step-wise effects with alleviated overthinking. The first row displays the results for each independent benchmark using the LLaMA backbone, whereas the second row reflects the results with the Mistral backbone.