Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening

Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer, Haitham Bou Ammar

TL;DR

This work tackles the problem that RL post-training largely sharpens an existing base-model distribution rather than imparting new capabilities. It theory-derives a link between the global power distribution $p^{(\text{pow})}_{\alpha}$ and a scaled low-temperature distribution $p^{(\text{low.temp})}_{\alpha}$ through a per-token factor $\zeta_t(x_t)$, enabling training-free, verifier-free autoregressive sampling. It introduces a jackknife-bias–reduced estimator and a scalable algorithm that uses Top-$K$, MC rollouts, and lookahead to approximate $p^{(\text{pow})}_{\alpha}$ efficiently, achieving over 10× inference-speedups versus MCMC while matching or exceeding RL-based gains on math, code, and QA benchmarks. Empirically, the method demonstrates robustness across four LLMs, preserves diversity at higher $K$, and shows practical impact by enabling training-free sharpening that approaches the performance of RL-tuned models with substantially lower computational cost. Overall, the approach offers a scalable path to training-free advanced reasoning with broad applicability and improved environmental and safety considerations.

Abstract

Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a novel formulation showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks across four LLMs, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over 10x compared to MCMC-based sampling.

Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening

TL;DR

This work tackles the problem that RL post-training largely sharpens an existing base-model distribution rather than imparting new capabilities. It theory-derives a link between the global power distribution and a scaled low-temperature distribution through a per-token factor , enabling training-free, verifier-free autoregressive sampling. It introduces a jackknife-bias–reduced estimator and a scalable algorithm that uses Top-, MC rollouts, and lookahead to approximate efficiently, achieving over 10× inference-speedups versus MCMC while matching or exceeding RL-based gains on math, code, and QA benchmarks. Empirically, the method demonstrates robustness across four LLMs, preserves diversity at higher , and shows practical impact by enabling training-free sharpening that approaches the performance of RL-tuned models with substantially lower computational cost. Overall, the approach offers a scalable path to training-free advanced reasoning with broad applicability and improved environmental and safety considerations.

Abstract

Reinforcement learning (RL) post-training is a dominant approach for improving the reasoning performance of large language models (LLMs), yet growing evidence suggests that its gains arise primarily from distribution sharpening rather than the acquisition of new capabilities. Recent work has shown that sampling from the power distribution of LLMs using Markov chain Monte Carlo (MCMC) can recover performance comparable to RL post-training without relying on external rewards; however, the high computational cost of MCMC makes such approaches impractical for widespread adoption. In this work, we propose a theoretically grounded alternative that eliminates the need for iterative MCMC. We derive a novel formulation showing that the global power distribution can be approximated by a token-level scaled low-temperature one, where the scaling factor captures future trajectory quality. Leveraging this insight, we introduce a training-free and verifier-free algorithm that sharpens the base model's generative distribution autoregressively. Empirically, we evaluate our method on math, QA, and code tasks across four LLMs, and show that our method matches or surpasses one-shot GRPO without relying on any external rewards, while reducing inference latency by over 10x compared to MCMC-based sampling.
Paper Structure (42 sections, 7 theorems, 56 equations, 6 figures, 6 tables, 2 algorithms)

This paper contains 42 sections, 7 theorems, 56 equations, 6 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Power & Low-Temperature Distributions
  3. Approximate Sampling from $p_{\alpha}^{(\text{pow})}$.
  4. Approximating Power Distributions
  5. Autoregressive Monte Carlo Estimation
  6. Bias Analysis and Correction
  7. Jackknife Estimator for Bias Reduction.
  8. Scalable Power Sampling
  9. Experiments & Results
  10. Building Intuition: Planning vs. Guessing
  11. Large-Scale Evaluation
  12. Models.
  13. Benchmarks.
  14. Baselines.
  15. Main Results.
  16. ...and 27 more sections

Key Result

Theorem 3.1

Let $p(x)$ be a pretrained large language model, $\bm{q}$ an input prompt and $\alpha > 1$ be an exponent parameter. Then, for any partially generated sequence $\bm{x}_{0:t-1} = \{x_0,\ldots, x_{t-1}\}$ we have: where for each token $x_t^{\prime} \in \mathcal{V}$, we have that: $\zeta_t(x^{\prime}_t) = \sum_{\bm{x}_{t+1:T}}p^{\alpha}(\bm{x}_{t+1:T}|\bm{q}, \bm{x}_{0:t-1}, x^{\prime}_t)$.

Figures (6)

  • Figure 1: Toy example comparing the target power distribution $p^\alpha$, the low-temperature distribution, and the empirical histograms of MCMC and our method ($\alpha=4$, $\tau=1/\alpha=0.25$).
  • Figure 2: Pass@1 performance and per-prompt inference time for Qwen-2.5-7B (top left), Qwen2.5-Math-7B (top right), DeepSeek-Math-7B (bottom left), and DeepSeek-Math-7B-RL (bottom right).
  • Figure 3: Pass@K performance on MATH500 with Qwen2.5-Math-7B of the base model, GRPO, MCMC, and our method. GRPO improves pass@1 but quickly shows diversity collapse.
  • Figure 4: Pass@1 on MATH500, HumanEval, and GPQA for Qwen2.5-Math-7B under different hyperparameters.
  • Figure 5: Pass@k performance $k\in\{1,2,4,8,16\}$ on out-of-domain tasks ((a) HumanEval and (b) GPQA). On HumanEval, GRPO exhibits similar scaling with $k$ to the base model (no obvious diversity collapse). On GPQA, GRPO improves pass@1 but scales more weakly with $k$, indicating reduced diversity even on an out-of-domain benchmark. In contrast, power sampling (MCMC and ours) improves pass@1 while preserving strong pass@k scaling.
  • ...and 1 more figures

Theorems & Definitions (11)

  • Theorem 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Theorem 1.1
  • proof
  • Lemma 1.2
  • proof
  • Lemma 1.3
  • proof
  • Corollary 1.4
  • ...and 1 more