Optimizing Anytime Reasoning via Budget Relative Policy Optimization

TL;DR

This work tackles the bottleneck of scalable test-time reasoning in large language models by introducing AnytimeReasoner, a budgeted RL framework that samples thinking budgets from a prior and trains with verifiable dense rewards. It decouples thinking and summarization into separate policies and introduces Budget Relative Policy Optimization (BRPO) to reduce variance and improve credit assignment. Through extensive ablations and experiments on math-domain benchmarks and a 7B-model, it demonstrates robust gains over GRPO across thinking budgets, enabling more token-efficient and flexible reasoning. The approach has practical impact for deploying responsive LLM services under varying resource constraints while maintaining accuracy.

Abstract

Scaling test-time compute is crucial for enhancing the reasoning capabilities of large language models (LLMs). Existing approaches typically employ reinforcement learning (RL) to maximize a verifiable reward obtained at the end of reasoning traces. However, such methods optimize only the final performance under a large and fixed token budget, which hinders efficiency in both training and deployment. In this work, we present a novel framework, AnytimeReasoner, to optimize anytime reasoning performance, which aims to improve token efficiency and the flexibility of reasoning under varying token budget constraints. To achieve this, we truncate the complete thinking process to fit within sampled token budgets from a prior distribution, compelling the model to summarize the optimal answer for each truncated thinking for verification. This introduces verifiable dense rewards into the reasoning process, facilitating more effective credit assignment in RL optimization. We then optimize the thinking and summary policies in a decoupled manner to maximize the cumulative reward. Additionally, we introduce a novel variance reduction technique, Budget Relative Policy Optimization (BRPO), to enhance the robustness and efficiency of the learning process when reinforcing the thinking policy. Empirical results in mathematical reasoning tasks demonstrate that our method consistently outperforms GRPO across all thinking budgets under various prior distributions, enhancing both training and token efficiency.

Figures (12)

• Figure 1: Left: We optimize anytime reasoning by sampling thinking budgets from a prior distribution $p_\mathcal{B}$ and maximizing the rewards at sampled budgets to push up the area under the curve. This objective naturally introduces verifiable dense rewards into the thinking process. Right: Budget Relative Policy Optimization (BRPO) leverages these dense rewards to improve advantage estimation via the Monte Carlo return ($R$) and an interpolated baseline that combines current progress ($V_1$) and the average return within the rollout group ($V_2$).
• Figure 2: By introducing dense rewards, we achieve better credit assignment during RL training. We assume a uniform distribution over thinking budgets and omit the probability for simplicity.
• Figure 3: Left: The correlation coefficient of $V_1$ and $V_2$ with $R(x, z, j_t)$. Right: The normalized variance of our BRPO. We evaluate the R1-Distill-1.5B model under the scenario where $\lambda=0.5$, and $p_{\mathcal{B}}$ is a uniform distribution over $\{1000, 2000, ..., 8000 \}$.
• Figure 4: The comparison of anytime reasoning performance between GRPO and our AnytimeReasoner with various prior budget distributions. Notably, the accuracies at the maximum token budget (8000) reflect the performance in the standard reasoning task.
• Figure 5: Ablation on verifiable dense rewards. For GRPO+length_penalty_v1, we follow aggarwal2025l10, assigning reward $1 - \frac{0.2|z|}{b_m}$ for the correct answer and 0 for wrong answer. For GRPO+length_penalty_v2, we follow arora2025training with $\alpha$ as 0.2.