Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data

Xuanming Zhang, Shwan Ashrafi, Aziza Mirsaidova, Amir Rezaeian, Miguel Ballesteros, Lydia B. Chilton, Zhou Yu, Dan Roth

TL;DR

This work tackles budget-aware reasoning by formalizing anytime reasoning and introducing the Anytime Index to quantify how quickly reasoning quality improves with more tokens. It couples this evaluation with an inference-time self-improvement method, Preference Data Prompting (PDP), which uses LLM-generated, contrastive reasoning pairs to guide intermediate steps without human labeling. Across NaturalPlan, AIME, and GPQA, PDP improves both intermediate and final performance and generally yields higher Anytime Indices across diverse models, including Grok-3, GPT-oss, GPT-4.1/4o, and Llama-3.3-70B, demonstrating that models can become faster and more reliable anytime reasoners under token budgets. The approach offers practical benefits for real-world deployments where latency and compute budgets constrain full deliberation, and it lays the groundwork for further training-time improvements and broader domain validation.

Abstract

We study the reasoning behavior of large language models (LLMs) under limited computation budgets. In such settings, producing useful partial solutions quickly is often more practical than exhaustive reasoning, which incurs high inference costs. Many real-world tasks, such as trip planning, require models to deliver the best possible output within a fixed reasoning budget. We introduce an anytime reasoning framework and the Anytime Index, a metric that quantifies how effectively solution quality improves as reasoning tokens increase. To further enhance efficiency, we propose an inference-time self-improvement method using LLM-synthesized preference data, where models learn from their own reasoning comparisons to produce better intermediate solutions. Experiments on NaturalPlan (Trip), AIME, and GPQA datasets show consistent gains across Grok-3, GPT-oss, GPT-4.1/4o, and LLaMA models, improving both reasoning quality and efficiency under budget constraints.

Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data

TL;DR

This work tackles budget-aware reasoning by formalizing anytime reasoning and introducing the Anytime Index to quantify how quickly reasoning quality improves with more tokens. It couples this evaluation with an inference-time self-improvement method, Preference Data Prompting (PDP), which uses LLM-generated, contrastive reasoning pairs to guide intermediate steps without human labeling. Across NaturalPlan, AIME, and GPQA, PDP improves both intermediate and final performance and generally yields higher Anytime Indices across diverse models, including Grok-3, GPT-oss, GPT-4.1/4o, and Llama-3.3-70B, demonstrating that models can become faster and more reliable anytime reasoners under token budgets. The approach offers practical benefits for real-world deployments where latency and compute budgets constrain full deliberation, and it lays the groundwork for further training-time improvements and broader domain validation.

Abstract

We study the reasoning behavior of large language models (LLMs) under limited computation budgets. In such settings, producing useful partial solutions quickly is often more practical than exhaustive reasoning, which incurs high inference costs. Many real-world tasks, such as trip planning, require models to deliver the best possible output within a fixed reasoning budget. We introduce an anytime reasoning framework and the Anytime Index, a metric that quantifies how effectively solution quality improves as reasoning tokens increase. To further enhance efficiency, we propose an inference-time self-improvement method using LLM-synthesized preference data, where models learn from their own reasoning comparisons to produce better intermediate solutions. Experiments on NaturalPlan (Trip), AIME, and GPQA datasets show consistent gains across Grok-3, GPT-oss, GPT-4.1/4o, and LLaMA models, improving both reasoning quality and efficiency under budget constraints.
Paper Structure (43 sections, 1 equation, 3 figures, 1 table)

This paper contains 43 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Anytime Reasoning Evaluation Framework
  3. Evaluation Pipeline
  4. Anytime Index
  5. LLM-Generated Preference Data Prompting
  6. Preference Data Generation
  7. Preference Data Prompting
  8. Experiments
  9. Dataset
  10. Models
  11. Evaluation Metrics
  12. Results
  13. Overall performance and model-specific trends.
  14. The role of contrastive preference supervision.
  15. Comparison with learning-from-mistakes baselines.
  16. ...and 28 more sections

Figures (3)

  • Figure 1: Overview of our anytime reasoning evaluation framework. The model generates $N$ CoT traces, each truncated at token budgets $b_i$ to evaluate intermediate solution quality $Q_i$. Final solutions are derived from full reasoning traces.
  • Figure 2: The Constraint Satisfaction Rate evaluated at different token budget checkpoints across different model families: Grok-3, GPT-4.1, and GPT-4o. Preference Data Prompting (dotted line) makes the models better anytime reasoners.
  • Figure 3: The accuracy (on AIME and GPQA) and constraint satisfaction rate (on NaturalPlan) evaluated at different token budget checkpoints across different models: Grok-3, Grok-3-mini, and GPT-4.1. Compared to other prompting techniques, Preference Data Prompting (red line) makes the models better anytime reasoners.