Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognition

Muyang Zhao, Qi Qi, Hao Sun

TL;DR

The paper tackles budgeted inference-time reasoning for large language models across multiple tasks under a global token budget. It formalizes the problem as an Ordered Stochastic Multiple-Choice Knapsack Problem (OS-MCKP) and introduces ROI-Reasoning, a two-stage framework combining Meta-Cognitive Fine-Tuning (MFT) and Rationality-Aware Reinforcement Learning (RARL) to enable budget-aware rationality. Across budgeted mathematical reasoning benchmarks, the approach yields higher scores and notably lower regret than baselines, including some large-scale models, under tight budgets. This work provides a principled pathway to endow LLMs with intrinsic meta-cognitive control for efficient, ROI-driven computation in sequential, budget-constrained settings, with potential extension to broader agentic tasks.

Abstract

Large language models (LLMs) can achieve strong reasoning performance with sufficient computation, but they do not inherently know how much computation a task requires. We study budgeted inference-time reasoning for multiple tasks under a strict global token constraint and formalize it as a Ordered Stochastic Multiple-Choice Knapsack Problem(OS-MCKP). This perspective highlights a meta-cognitive requirement -- anticipating task difficulty, estimating return over investment (ROI), and allocating computation strategically. We propose ROI-Reasoning, a two-stage framework that endows LLMs with intrinsic, budget-aware rationality. In the first stage, Meta-Cognitive Fine-Tuning teaches models to predict reasoning cost and expected utility before generation, enabling explicit solve-or-skip decisions. Next, Rationality-Aware Reinforcement Learning optimizes sequential decision making under a hard token budget, allowing models to learn long-horizon allocation strategies. Across budgeted mathematical reasoning benchmarks, ROI-Reasoning consistently improves overall score while substantially reducing regret under tight computation budgets.

ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognition

TL;DR

The paper tackles budgeted inference-time reasoning for large language models across multiple tasks under a global token budget. It formalizes the problem as an Ordered Stochastic Multiple-Choice Knapsack Problem (OS-MCKP) and introduces ROI-Reasoning, a two-stage framework combining Meta-Cognitive Fine-Tuning (MFT) and Rationality-Aware Reinforcement Learning (RARL) to enable budget-aware rationality. Across budgeted mathematical reasoning benchmarks, the approach yields higher scores and notably lower regret than baselines, including some large-scale models, under tight budgets. This work provides a principled pathway to endow LLMs with intrinsic meta-cognitive control for efficient, ROI-driven computation in sequential, budget-constrained settings, with potential extension to broader agentic tasks.

Abstract

Large language models (LLMs) can achieve strong reasoning performance with sufficient computation, but they do not inherently know how much computation a task requires. We study budgeted inference-time reasoning for multiple tasks under a strict global token constraint and formalize it as a Ordered Stochastic Multiple-Choice Knapsack Problem(OS-MCKP). This perspective highlights a meta-cognitive requirement -- anticipating task difficulty, estimating return over investment (ROI), and allocating computation strategically. We propose ROI-Reasoning, a two-stage framework that endows LLMs with intrinsic, budget-aware rationality. In the first stage, Meta-Cognitive Fine-Tuning teaches models to predict reasoning cost and expected utility before generation, enabling explicit solve-or-skip decisions. Next, Rationality-Aware Reinforcement Learning optimizes sequential decision making under a hard token budget, allowing models to learn long-horizon allocation strategies. Across budgeted mathematical reasoning benchmarks, ROI-Reasoning consistently improves overall score while substantially reducing regret under tight computation budgets.
Paper Structure (43 sections, 10 equations, 5 figures, 2 tables)

This paper contains 43 sections, 10 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Knapsack Problems.
  5. Methodology
  6. Overview
  7. Meta-Cognitive Fine-Tuning
  8. Rationality-Aware Reinforcement Learning
  9. Experiments
  10. Data.
  11. Test Paper Construction and Budgets.
  12. Models and Baselines.
  13. Evaluation Metrics.
  14. Results.
  15. Conclusion
  16. ...and 28 more sections

Figures (5)

  • Figure 1: LLM is taking a multi-question test under a fixed token budget.
  • Figure 2: Overview of the ROI-Reasoning framework, including Meta-Cognitive Fine-Tuning and Rationality-Aware Reinforcement Learning.
  • Figure 3: Structure of the ROI-Reasoning input prompt.
  • Figure 4: Token length distributions across varying difficulties and constraints. The histograms compare the reasoning length patterns of Qwen2.5-1.5B-Instruct, Qwen2.5-1.5B-Instruct + MFT, and Qwen2.5-1.5B-Instruct + MFT + RARL under four distinct settings.
  • Figure 5: Composition of predicted-level errors (correst, under-, and over-estimated) under different budgets on Medium and Hard test papers, comparing RFT and RFT+RARL.