Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rational Metareasoning for Large Language Models

C. Nicolò De Sabbata, Theodore R. Sumers, Badr AlKhamissi, Antoine Bosselut, Thomas L. Griffiths

TL;DR

The paper tackles the rising inference-time costs of reasoning in large language models by introducing Rational Metareasoning for LLMs (RaM), a method that uses a Value of Computation (VOC)–inspired reward within an Expert Iteration framework to train models to adaptively decide when and how much reasoning to perform. By ranking multiple generated reasoning chains with VOC-based rewards and iteratively fine-tuning the policy, RaM reduces token generation by roughly 23–42% across diverse reasoning tasks while maintaining or improving task performance. The approach demonstrates adaptive computation, outperforming static prompting and prior bootstrapping methods, and generalizes to out-of-distribution benchmarks like MMLU-CF. The work also situates RaM within broader literature on inference-cost reduction and chain-of-thought reasoning, highlighting its cognitively grounded, single-model framework and potential for integration with instruction tuning and future extensions to agentic settings.

Abstract

Being prompted to engage in reasoning has emerged as a core technique for using large language models (LLMs), deploying additional inference-time compute to improve task performance. However, as LLMs increase in both size and adoption, inference costs are correspondingly becoming increasingly burdensome. How, then, might we optimize reasoning's cost-performance tradeoff? This work introduces a novel approach based on computational models of metareasoning used in cognitive science, training LLMs to selectively use intermediate reasoning steps only when necessary. We first develop a reward function that incorporates the Value of Computation by penalizing unnecessary reasoning, then use this reward function with Expert Iteration to train the LLM. Compared to few-shot chain-of-thought prompting and STaR, our method significantly reduces inference costs (20-37\% fewer tokens generated across three models) while maintaining task performance across diverse datasets.

Rational Metareasoning for Large Language Models

TL;DR

The paper tackles the rising inference-time costs of reasoning in large language models by introducing Rational Metareasoning for LLMs (RaM), a method that uses a Value of Computation (VOC)–inspired reward within an Expert Iteration framework to train models to adaptively decide when and how much reasoning to perform. By ranking multiple generated reasoning chains with VOC-based rewards and iteratively fine-tuning the policy, RaM reduces token generation by roughly 23–42% across diverse reasoning tasks while maintaining or improving task performance. The approach demonstrates adaptive computation, outperforming static prompting and prior bootstrapping methods, and generalizes to out-of-distribution benchmarks like MMLU-CF. The work also situates RaM within broader literature on inference-cost reduction and chain-of-thought reasoning, highlighting its cognitively grounded, single-model framework and potential for integration with instruction tuning and future extensions to agentic settings.

Abstract

Being prompted to engage in reasoning has emerged as a core technique for using large language models (LLMs), deploying additional inference-time compute to improve task performance. However, as LLMs increase in both size and adoption, inference costs are correspondingly becoming increasingly burdensome. How, then, might we optimize reasoning's cost-performance tradeoff? This work introduces a novel approach based on computational models of metareasoning used in cognitive science, training LLMs to selectively use intermediate reasoning steps only when necessary. We first develop a reward function that incorporates the Value of Computation by penalizing unnecessary reasoning, then use this reward function with Expert Iteration to train the LLM. Compared to few-shot chain-of-thought prompting and STaR, our method significantly reduces inference costs (20-37\% fewer tokens generated across three models) while maintaining task performance across diverse datasets.
Paper Structure (21 sections, 4 equations, 7 figures, 5 tables, 1 algorithm)

This paper contains 21 sections, 4 equations, 7 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Rational Metareasoning
  3. Rational Metareasoning with Large Language Models
  4. Reward modeling
  5. Metareasoning Training
  6. Experiments
  7. Datasets
  8. Baselines
  9. Training Details
  10. Results
  11. Performance vs Cost
  12. Adaptive computation
  13. Generalization
  14. Related Work
  15. Reducing inference costs
  16. ...and 6 more sections

Figures (7)

  • Figure 1: RaM training. We iteratively generate reasoning chains using the current policy, score and filter them to approximate the optimal policy, and then finetune the base policy.
  • Figure 2: Cost and performance. Accuracy is plotted against the output tokens. Our method, RaM, eliminates the need for few-shot prompting (reducing input tokens) and trains the model to use fewer reasoning tokens than STaR.
  • Figure 3: Ratio of output length reduction. RaM ’s output length relative to STaR’s, for Llama-3.2-3B. Tasks requiring heavier reasoning tend to exhibit a lower length reduction ratio, whereas those relying more on knowledge display a more pronounced length reduction.
  • Figure 4: Qualitative examples of reasoning processes for ARC dataset
  • Figure 5: Qualitative examples of reasoning processes for CommonsenseQA dataset
  • ...and 2 more figures