Increasing the Thinking Budget is Not All You Need
Ignacio Iacobacci, Zhaozhi Qian, Faroq AL-Tam, Muhammad AL-Qurishi, Riad Souissi
TL;DR
This paper investigates how the thinking budget in LLM reasoning influences performance and cost, arguing that more thinking tokens alone are not always beneficial. It introduces a framework to compare reasoning strategies (Vanilla, Self-consistency, Summary, Reflection) across varying budgets and computes their interactions with API-call patterns. Key findings show that while larger budgets help, the Summary configuration often yields the best results, outperforming simple budget extension and sometimes outperforming Self-consistency. The work provides actionable guidelines for allocating compute across reasoning strategies and lays a foundation for more efficient, adaptive deployment of reasoning-enabled AI systems.
Abstract
Recently, a new wave of thinking-capable Large Language Models has emerged, demonstrating exceptional capabilities across a wide range of reasoning benchmarks. Early studies have begun to explore how the amount of compute in terms of the length of the reasoning process, the so-called thinking budget, impacts model performance. In this work, we propose a systematic investigation of the thinking budget as a key parameter, examining its interaction with various configurations such as self-consistency, reflection, and others. Our goal is to provide an informative, balanced comparison framework that considers both performance outcomes and computational cost. Among our findings, we discovered that simply increasing the thinking budget is not the most effective use of compute. More accurate responses can instead be achieved through alternative configurations, such as self-consistency and self-reflection.