Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs

TL;DR

This study empirically probes how reasoning length in large language models relates to answer correctness, revealing a non-monotonic relationship where neither very short nor very long chains maximize accuracy. By analyzing both sample-level and question-level dynamics on GSM8K and MATH with two reasoning-capable models, the authors show underthinking can occur on hard problems while overthinking persists on easy ones, and that length serves as a meaningful signal of perceived difficulty. They further demonstrate that a purely length-driven optimization (SimPO) can substantially reduce generated token length with minimal accuracy loss, highlighting potential for more efficient reasoning. Overall, the work calls for adaptive length control and self-awareness in reasoning length as a key axis of model capability and efficiency.

Abstract

Large language models (LLMs) are increasingly optimized for long reasoning, under the assumption that more reasoning leads to better performance. However, emerging evidence suggests that longer responses can sometimes degrade accuracy rather than improve it. In this paper, we conduct a systematic empirical study of the relationship between reasoning length and answer correctness. We find that LLMs tend to overthink simple problems, generating unnecessarily long outputs, and underthink harder ones, failing to extend their reasoning when it is most needed. This indicates that models might misjudge problem difficulty and fail to calibrate their response length appropriately. Furthermore, we investigate the effects of length reduction with a preference optimization algorithm when simply preferring the shorter responses regardless of answer correctness. Experiments show that the generation length can be significantly reduced while maintaining acceptable accuracy. Our findings highlight generation length as a meaningful signal for reasoning behavior and motivate further exploration into LLMs' self-awareness in reasoning length adaptation.

Figures (16)

• Figure 1: Reasoning length $L_r$ of the $r$-th shortest response v.s. accuracy $\text{Acc}_r$ ($N=10$). The red marker $r^*$ denotes the rank of responses with the highest accuracy. Results for R1-Distill and R1-Preview on the GSM8K and MATH datasets suggest that both overly short and excessively long reasoning can degrade performance.
• Figure 2: Percentage of questions for which the shortest correct response occurs at rank $i$.
• Figure 3: Average token length and perplexity of model responses across questions with different accuracy.
• Figure 4: Testing accuracy and average token length when applying the preference optimization algorithm SimPO meng2024simpo, trained with preference pairs favoring shorter generations irrespective of correctness.
• Figure 5: Average token length for correct and incorrect responses across different training steps.