Reasoning Models Reason Well, Until They Don't

TL;DR

This work investigates the limits of reasoning in large reasoning models (LRMs) by introducing the Deep Reasoning Dataset (DeepRD), a generator-driven suite that scales reasoning complexity along two axes: lookahead $L$ and branching $B$. Through controlled experiments on graph connectivity and proof planning, the authors show LRMs exhibit abrupt performance drops as complexity increases and fail to generalize beyond training-distribution complexity, even when token budgets are ample. They compare LRMs to standard LLMs on NLGraph and demonstrate that improvements from prompting techniques and self-verification do not fully resolve depth-dependent failures. By analyzing real-world datasets (open graph benchmarks and NaturalProofs) and a broad set of failure modes, the study reveals a long tail of harder instances that LRMs struggle with, underscoring the need for methods that promote robust, out-of-distribution reasoning. Overall, DeepRD provides a scalable, controllable platform to benchmark reasoning under increasing complexity and highlights the practical utility of LRMs while clarifying their limitations for high-stakes, complex reasoning tasks.

Abstract

Large language models (LLMs) have shown significant progress in reasoning tasks. However, recent studies show that transformers and LLMs fail catastrophically once reasoning problems exceed modest complexity. We revisit these findings through the lens of large reasoning models (LRMs) -- LLMs fine-tuned with incentives for step-by-step argumentation and self-verification. LRM performance on graph and reasoning benchmarks such as NLGraph seem extraordinary, with some even claiming they are capable of generalized reasoning and innovation in reasoning-intensive fields such as mathematics, physics, medicine, and law. However, by more carefully scaling the complexity of reasoning problems, we show existing benchmarks actually have limited complexity. We develop a new dataset, the Deep Reasoning Dataset (DeepRD), along with a generative process for producing unlimited examples of scalable complexity. We use this dataset to evaluate model performance on graph connectivity and natural language proof planning. We find that the performance of LRMs drop abruptly at sufficient complexity and do not generalize. We also relate our LRM results to the distributions of the complexities of large, real-world knowledge graphs, interaction graphs, and proof datasets. We find the majority of real-world examples fall inside the LRMs' success regime, yet the long tails expose substantial failure potential. Our analysis highlights the near-term utility of LRMs while underscoring the need for new methods that generalize beyond the complexity of examples in the training distribution.

Figures (22)

• Figure 1: An example from DeepRD. The example has complexity metrics: lookahead $L=2$ and number of branches $B=2$ (see Section \ref{['sec:complexity']} for definitions of complexity metrics). The same underlying graph is transformed into a symbolic graph query for an LLM to find a valid path, and into a natural language query for an LLM to plan the next step in a proof.
• Figure 2: We evaluate the models V3, R1, 4o, o3-mini, and o3 on reasoning about graph connectivity. We show the average full path accuracy for each $B, L$ pair, for $B= 2, 4, 8, 16$ and $L$ between 2 and 800.
• Figure 3: For the graph connectivity problem, we show the average full path accuracy for test cases with graph depths between 2 to 1536, when constraining the graph to be a chain graph (one node after the other).
• Figure 4: For the graph connectivity problem, we show the average token completion count for $B=8$ and $L$ between 2 and 800. We can only run the large o3 on a small range of $L$ due to budget constraints.
• Figure 5: We evaluate the models on the proof planning task. We show the average next step prediction accuracy for each $(B, L)$ pair, for $B= 2, 4, 8, 16$ and $L$ between 2 and 800.