Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems

Qifan Zhang, Jianhao Ruan, Aochuan Chen, Kang Zeng, Nuo Chen, Jing Tang, Jia Li

TL;DR

GrAlgoBench introduces a graph-algorithm benchmark to rigorously evaluate large reasoning models on long-context reasoning tasks. By organizing problems into Enumeration, Exploration, and Intuition across nine tasks with scalable graph sizes (up to 160 nodes) sourced from real-world networks, the work enables precise programmatic evaluation via metrics like pass@k and cons@k. Key findings show accuracy collapse as context grows and a pronounced, largely ineffective self-verification (over-thinking) behavior in LRMs. The benchmark demonstrates strong cross-domain relevance and correlates with established reasoning benchmarks, offering a practical foundation for advancing LRMs' reasoning robustness and efficiency.

Abstract

Large Reasoning Models (LRMs) have advanced rapidly; however, existing benchmarks in mathematics, code, and common-sense reasoning remain limited. They lack long-context evaluation, offer insufficient challenge, and provide answers that are difficult to verify programmatically. We introduce GrAlgoBench, a benchmark designed to evaluate LRMs through graph algorithm problems. Such problems are particularly well suited for probing reasoning abilities: they demand long-context reasoning, allow fine-grained control of difficulty levels, and enable standardized, programmatic evaluation. Across nine tasks, our systematic experiments reveal two major weaknesses of current LRMs. First, accuracy deteriorates sharply as context length increases, falling below 50% once graphs exceed 120 nodes. This degradation is driven by frequent execution errors, weak memory, and redundant reasoning. Second, LRMs suffer from an over-thinking phenomenon, primarily caused by extensive yet largely ineffective self-verification, which inflates reasoning traces without improving correctness. By exposing these limitations, GrAlgoBench establishes graph algorithm problems as a rigorous, multidimensional, and practically relevant testbed for advancing the study of reasoning in LRMs. Code is available at https://github.com/Bklight999/GrAlgoBench.

Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems

TL;DR

GrAlgoBench introduces a graph-algorithm benchmark to rigorously evaluate large reasoning models on long-context reasoning tasks. By organizing problems into Enumeration, Exploration, and Intuition across nine tasks with scalable graph sizes (up to 160 nodes) sourced from real-world networks, the work enables precise programmatic evaluation via metrics like pass@k and cons@k. Key findings show accuracy collapse as context grows and a pronounced, largely ineffective self-verification (over-thinking) behavior in LRMs. The benchmark demonstrates strong cross-domain relevance and correlates with established reasoning benchmarks, offering a practical foundation for advancing LRMs' reasoning robustness and efficiency.

Abstract

Large Reasoning Models (LRMs) have advanced rapidly; however, existing benchmarks in mathematics, code, and common-sense reasoning remain limited. They lack long-context evaluation, offer insufficient challenge, and provide answers that are difficult to verify programmatically. We introduce GrAlgoBench, a benchmark designed to evaluate LRMs through graph algorithm problems. Such problems are particularly well suited for probing reasoning abilities: they demand long-context reasoning, allow fine-grained control of difficulty levels, and enable standardized, programmatic evaluation. Across nine tasks, our systematic experiments reveal two major weaknesses of current LRMs. First, accuracy deteriorates sharply as context length increases, falling below 50% once graphs exceed 120 nodes. This degradation is driven by frequent execution errors, weak memory, and redundant reasoning. Second, LRMs suffer from an over-thinking phenomenon, primarily caused by extensive yet largely ineffective self-verification, which inflates reasoning traces without improving correctness. By exposing these limitations, GrAlgoBench establishes graph algorithm problems as a rigorous, multidimensional, and practically relevant testbed for advancing the study of reasoning in LRMs. Code is available at https://github.com/Bklight999/GrAlgoBench.
Paper Structure (31 sections, 6 equations, 28 figures, 8 tables)

This paper contains 31 sections, 6 equations, 28 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Benchmark Construction
  3. Graph Reasoning Taxonomy
  4. Task Design
  5. Graph Collection
  6. Evaluation
  7. Exploring LRMs’ Long-Context Reasoning Ability (RQ1)
  8. Exploring Underlying Mechanisms of LRMs' Over-Thinking (RQ2)
  9. Discussion
  10. Conclusion
  11. Ethic Statement
  12. Reproductivity Statement
  13. The Use of Large Language Models (LLMs)
  14. Related Works
  15. Experimental Setup
  16. ...and 16 more sections

Figures (28)

  • Figure 1: Benchmark overview.
  • Figure 2: Illustrative problem description.
  • Figure 3: Models pass@k performance across different context length.
  • Figure 4: Error type distributions across reasoning taxonomies for Qwen3-32B (top) and its non-reasoning variant Qwen3-32B-no-thinking (bottom).
  • Figure 5: High-entropy tokens in LRMs inference traces
  • ...and 23 more figures