AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms?

TL;DR

AlgBench introduces an algorithm-centric benchmarking framework to rigorously assess whether large reasoning models truly internalize algorithmic principles. By curating a contamination-free dataset of over 3,000 problems across 27 algorithms and evaluating 25 LRMs with Pass@$k$ and z-score normalization, the study uncovers substantial performance heterogeneity: models perform well on Euclidean-structured and non-optimized tasks but struggle with global- and heuristic-optimized problems, with scaling yielding limited gains in the harder domains. Error analysis reveals strategic over-shifts driven by low-entropy tokens, suggesting that prevailing problem-centric RL training fails to induce robust algorithmic reasoning. The findings advocate for an algorithm-centric RL paradigm to enable scalable, transferable algorithmic understanding in LRMs, with practical implications for designing future reasoning systems and benchmarks.

Abstract

Reasoning ability has become a central focus in the advancement of Large Reasoning Models (LRMs). Although notable progress has been achieved on several reasoning benchmarks such as MATH500 and LiveCodeBench, existing benchmarks for algorithmic reasoning remain limited, failing to answer a critical question: Do LRMs truly master algorithmic reasoning? To answer this question, we propose AlgBench, an expert-curated benchmark that evaluates LRMs under an algorithm-centric paradigm. AlgBench consists of over 3,000 original problems spanning 27 algorithms, constructed by ACM algorithmic experts and organized under a comprehensive taxonomy, including Euclidean-structured, non-Euclidean-structured, non-optimized, local-optimized, global-optimized, and heuristic-optimized categories. Empirical evaluations on leading LRMs (e.g., Gemini-3-Pro, DeepSeek-v3.2-Speciale and GPT-o3) reveal substantial performance heterogeneity: while models perform well on non-optimized tasks (up to 92%), accuracy drops sharply to around 49% on globally optimized algorithms such as dynamic programming. Further analysis uncovers \textbf{strategic over-shifts}, wherein models prematurely abandon correct algorithmic designs due to necessary low-entropy tokens. These findings expose fundamental limitations of problem-centric reinforcement learning and highlight the necessity of an algorithm-centric training paradigm for robust algorithmic reasoning.

Figures (11)

• Figure 1: The difference between math reasoning and algorithmic reasoning.
• Figure 2: The overview of the proposed benchmark.
• Figure 3: The normalized z-scores of LRMs on different algorithm taxonomies.
• Figure 4: Error type distributions across algorithmic reasoning taxonomies for Qwen3-8B.
• Figure 5: The results of models' parameters scaling.