Evaluating the Logical Reasoning Abilities of Large Reasoning Models

TL;DR

LogiEval addresses a gap in evaluating pure logical reasoning in large reasoning models by introducing a domain-agnostic benchmark drawn from high-stakes exams that spans deductive, inductive, analogical, and abductive reasoning across 10 task formats. The authors evaluate seven 2025-era models (open-weighted and proprietary) and reveal that while models can exceed human performance on certain 4-choice argument analyses and analogies, they exhibit pronounced weaknesses in syllogistic and deductive tasks and show uneven generalization across formats. A key contribution is LogiEval-Hard, a challenging subset identified via small-model screening (3B parameters) that reliably predicts universal difficulties for larger models, exposing fundamental reasoning bottlenecks that persist across scales. These findings highlight the need for more robust, architecture-aware approaches to logical reasoning and demonstrate that small-model diagnostics can meaningfully forecast large-model challenges, with implications for safer, more reliable reasoning systems.

Abstract

Large reasoning models, often post-trained on long chain-of-thought (long CoT) data with reinforcement learning, achieve state-of-the-art performance on mathematical, coding, and domain-specific reasoning benchmarks. However, their logical reasoning capabilities - fundamental to human cognition and independent of domain knowledge - remain understudied. To address this gap, we introduce LogiEval, a holistic benchmark for evaluating logical reasoning in large reasoning models. LogiEval spans diverse reasoning types (deductive, inductive, analogical, and abductive) and task formats (e.g., logical sequence, argument analysis), sourced from high-quality human examinations (e.g., LSAT, GMAT). Our experiments demonstrate that modern reasoning models excel at 4-choice argument analysis problems and analogical reasoning, surpassing human performance, yet exhibit uneven capabilities across reasoning types and formats, highlighting limitations in their generalization. Our analysis reveals that human performance does not mirror model failure distributions. To foster further research, we curate LogiEval-Hard, a challenging subset identified through a novel screening paradigm where small-model failures (Qwen3-30B-A3B) reliably predict difficulties for larger models. Modern models show striking, consistent failures on LogiEval-Hard. This demonstrates that fundamental reasoning bottlenecks persist across model scales, and establishes LogiEval-Hard as both a diagnostic tool and a rigorous testbed for advancing logical reasoning in LLMs.

Figures (2)

• Figure 1: Model performance comparison on different subtasks
• Figure 2: Examples from the LoigEval benchmark