ReEfBench: Quantifying the Reasoning Efficiency of LLMs

TL;DR

ReEfBench introduces a neuro-symbolic, FOL-grounded framework to quantify LLM reasoning efficiency by separating logical depth from computational cost. The pipeline generates scalable reasoning datasets, uses LLM-based parsing to build formal reasoning graphs, and applies rule-based verification to yield six diagnostic metrics, then clusters models into four behavioral archetypes. Key findings show that deep reasoning does not require extended token generation, that mixing long and short CoT data can disrupt strategies, and that distillation yields behavioral mimicry without guaranteed depth gains, with capacity constraints playing a critical role. The work provides a principled, scalable tool for evaluating reasoning processes, with potential to guide training, prompting, and deployment choices in practical AI systems.

Abstract

Test-time scaling has enabled Large Language Models (LLMs) to tackle complex reasoning, yet the limitations of current Chain-of-Thought (CoT) evaluation obscures whether performance gains stem from genuine reasoning or mere verbosity. To address this, (1) we propose a novel neuro-symbolic framework for the non-intrusive, comprehensive process-centric evaluation of reasoning. (2) Through this lens, we identify four distinct behavioral prototypes and diagnose the failure modes. (3) We examine the impact of inference mode, training strategy, and model scale. Our analysis reveals that extended token generation is not a prerequisite for deep reasoning. Furthermore, we reveal critical constraints: mixing long and short CoT data in training risks in premature saturation and collapse, while distillation into smaller models captures behavioral length but fails to replicate logical efficacy due to intrinsic capacity limits.

Figures (10)

• Figure 1: Overview of our framework. We generate scalable, controllable first-order logic (FOL) data that enables precise verification of logical depth (Phase A). Target LLM generates a response for each query (Phase B). The pipeline parses a target LLM’s response into formal representations (Phase C), computes its logical depth and correctness via rule-based verifiers (Phase D), evaluates the normalized output along six dimensions—Logical Depth, Cost, Exploration, Efficiency, Coherence, and Redundancy—and finally classifies it into one of four behavioral prototypes: EffectiveSolver, DeepWanderer, HollowMimic, and LazyGuesser (Phase E).
• Figure 2: Example of our dataset: Premise, Intermediate and Conclusion. Dataset Complexity = max(Logical depth) = 2.
• Figure 3: Models plotted by Logical Depth ($S_\textit{ld}$) vs. Cost ($S_\textit{cost}$); triangles = Long CoT, circles = Short CoT, stars = centroids. Model IDs listed in Table \ref{['tab:model_classification']}.
• Figure 4: Performance trajectories across varying complexity levels (3--11). The visualization illustrates three distinct behavioral patterns defined in Section 4.3: (1) Adaptive Scaling (e.g., Qwen3-235B-thinking, Claude-Opus-4.5.long); (2) Diluted Expansion (e.g., Claude-Sonnet-4.5.short); and (3) Saturation (e.g., DS-Distill-Qwen-7B) & Collapse (e.g., Qwen3-4B.long). Complete trajectories for all evaluated models are presented in Figure \ref{['fig:trajectory']} in Appendix \ref{['app:experiment_details']}.
• Figure 5: Qwen3-32B vs 14B vs 8B vs 4B. Three subplots compare key reasoning behaviors (logical depth, reflection ratio and quality) across model sizes as problem complexity increases. Smaller distilled models exhibit more sophisticated behaviors, but fail to emulate behavioral efficiency and cannot translate these sophisticated behaviors into deeper reasoning.