Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unmasking Reasoning Processes: A Process-aware Benchmark for Evaluating Structural Mathematical Reasoning in LLMs

Xiang Zheng, Weiqi Zhai, Wei Wang, Boyu Yang, Wenbo Li, Ruixiang Luo, Haoxiang Sun, Yucheng Wang, Zhengze Li, Meng Wang, Yuetian Du, Guojie Lin, Yaxuan Wang, Xiaoxiao Xu, Yanhu Mo, Xuan Ren, Hu Wei, Ze Xu

TL;DR

ReasoningMath-Plus addresses the saturation of mathematical reasoning benchmarks by focusing on structural reasoning through $150$ curated problems annotated with minimal reasoning skeletons. It introduces a two-branch process-aware evaluation: HCRS, a hazard-aware, skeleton-guided deterministic score, and PRM, an outcome-conditioned learned verifier trained on skeletons; together they reveal gaps between final-answer accuracy and reasoning quality. Empirically, final-answer accuracy reaches up to $5.8/10$ while HCRS-based holistic scores average $4.36/10$ (best $5.14/10$), illustrating that answer-based metrics overestimate reasoning robustness. The work demonstrates strong alignment with human judgments and shows that simple rule-based penalties can improve verifier performance, offering a scalable, transparent paradigm for diagnosing and verifying long-horizon mathematical reasoning.

Abstract

Recent large language models (LLMs) achieve near-saturation accuracy on many established mathematical reasoning benchmarks, raising concerns about their ability to diagnose genuine reasoning competence. This saturation largely stems from the dominance of template-based computation and shallow arithmetic decomposition in existing datasets, which underrepresent reasoning skills such as multi-constraint coordination, constructive logical synthesis, and spatial inference. To address this gap, we introduce ReasoningMath-Plus, a benchmark of 150 carefully curated problems explicitly designed to evaluate structural reasoning. Each problem emphasizes reasoning under interacting constraints, constructive solution formation, or non-trivial structural insight, and is annotated with a minimal reasoning skeleton to support fine-grained process-level evaluation. Alongside the dataset, we introduce HCRS (Hazard-aware Chain-based Rule Score), a deterministic step-level scoring function, and train a Process Reward Model (PRM) on the annotated reasoning traces. Empirically, while leading models attain relatively high final-answer accuracy (up to 5.8/10), HCRS-based holistic evaluation yields substantially lower scores (average 4.36/10, best 5.14/10), showing that answer-only metrics can overestimate reasoning robustness.

Unmasking Reasoning Processes: A Process-aware Benchmark for Evaluating Structural Mathematical Reasoning in LLMs

TL;DR

ReasoningMath-Plus addresses the saturation of mathematical reasoning benchmarks by focusing on structural reasoning through curated problems annotated with minimal reasoning skeletons. It introduces a two-branch process-aware evaluation: HCRS, a hazard-aware, skeleton-guided deterministic score, and PRM, an outcome-conditioned learned verifier trained on skeletons; together they reveal gaps between final-answer accuracy and reasoning quality. Empirically, final-answer accuracy reaches up to while HCRS-based holistic scores average (best ), illustrating that answer-based metrics overestimate reasoning robustness. The work demonstrates strong alignment with human judgments and shows that simple rule-based penalties can improve verifier performance, offering a scalable, transparent paradigm for diagnosing and verifying long-horizon mathematical reasoning.

Abstract

Recent large language models (LLMs) achieve near-saturation accuracy on many established mathematical reasoning benchmarks, raising concerns about their ability to diagnose genuine reasoning competence. This saturation largely stems from the dominance of template-based computation and shallow arithmetic decomposition in existing datasets, which underrepresent reasoning skills such as multi-constraint coordination, constructive logical synthesis, and spatial inference. To address this gap, we introduce ReasoningMath-Plus, a benchmark of 150 carefully curated problems explicitly designed to evaluate structural reasoning. Each problem emphasizes reasoning under interacting constraints, constructive solution formation, or non-trivial structural insight, and is annotated with a minimal reasoning skeleton to support fine-grained process-level evaluation. Alongside the dataset, we introduce HCRS (Hazard-aware Chain-based Rule Score), a deterministic step-level scoring function, and train a Process Reward Model (PRM) on the annotated reasoning traces. Empirically, while leading models attain relatively high final-answer accuracy (up to 5.8/10), HCRS-based holistic evaluation yields substantially lower scores (average 4.36/10, best 5.14/10), showing that answer-only metrics can overestimate reasoning robustness.
Paper Structure (46 sections, 4 equations, 15 figures, 5 tables)

This paper contains 46 sections, 4 equations, 15 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mathematical and logical reasoning benchmarks.
  4. Evaluating reasoning processes.
  5. Benchmark Curation
  6. Motivation
  7. Data Collection
  8. Methodology
  9. Branch A: Skeleton-guided Diagnosis and HCRS Aggregation
  10. Motivation.
  11. HCRS Framework.
  12. Step Validity and Base Score.
  13. Format Deviation Penalty ($P_{\text{fmt}}$).
  14. Hazard Penalty ($P_{\text{haz}}$).
  15. Aggregation.
  16. ...and 31 more sections

Figures (15)

  • Figure 1: Overall framework.(a) Structured CoT elicitation for process-level evaluation. (b) Branch A (Skeleton-guided diagnosis): a judge checks step validity against a minimal set of necessary skeleton assertions (paraphrase-tolerant; not exact-match), aggregated by HCRS with format and hazard penalties. (c) Branch B (Outcome-conditioned verification): a PRM distilled from teacher-judge labels and applied at inference time using only the problem and the gold final answer.
  • Figure 2: Evaluation leaderboards and correlation analysis.(a) HCRS leaderboard across domains (grey segments indicate deductions from format and hazard penalties). (b) PRM process-score leaderboard under outcome-conditioned step verification. (c) Pearson correlation of evaluation methods (Reference-guided, teacher judge, and PRM) against human judgments.
  • Figure 3: Subject-wise capability analysis. Radar plots comparing (a) Holistic Score (0--100) and (b) Answer Accuracy (0--100%) across five domains. Curves correspond to the Best, Median, and Worst models selected by overall Holistic Score, together with the overall average.
  • Figure 4: Judge performance comparison. Comparison of candidate judge models based on average Pearson correlation with human annotations ($N{=}2,100$). Gemini-3-Pro exhibits the strongest alignment ($R{=}0.64$).
  • Figure 5: Bimodal distribution of process quality conditioned on correct final answers. Histogram of the process-only HCRS score $S_{\text{HCRS}}$ ... for 996 answer-correct traces (out of 2,100 total model traces). Vertical dashed lines indicate cumulative thresholds at $S_{\text{HCRS}}\le k$ ($k\in\{1,2,3,4,5\}$), with callouts reporting cumulative counts and percentages. Notably, $6.63\%$ (66/996) of correct answers have $S_{\text{HCRS}}\le 3$, suggesting that outcome correctness can coincide with low-quality or inconsistent reasoning ("lucky guesses").
  • ...and 10 more figures