Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches

Linyang He, Qiyao Yu, Hanze Dong, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, Nima Mesgarani

Abstract

Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a central question in artificial intelligence and cognitive science. As LLMs are increasingly integrated into scientific workflows, rigorous evaluation of their mathematical capabilities becomes a practical necessity. Existing benchmarks are limited by synthetic settings and data contamination. We present LiveMathematicianBench, a dynamic multiple-choice benchmark for research-level mathematical reasoning built from recent arXiv papers published after model training cutoffs. By grounding evaluation in newly published theorems, it provides a realistic testbed beyond memorized patterns. The benchmark introduces a thirteen-category logical taxonomy of theorem types (e.g., implication, equivalence, existence, uniqueness), enabling fine-grained evaluation across reasoning forms. It employs a proof-sketch-guided distractor pipeline that uses high-level proof strategies to construct plausible but invalid answer choices reflecting misleading proof directions, increasing sensitivity to genuine understanding over surface-level matching. We also introduce a substitution-resistant mechanism to distinguish answer recognition from substantive reasoning. Evaluation shows the benchmark is far from saturated: Gemini-3.1-pro-preview, the best model, achieves only 43.5%. Under substitution-resistant evaluation, accuracy drops sharply: GPT-5.4 scores highest at 30.6%, while Gemini-3.1-pro-preview falls to 17.6%, below the 20% random baseline. A dual-mode protocol reveals that proof-sketch access yields consistent accuracy gains, suggesting models can leverage high-level proof strategies for reasoning. Overall, LiveMathematicianBench offers a scalable, contamination-resistant testbed for studying research-level mathematical reasoning in LLMs.

LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches

Abstract

Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a central question in artificial intelligence and cognitive science. As LLMs are increasingly integrated into scientific workflows, rigorous evaluation of their mathematical capabilities becomes a practical necessity. Existing benchmarks are limited by synthetic settings and data contamination. We present LiveMathematicianBench, a dynamic multiple-choice benchmark for research-level mathematical reasoning built from recent arXiv papers published after model training cutoffs. By grounding evaluation in newly published theorems, it provides a realistic testbed beyond memorized patterns. The benchmark introduces a thirteen-category logical taxonomy of theorem types (e.g., implication, equivalence, existence, uniqueness), enabling fine-grained evaluation across reasoning forms. It employs a proof-sketch-guided distractor pipeline that uses high-level proof strategies to construct plausible but invalid answer choices reflecting misleading proof directions, increasing sensitivity to genuine understanding over surface-level matching. We also introduce a substitution-resistant mechanism to distinguish answer recognition from substantive reasoning. Evaluation shows the benchmark is far from saturated: Gemini-3.1-pro-preview, the best model, achieves only 43.5%. Under substitution-resistant evaluation, accuracy drops sharply: GPT-5.4 scores highest at 30.6%, while Gemini-3.1-pro-preview falls to 17.6%, below the 20% random baseline. A dual-mode protocol reveals that proof-sketch access yields consistent accuracy gains, suggesting models can leverage high-level proof strategies for reasoning. Overall, LiveMathematicianBench offers a scalable, contamination-resistant testbed for studying research-level mathematical reasoning in LLMs.

Paper Structure

This paper contains 40 sections, 58 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Benchmark
  4. Evaluation Results
  5. Even frontier models suffer.
  6. Category-wise pattern and monthly trajectories.
  7. Substitution-resistant options sharply reduce accuracy.
  8. Proof sketches yields a consistent improvement.
  9. Endnote and Discussion
  10. Related Work
  11. Standard and Advanced Mathematical Benchmarks
  12. Contamination Mitigation, Formal Mathematics, and Frontier-Level Evaluation
  13. Human Validation Protocol
  14. Detailed Logical Taxonomy of Theorem Categories
  15. 1. Algorithmic / Constructive.
  16. ...and 25 more sections

Figures (8)

  • Figure 1: Overall accuracy on LiveMathematicianBench across different LLMs. Even recent frontier models remain far from saturation. Random: 20%.
  • Figure 2: Our benchmark construction includes 7 stages.
  • Figure 3: Category statistics for the hard split of LiveMathematicianBench. (a) aggregate distribution of logical categories across all months. (b) month-by-month category breakdown from November 2025 to February 2026. Counts correspond to category memberships rather than unique questions, since a single theorem can receive multiple logical labels.
  • Figure 4: (a) accuracy by logical category aggregated across benchmark months; rare categories with fewer than ten examples are grouped into Other. (b) monthly overall accuracy from November 2025 to February 2026.
  • Figure 5: (a) Accuracy by answer-choice format on LiveMathematicianBench. Each model is evaluated separately on the original-choice subset and the substitution-resistant subset. Ranked by substitution-resistant performance. (b) Accuracy gain from adding proof sketches. Each panel compares the same model setting with and without sketch access across benchmark months.
  • ...and 3 more figures