RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics

TL;DR

RealMath presents a continuous benchmark for evaluating LLMs on research-level mathematics by automatically harvesting verifiable statements from $arXiv$ papers and Mathematics Stack Exchange and turning them into fixed-answer QA pairs. The authors implement a refreshable data-pipeline, emphasize automated verification over proofs, and demonstrate that frontier models achieve notable accuracy on authentic research math, with performance varying by difficulty and domain. They also analyze errors, the impact of context, data contamination, and fine-tuning, showing that context and content quality significantly influence results. The findings suggest LLMs can be valuable assistants for mathematicians today, while the benchmark’s refreshable design mitigates contamination and keeps pace with evolving mathematical practice. RealMath thus offers a pragmatic, scalable framework for evaluating and improving AI assistants in mathematical research.

Abstract

Existing benchmarks for evaluating mathematical reasoning in large language models (LLMs) rely primarily on competition problems, formal proofs, or artificially challenging questions -- failing to capture the nature of mathematics encountered in actual research environments. We introduce RealMath, a novel benchmark derived directly from research papers and mathematical forums that assesses LLMs' abilities on authentic mathematical tasks. Our approach addresses three critical challenges: sourcing diverse research-level content, enabling reliable automated evaluation through verifiable statements, and designing a continually refreshable dataset to mitigate contamination risks. Experimental results across multiple LLMs reveal surprising capabilities in handling research mathematics compared to competition problems, suggesting current models may already serve as valuable assistants for working mathematicians despite limitations on highly challenging problems. The code and dataset for RealMath are publicly available.

Figures (15)

• Figure 1: LLM performance on a hard subset of RealMath from arXiv Mathematics papers.
• Figure 2: The data collection pipeline for arXiv papers. The core step is to ensure that each extracted theorem from arXiv papers has a single, exact answer. To maintain data quality, we apply filtering mechanisms, e.g., prompting an LLM to discard trivial samples that can be easily solved.
• Figure 3: Illustration of the theorem-to-QA conversion process and samples that are filtered out. The top panel shows examples of high-quality question-answer pairs generated from mathematical theorems that contain fixed, verifiable answers. The bottom panel provides examples of theorems that were filtered out due to ambiguity or the lack of a fixed answer.
• Figure 4: (a) Accuracy of the best-performing models across different difficulty levels of Math.arXiv and (b) distribution of difficulty levels on Math.arXiv. More details and results in \ref{['app:difficulty']}.
• Figure 5: Evaluation of model performance on the Math.arXiv dataset across domain-specific and temporal dimensions. Left: Accuracy across mathematical domains for two models, showing significant variation by topic. Right: Accuracy of GPT-4o-mini, Llama-3.1-405B, and Claude-3.5-Sonnet on questions published before vs. after their training cutoff.