MathClean: A Benchmark for Synthetic Mathematical Data Cleaning

TL;DR

MathClean introduces a rigorous benchmark to evaluate cleaning of synthetic mathematical data, addressing the need for high-quality synthetic QA in LLM pretraining. It provides 4,000 questions and 4,000 answers across correct/erroneous categories, with detailed error-type annotations and diverse augmentation prompts to simulate real-world data faults. Extensive experiments across closed- and open-source LLMs and reward-model baselines reveal that even leading models struggle with correctness and error-type detection, validating MathClean’s difficulty and utility for guiding data-cleaning methods. The benchmark offers practical impact by enabling targeted improvements in data generation and filtering for math-intensive AI systems.

Abstract

With the rapid development of large language models (LLMs), the quality of training data has become crucial. Among the various types of training data, mathematical data plays a key role in enabling LLMs to acquire strong reasoning abilities. While high-quality open-source data is important, it is often insufficient for pre-training, necessitating the addition of synthetic math problems. However, synthetic math questions and answers can introduce inaccuracies, which may degrade both the training data and web data. Therefore, an effective method for cleaning synthetic math data is essential. In this paper, we propose the MathClean benchmark to evaluate the effectiveness of math data cleaning models. The MathClean benchmark consists of 2,000 correct questions and 2,000 erroneous questions with additional 2,000 correct and erroneous answers sourced from augmented data based on GSM8K and MATH. Moreover, we also annotate error types for each question or answer, since it can assess whether models can correctly identify the error categories for future improvements. Finally, we present comprehensive evaluations using state-of-the-art (SOTA) models. Our results demonstrate that even strong models like GPT-o1 and DeepSeek-R1 perform poorly on this benchmark, highlighting the utility of MathClean. Our code and data is available at https://github.com/YuYingLi0/MathClean.

Figures (12)

• Figure 1: Overview of MathClean benchmark. MathClean proposed challenges in detecting errors in mathematical Questions and Answers, as well as in identifying the specific error types within these mathematical problems.
• Figure 2: Two erroneous examples from the MathClean dataset are presented: a simple difficulty question with an unrealistic error and a challenging difficulty problem, which includes both the question and the answer, with a logic error.
• Figure 3: Proportion of different difficulty levels and error types in the Question dataset of MathClean.
• Figure 4: Proportion of different difficulty levels and error types in the Answer dataset of MathClean.
• Figure 5: Failure case of GPT-o1 in the detection of question correctness in the MathClean benchmark, with explanation. The model fails to recognize the contradiction in the given conditions.