One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs

TL;DR

This work introduces CounterMATH, a benchmark designed to probe counterexample-driven conceptual reasoning in mathematical LLMs, addressing the limitation that many models rely on exposure to proofs rather than true conceptual understanding. It builds CounterMATH from 1,216 university-level statement–rationale pairs derived from textbooks and pairs this with a data-engineering framework to automatically curate and refine counterexample-focused training data. Through extensive evaluations across a wide range of models, the authors show substantial gaps in counterexample-based reasoning, especially in topology and real analysis, while a modest finetuning regime (1,025 samples) yields improvements on CounterMATH and transfers to OOD benchmarks. The work provides a new benchmark and a practical training framework for advancing genuine mathematical reasoning in LLMs, highlighting promising directions for future research in higher-level mathematical domains.

Abstract

Leveraging mathematical Large Language Models (LLMs) for proof generation is a fundamental topic in LLMs research. We argue that the ability of current LLMs to prove statements largely depends on whether they have encountered the relevant proof process during training. This reliance limits their deeper understanding of mathematical theorems and related concepts. Inspired by the pedagogical method of "proof by counterexamples" commonly used in human mathematics education, our work aims to enhance LLMs' ability to conduct mathematical reasoning and proof through counterexamples. Specifically, we manually create a high-quality, university-level mathematical benchmark, CounterMATH, which requires LLMs to prove mathematical statements by providing counterexamples, thereby assessing their grasp of mathematical concepts. Additionally, we develop a data engineering framework to automatically obtain training data for further model improvement. Extensive experiments and detailed analyses demonstrate that CounterMATH is challenging, indicating that LLMs, such as OpenAI o1, have insufficient counterexample-driven proof capabilities. Moreover, our exploration into model training reveals that strengthening LLMs' counterexample-driven conceptual reasoning abilities is crucial for improving their overall mathematical capabilities. We believe that our work offers new perspectives on the community of mathematical LLMs.

Figures (8)

• Figure 1: Comparison between drill-based learning and example-based learning. The first two math LLMs fail when confronted with advanced mathematics, and "Proving by examples" is a highly creative and concept-intensive mathematical skill.
• Figure 2: Overview the construction process of CounterMATH. CounterMATH was first extracted from photocopied mathematical textbooks by crowd-sourced labelers with the OCR tool. For the next step, authors with bachelor degrees in applied mathematics as annotation experts would filter and correct improper statement-rationale pairs. Finally, GPT-4o was prompted to translate the validated data into English under experts' supervision.
• Figure 3: Data Distribution of CounterMATH.
• Figure 4: The overview of our training data engineering framework.
• Figure 5: Fine-grained evaluation results of different fields in CounterMATH.