Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answers

Chaitanya Manem, Pratik Prabhanjan Brahma, Prakamya Mishra, Zicheng Liu, Emad Barsoum

TL;DR

SAND-Math addresses the bottleneck of scarce high-quality math training data for LLMs by introducing a fully automated data-generation pipeline that synthesizes novel, challenging problems and solutions. A key innovation is Difficulty Hiking, which systematically increases problem complexity and improves downstream reasoning capabilities, evidenced by significant gains when used for augmentation on strong baselines. The pipeline incorporates multiple filtering stages (self-consistency, de-duplication, decontamination, difficulty rating, novelty) and demonstrates competitive standalone performance as well as data-efficient improvements in augmentation settings on AIME/AMC/MATH500 benchmarks. The work highlights practical scalability and provides a foundation for generating reasoning-focused datasets across scientific domains, with future gains anticipated from larger teacher models and expanded post-training.

Abstract

The demand for Large Language Models (LLMs) at multiple scales, capable of sophisticated and sound mathematical reasoning, continues to grow. However, the development of performant mathematical LLMs is often bottlenecked by the scarcity of useful training data containing problems with significant complexity. We introduce \textbf{SAND-Math} (\textbf{S}ynthetic \textbf{A}ugmented \textbf{N}ovel and \textbf{D}ifficult Mathematics problems and solutions), a pipeline that addresses this by first synthesizing high-quality problems from scratch and then systematically elevating their complexity via a our newly proposed \textbf{Difficulty Hiking} step. We demonstrate the effectiveness of our approach through two key findings: \textbf{(1)} Augmenting a strong post-training baseline with a small 500-sample SAND-Math dataset significantly boosts performance, outperforming the next-best synthetic dataset by $\uparrow$ 17.85 absolute points on AIME25 benchmark. \textbf{(2)} In a dedicated ablation study, we show the effectiveness of our Difficulty Hiking process in increasing average problem difficulty from 5.02 to 5.98. This step consequently lifts AIME25 results from 46.38\% to 49.23\%. The full generation pipeline, final dataset, and a fine-tuned model form a practical and scalable toolkit for building capable and efficient mathematical reasoning LLMs.

SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answers

TL;DR

SAND-Math addresses the bottleneck of scarce high-quality math training data for LLMs by introducing a fully automated data-generation pipeline that synthesizes novel, challenging problems and solutions. A key innovation is Difficulty Hiking, which systematically increases problem complexity and improves downstream reasoning capabilities, evidenced by significant gains when used for augmentation on strong baselines. The pipeline incorporates multiple filtering stages (self-consistency, de-duplication, decontamination, difficulty rating, novelty) and demonstrates competitive standalone performance as well as data-efficient improvements in augmentation settings on AIME/AMC/MATH500 benchmarks. The work highlights practical scalability and provides a foundation for generating reasoning-focused datasets across scientific domains, with future gains anticipated from larger teacher models and expanded post-training.

Abstract

The demand for Large Language Models (LLMs) at multiple scales, capable of sophisticated and sound mathematical reasoning, continues to grow. However, the development of performant mathematical LLMs is often bottlenecked by the scarcity of useful training data containing problems with significant complexity. We introduce \textbf{SAND-Math} (\textbf{S}ynthetic \textbf{A}ugmented \textbf{N}ovel and \textbf{D}ifficult Mathematics problems and solutions), a pipeline that addresses this by first synthesizing high-quality problems from scratch and then systematically elevating their complexity via a our newly proposed \textbf{Difficulty Hiking} step. We demonstrate the effectiveness of our approach through two key findings: \textbf{(1)} Augmenting a strong post-training baseline with a small 500-sample SAND-Math dataset significantly boosts performance, outperforming the next-best synthetic dataset by 17.85 absolute points on AIME25 benchmark. \textbf{(2)} In a dedicated ablation study, we show the effectiveness of our Difficulty Hiking process in increasing average problem difficulty from 5.02 to 5.98. This step consequently lifts AIME25 results from 46.38\% to 49.23\%. The full generation pipeline, final dataset, and a fine-tuned model form a practical and scalable toolkit for building capable and efficient mathematical reasoning LLMs.

Paper Structure

This paper contains 34 sections, 3 equations, 4 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. SAND-Math Pipeline
  3. Question and Solution Generation
  4. Solution Correctness Filtering
  5. De-duplication and Decontamination
  6. Difficulty Filtering and Rating
  7. Novelty Filtering
  8. Difficulty Hiking: Systematically Increasing Problem Complexity
  9. Experiments and Results
  10. Implementation Details
  11. Experimental Design
  12. Results
  13. Conclusion
  14. Pipeline Yield and Teacher Model Ablation
  15. Detailed Analysis of Difficulty Hiking
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Data Generation and Filtering pipeline for SAND-Math.
  • Figure 2: Difficulty distribution of SAND-Math (500) dataset compared with other math datasets
  • Figure 3: Impact of Difficulty Hiking on Data Distribution. Comparison of difficulty ratings for a SAND-Math sample across three variants: the original Base (non-difficulty hiked) data, the DH (Difficulty Hiked) version, and the DH_w_LF version (DH with a 32k length filter). The hiking process successfully shifts the questions difficulty distribution more towards (6 to 8) range.
  • Figure 4: Performance trend when augmenting the LIMO training data with SAND-Math (SM) samples. The 'DH' (Difficulty Hiked) condition shows greater improvement with 1500 additional samples.