Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch

TL;DR

ScaleQuest addresses the shortage of open-source, high-quality mathematical reasoning data by enabling lightweight 7B-scale models to generate 1M diverse problem-solution pairs from scratch. The authors introduce a two-stage question-tuning pipeline (QFT and QPO) and a multi-faceted filtering and reward framework to produce high-quality questions and solutions at low cost. Empirical results show ScaleQuest outperforms existing open-source data synthesis baselines on in-domain and out-of-domain benchmarks and scales favorably with more data, with notable gains in code and long-chain-of-thought tasks. The work provides a practical, open-source path to improve mathematical reasoning in LLMs, while acknowledging limitations on larger models and room for further improvement in question preference alignment.

Abstract

Improving the mathematical reasoning capabilities of Large Language Models (LLMs) is critical for advancing artificial intelligence. However, access to extensive, diverse, and high-quality reasoning datasets remains a significant challenge, particularly for the open-source community. In this paper, we propose ScaleQuest, a novel, scalable, and cost-effective data synthesis method that enables the generation of large-scale mathematical reasoning datasets using lightweight 7B-scale models. ScaleQuest introduces a two-stage question-tuning process comprising Question Fine-Tuning (QFT) and Question Preference Optimization (QPO) to unlock the question generation capabilities of problem-solving models. By generating diverse questions from scratch -- without relying on powerful proprietary models or seed data -- we produce a dataset of 1 million problem-solution pairs. Our experiments demonstrate that models trained on our data outperform existing open-source datasets in both in-domain and out-of-domain evaluations. Furthermore, our approach shows continued performance improvement as the volume of training data increases, highlighting its potential for ongoing data scaling. The extensive improvements observed in code reasoning tasks demonstrate the generalization capabilities of our proposed method. Our work provides the open-source community with a practical solution to enhance the mathematical reasoning abilities of LLMs.

Figures (19)

• Figure 1: Results of Llama3-8B-Base fine-tuned on publicly available datasets in MATH and OlympiadBench. Our approach demonstrates strong scalability and significant potential for further improvement.
• Figure 2: Overview of our ScaleQuest method.
• Figure 3: The difficulty distribution of two real-world datasets and two synthetic datasets. The difficulty score is calculated based solely on the problem part.
• Figure 4: The solvability and difficulty of the raw questions generated by the QFT model and the optimized ones. We use GPT-4o as a manual substitute to evaluate the effectiveness of solvability and difficulty optimization, with evaluation prompt in Figure \ref{['fig:prompt_solvability_check']} and \ref{['fig:prompt_difficulty_check']}.
• Figure 5: A comparison of the synthetic dataset generated by the raw instruct model, the model after QFT, the model after QPO, and the final dataset after applying reward filtering. Left: The solvable ratio and difficulty score of the generated questions. The solvable ratio refers to the proportion of generated questions that are judged as "solvable", while the difficulty score represents the average difficulty rating assigned to each generated question. For difficulty evaluation, we calculate the dataset’s average difficulty score based on ratings for each question: "very easy" is rated as 20 points, "easy" as 40 points, "medium" as 60 points, "hard" as 80 points, and "very hard" as 100 points. Right: The instruction tuning effectiveness on Llama3-8B-Base.