Rethinking the Generation of High-Quality CoT Data from the Perspective of LLM-Adaptive Question Difficulty Grading

TL;DR

This work addresses the challenge of generating high-quality chain-of-thought data for small to mid-size LLMs by proposing an LLM-Adaptive Difficulty Grading framework. It builds a model-adaptive question library, samples from a difficulty distribution, and uses a powerful teacher model to generate CoT data that is verified for correctness, enabling efficient supervised fine-tuning. Empirical results show substantial gains in math and coding reasoning across multiple model scales, with ablations indicating that PRM-based grading, model-specific distributions, and evaluation-driven sampling are key drivers. The approach reduces data-generation costs while improving reasoning performance, and it points to future integrations with reinforcement learning or reject sampling to further enhance capabilities.

Abstract

Recently, DeepSeek-R1 (671B) (DeepSeek-AIet al., 2025) has demonstrated its excellent reasoning ability in complex tasks and has publiclyshared its methodology. This provides potentially high-quality chain-of-thought (CoT) data for stimulating the reasoning abilities of small-sized large language models (LLMs). To generate high-quality CoT data for different LLMs, we seek an efficient method for generating high-quality CoT data with LLM-Adaptive questiondifficulty levels. First, we grade the difficulty of the questions according to the reasoning ability of the LLMs themselves and construct a LLM-Adaptive question database. Second, we sample the problem database based on a distribution of difficulty levels of the questions and then use DeepSeek-R1 (671B) (DeepSeek-AI et al., 2025) to generate the corresponding high-quality CoT data with correct answers. Thanks to the construction of CoT data with LLM-Adaptive difficulty levels, we have significantly reduced the cost of data generation and enhanced the efficiency of model supervised fine-tuning (SFT). Finally, we have validated the effectiveness and generalizability of the proposed method in the fields of complex mathematical competitions and code generation tasks. Notably, with only 2k high-quality mathematical CoT data, our ZMath-32B surpasses DeepSeek-Distill-32B in math reasoning task. Similarly, with only 2k high-quality code CoT data, our ZCode-32B surpasses DeepSeek-Distill-32B in code reasoning tasks.

Figures (2)

• Figure 1: Constrcution of CoT Data with/without LLM-Adaptive question difficulty grading. For LLms of different parameters, the former consistently outperforms the latter in reasoning performance on the mathematical competition dataset AIME24 aime.
• Figure 2: The Framework for CoT Data Generation via LLM-Adaptive Question Difficulty Grading , comprising three core components: Distribution Construction, LLM-Adaptive Question Difficulty Grading & Distribution Sampling, and LLM-Adaptive Chain-of-Thought (CoT) Generation