Dual-Phase LLM Reasoning: Self-Evolved Mathematical Frameworks

TL;DR

This paper addresses the challenge of improving mathematical reasoning in large language models without heavy reliance on RL by introducing a two-stage supervised fine-tuning framework built on self-generated long chain-of-thought data. The first stage creates $D_{ ext{multi}}$ via a multi-turn reasoning strategy, filtering high-quality samples to obtain $oldsymbol{Cpi_{sft}}$; the second stage applies difficulty-aware rejection sampling to produce $D_{ ext{rej}}$, which is merged into $D_{ ext{multi+rej}}$ for further fine-tuning to $oldsymbol{Cpi_{sft+rej}}$. The method explicitly encodes four reasoning capabilities—verification, backtracking, subgoal construction, and backward chaining—into the data, enabling scalable improvements on benchmarks like GSM8K, MATH500, and AIME24 with reduced computational cost compared to distillation from larger models. Experimental results demonstrate consistent gains across diverse mathematical tasks and show how marker-based data merging can tailor the model toward interactive or symbolic reasoning modalities. The work offers a practical pathway for enhancing complex reasoning in LLMs through self-supervised data generation and targeted sampling strategies, with open-sourced code anticipated.

Abstract

In recent years, large language models (LLMs) have demonstrated significant potential in complex reasoning tasks like mathematical problem-solving. However, existing research predominantly relies on reinforcement learning (RL) frameworks while overlooking supervised fine-tuning (SFT) methods. This paper proposes a new two-stage training framework that enhances models' self-correction capabilities through self-generated long chain-of-thought (CoT) data. During the first stage, a multi-turn dialogue strategy guides the model to generate CoT data incorporating verification, backtracking, subgoal decomposition, and backward reasoning, with predefined rules filtering high-quality samples for supervised fine-tuning. The second stage employs a difficulty-aware rejection sampling mechanism to dynamically optimize data distribution, strengthening the model's ability to handle complex problems. The approach generates reasoning chains extended over 4 times longer while maintaining strong scalability, proving that SFT effectively activates models' intrinsic reasoning capabilities and provides a resource-efficient pathway for complex task optimization. Experimental results demonstrate performance improvements on mathematical benchmarks including GSM8K and MATH500, with the fine-tuned model achieving a substantial improvement on competition-level problems like AIME24. Code will be open-sourced.

Figures (3)

• Figure 1: A two-stage self-improvement framework for enhancing LLMs' mathematical reasoning. Stage 1 generates high-quality CoT data via multi-turn reasoning and rule-based filtering for SFT ($\pi_\text{sft}$). Stage 2 employs difficulty-aware rejection sampling on $\pi_\text{sft}$'s outputs to refine reasoning on complex problems, yielding an optimized model $\pi_\text{rej}$
• Figure 2: Demonstration of multi-turn self-correction. The model first produces an incorrect answer "4" through suboptimal cancellation, then performs backtracking and strategic factorization to identify intermediate optimization opportunities. After recalculating, it verifies correctness and commits to the final refined answer "$\frac{4}{7}$", showcasing activation of intrinsic reasoning capabilities.
• Figure 3: Flow diagram for self-generating Long Chain-of-Thought data. Dual-step answers ($\text{Answer}_1$, $\text{Answer}_2$) are derived from Problem and Ground Truth. Evaluation produces Transition Text through answer analysis; Summary yields synthetic verification.