AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy

TL;DR

This work systematically investigates how supervised fine-tuning (SFT) data scaling and reinforcement learning (RL) synergy can elevate math and code reasoning in 7B-scale models. By building AceReason-Nemotron-1.1-7B on a strong SFT foundation and applying a multi-stage, verification-driven RL curriculum (math-focused stages followed by code-focused stages), the authors achieve state-of-the-art results among Qwen2.5-7B models on challenging benchmarks like AIME and LiveCodeBench. Key findings show that scaling SFT data, especially increasing the number of unique prompts, yields substantial gains, while RL progressively narrows gaps between starting SFT models and reaches high performance when sampling temperature is tuned to maintain modest entropy (around 0.3). The results underscore the importance of data quality, stage-wise training dynamics, and careful trade-offs in overlong filtering, culminating in a 7B model that sets new benchmarks for math and code reasoning and showcasing the practical viability of the post-training recipe. The work provides concrete guidelines for RL temperature, stage durations, and data scaling, and releases the model and data for broader use.

Abstract

In this work, we investigate the synergy between supervised fine-tuning (SFT) and reinforcement learning (RL) in developing strong reasoning models. We begin by curating the SFT training data through two scaling strategies: increasing the number of collected prompts and the number of generated responses per prompt. Both approaches yield notable improvements in reasoning performance, with scaling the number of prompts resulting in more substantial gains. We then explore the following questions regarding the synergy between SFT and RL: (i) Does a stronger SFT model consistently lead to better final performance after large-scale RL training? (ii) How can we determine an appropriate sampling temperature during RL training to effectively balance exploration and exploitation for a given SFT initialization? Our findings suggest that (i) holds true, provided effective RL training is conducted, particularly when the sampling temperature is carefully chosen to maintain the temperature-adjusted entropy around 0.3, a setting that strikes a good balance between exploration and exploitation. Notably, the performance gap between initial SFT models narrows significantly throughout the RL process. Leveraging a strong SFT foundation and insights into the synergistic interplay between SFT and RL, our AceReason-Nemotron-1.1 7B model significantly outperforms AceReason-Nemotron-1.0 and achieves new state-of-the-art performance among Qwen2.5-7B-based reasoning models on challenging math and code benchmarks, thereby demonstrating the effectiveness of our post-training recipe. We release the model and data at: https://huggingface.co/nvidia/AceReason-Nemotron-1.1-7B

Figures (17)

• Figure 1: Benchmark accuracy of AceReason-Nemotron-1.1-7B on AIME 2024/2025 (avg@64), HMMT 2025 (avg@64), LiveCodeBench v5 (2024/08/01-2025/02/01, avg@8), and v6 (2025/02/01-2025/05/01, avg@8) using 32768.0 output length.
• Figure 2: Training Pipeline of AceReason-Nemotron 1.1. We start by performing math and code SFT on a base pretrained model. Next, we conduct three stages of math-only RL training with progressively growing response length, i.e., Stage-1 (8K), Stage-2 (16K), and Stage-3 (24K), to develop a math-specialized RL model. We then apply code-only RL training to enhance model's coding capability. Lastly, we carry out a final stage of math-only RL to produce AceReason-Nemotron 1.1.
• Figure 3: Response token length distributions for the math SFT dataset (left) and the code SFT dataset (right).
• Figure 4: Log-scaled data statistics for the number of math and code prompts and the average number of responses per prompt. Each SFT dataset consist of both math and code SFT samples.
• Figure 5: Accuracies on AIME24, AIME25, and LiveCodeBench V5 and V6 for different SFT datasets. For each SFT blend, the model is trained until the accuracy plateaus.