OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling

TL;DR

This work probes why base language models diverge under reinforcement learning aimed at reasoning, highlighting that high-quality, reasoning-focused pre-training data is crucial for RL success. The authors develop a two-stage mid-training regimen, Stable-then-Decay, to make Llama-based models more RL-friendly, culminating in the OctoThinker family that narrows the gap with Qwen on mathematical benchmarks. They show that data type, QA format, and reward-aligned instruction data jointly shape RL dynamics, and that longer chain-of-thought data can destabilize training without careful prompt design and length scheduling. The study provides open-source models and a million-scale math corpus (MegaMath-Web-Pro-Max) to enable broader exploration of RL-aligned pre-training strategies.

Abstract

Different base language model families, such as Llama and Qwen, exhibit divergent behaviors during post-training with reinforcement learning (RL), especially on reasoning-intensive tasks. What makes a base language model suitable for reinforcement learning? Gaining deeper insight into this question is essential for developing RL-scalable foundation models of the next generation. In this work, we investigate how mid-training strategies shape RL dynamics, focusing on two representative model families: Qwen and Llama. Our study reveals that (1) high-quality mathematical corpora, such as MegaMath-Web-Pro, significantly improve both base model and RL performance, while existing alternatives (e.g., FineMath-4plus) fail to do so; (2) further adding QA-style data, particularly long chain-of-thought (CoT) reasoning examples, enhances RL outcomes, and instruction data further unlocks this effect; (3) while long-CoT improves reasoning depth, it can also induce verbosity of model responses and unstability of RL training, underscoring the importance of data formatting; (4) scaling mid-training consistently leads to stronger downstream RL performance. Building on these insights, we introduce a two-stage mid-training strategy, Stable-then-Decay, in which base models are first trained on 200B tokens with a constant learning rate, followed by 20B tokens across three CoT-focused branches with learning rate decay. This yields OctoThinker, a family of models demonstrating strong RL compatibility and closing the performance gap with more RL-friendly model families, i.e., Qwen. We hope our work will help shape pre-training strategies for foundation models in the RL era. To support further research, we release our open-source models along with a curated math reasoning-intensive corpus of over 70 billion tokens (i.e., MegaMath-Web-Pro-Max).

Figures (17)

• Figure 1: Our strategic mid-training incentivizes Llama's RL scaling, matching Qwen2.5 performance.
• Figure 2: Training dynamics comparison (downstream performance and the average length of correct responses) between Llama-3.2-3B and Qwen2.5-3B. The dashed line indicates the few-shot evaluation performance and average length of correct responses of the corresponding base models.
• Figure 3: Potential factors in mid-training that could impact the post-training stage.
• Figure 4: Comparison between our fasttext-recalled corpus (w/o LLM refinement) and MegaMath-Web, following its yearly dump comparison setup under a 5B-token pre-training budget. Recall thresholds shown accordingly.
• Figure 5: The effect of different math web corpora during mid-training. We performed mid-training on each corpus with a 20B-token training budget.