Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Models

Chen Yang, Guangyue Peng, Jiaying Zhu, Ran Le, Ruixiang Feng, Tao Zhang, Wei Ruan, Xiaoqi Liu, Xiaoxue Cheng, Xiyun Xu, Yang Song, Yanzipeng Gao, Yiming Jia, Yun Xing, Yuntao Wen, Zekai Wang, Zhenwei An, Zhicong Sun, Zongchao Chen

TL;DR

Nanbeige4-3B demonstrates that compact language models can achieve competitive performance with a carefully designed data and training pipeline. The approach combines a 23T high-quality pre-training corpus with Hybrid Data Filtering, FG-WSD data scheduling, and post-training refinements including deliberative generation, CoT reconstruction, and Dual Preference Distillation, followed by multi-stage RL. Across diverse reasoning, coding, and alignment benchmarks, Nanbeige4-3B-Base and Nanbeige4-3B-Thinking outperform larger Qwen models and set state-of-the-art results on tasks like AIME and GPQA, while maintaining strong tool-use and writing capabilities. The work provides an integrated, open-source methodology for building strong small-scale models that close the gap with larger architectures at lower resource costs.

Abstract

We present Nanbeige4-3B, a family of small-scale but high-performing language models. Pretrained on 23T high-quality tokens and finetuned on over 30 million diverse instructions, we extend the boundary of the scaling law for small language models. In pre-training, we design a Fine-Grained Warmup-Stable-Decay (FG-WSD) training scheduler, which progressively refines data mixtures across stages to boost model performance. In post-training, to improve the quality of the SFT data, we design a joint mechanism that integrates deliberative generation refinement and chain-of-thought reconstruction, yielding substantial gains on complex tasks. Following SFT, we employ our flagship reasoning model to distill Nanbeige4-3B through our proposed Dual Preference Distillation (DPD) method, which leads to further performance gains. Finally, a multi-stage reinforcement learning phase was applied, leveraging verifiable rewards and preference modeling to strengthen abilities on both reasoning and human alignment. Extensive evaluations show that Nanbeige4-3B not only significantly outperforms models of comparable parameter scale but also rivals much larger models across a wide range of benchmarks. The model checkpoints are available at https://huggingface.co/Nanbeige.

Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Models

TL;DR

Nanbeige4-3B demonstrates that compact language models can achieve competitive performance with a carefully designed data and training pipeline. The approach combines a 23T high-quality pre-training corpus with Hybrid Data Filtering, FG-WSD data scheduling, and post-training refinements including deliberative generation, CoT reconstruction, and Dual Preference Distillation, followed by multi-stage RL. Across diverse reasoning, coding, and alignment benchmarks, Nanbeige4-3B-Base and Nanbeige4-3B-Thinking outperform larger Qwen models and set state-of-the-art results on tasks like AIME and GPQA, while maintaining strong tool-use and writing capabilities. The work provides an integrated, open-source methodology for building strong small-scale models that close the gap with larger architectures at lower resource costs.

Abstract

We present Nanbeige4-3B, a family of small-scale but high-performing language models. Pretrained on 23T high-quality tokens and finetuned on over 30 million diverse instructions, we extend the boundary of the scaling law for small language models. In pre-training, we design a Fine-Grained Warmup-Stable-Decay (FG-WSD) training scheduler, which progressively refines data mixtures across stages to boost model performance. In post-training, to improve the quality of the SFT data, we design a joint mechanism that integrates deliberative generation refinement and chain-of-thought reconstruction, yielding substantial gains on complex tasks. Following SFT, we employ our flagship reasoning model to distill Nanbeige4-3B through our proposed Dual Preference Distillation (DPD) method, which leads to further performance gains. Finally, a multi-stage reinforcement learning phase was applied, leveraging verifiable rewards and preference modeling to strengthen abilities on both reasoning and human alignment. Extensive evaluations show that Nanbeige4-3B not only significantly outperforms models of comparable parameter scale but also rivals much larger models across a wide range of benchmarks. The model checkpoints are available at https://huggingface.co/Nanbeige.

Paper Structure

This paper contains 17 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Pre-Training
  3. Pre-Training Data
  4. Pre-Training Recipe
  5. Data Quality Identify
  6. Data Utility Scheduler
  7. Post-SFT Evaluation for Base Model
  8. Post-Training
  9. Cold Start Supervised Fine-tuning
  10. Overall Supervised Fine-Tuning
  11. Dual-level Preference Distillation
  12. Reinforcement Learning
  13. On-Policy Data Filtering
  14. Multi-Stage Reinforcement Learning
  15. Post-Training Evaluation
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Performance Comparison between Nanbeige4-3B-Thinking and Qwen series models.
  • Figure 2: Nanbeige4-3B-Thinking Post-Training Pipeline
  • Figure 3: Cold-start SFT scaling (0.5M–35M) on NBG4-3B reasoning benchmarks.
  • Figure 4: Deliberative generation refinement and CoT completion
  • Figure 5: Overview of the Dual-Level Preference Distillation (DPD) framework