Xmodel-2 Technical Report

TL;DR

Xmodel-2 introduces a 1.2B-parameter reasoning-focused architecture that leverages cross-scale hyperparameter sharing via μP, a Warmup-Stable-Decay scheduler, and strategic data-ratio optimization to boost complex reasoning and agent performance at low cost. The two-stage pretraining regime, diverse data mixtures, and SFT data emphasis in the decay phase yield strong performance across commonsense, complex reasoning, and agent tasks, with calibrated predictions and a post-training scaling law linking context length to perplexity. The work also provides open-source code and wind-tunnel style validation, highlighting practical impact for research and deployment in interactive and automated settings.

Abstract

Xmodel-2 is a 1.2-billion-parameter large language model designed specifically for reasoning tasks. Its architecture enables different model scales to share a unified set of hyperparameters, allowing for extensive experimentation on smaller models and seamless transfer of optimal configurations to larger models. To maximize training efficiency and stability, Xmodel-2 employs the WSD learning rate scheduler from MiniCPM. Pretrained on 1.5 trillion tokens from diverse sources, Xmodel-2 achieves state-of-the-art performance in complex reasoning and agent-based tasks, while maintaining low training costs. These results highlight the potential of efficient model design and training strategies in advancing reasoning capabilities. Model checkpoints and code are publicly available on GitHub at https://github.com/XiaoduoAILab/Xmodel-2

Figures (5)

• Figure 2: Data mixture of different training stages.The left side represents the stable training phase, and the right side represents the decay phase.
• Figure 3: Loss curve for Xmodel-2-1.2B.
• Figure 4: Calibration plot for the pre-trained Xmodel-2-1.2B model on the MMLU dataset.
• Figure 5: Post-training Scaling Law for Xmodel-2-1.2B on the Wikitext-2 dataset.
• Figure 6: Grid search over the $\mu$P parameterization spaces.