Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM

Yang Liu, Xiaolong Zhong, Ling Jiang

TL;DR

This work addresses the need for data-efficient, edge-friendly reasoning in small language models by introducing Xmodel-2.5, a 1.3B decoder-only model built with maximal-update parameterisation ($\mu$P). It combines a three-phase Warmup--Stable--Decay curriculum, FP8 mixed-precision training, and a late-stage optimizer switch (AdamW to Muon) to boost multi-step reasoning while using far fewer training tokens ($\approx 1.4$T) than larger counterparts. Despite being smaller and trained with drastically less data, Xmodel-2.5 achieves 52.49% average on 13 reasoning benchmarks, closing 71% of the gap to a 1--2B leader and only 4.47pp behind Qwen-3, demonstrating strong data-efficiency and effective long-context adaptation up to 16k tokens. The results indicate that carefully paced training and lightweight context expansion can yield robust, edge-deployable reasoning agents, and the work provides reproducible recipes and resources for the community.

Abstract

Large language models deliver strong reasoning and tool-use skills, yet their computational demands make them impractical for edge or cost-sensitive deployments. We present \textbf{Xmodel-2.5}, a 1.3-billion-parameter small language model designed as a \emph{drop-in agent core}. Training with maximal-update parameterization ($μ$P) allows hyper-parameters tuned on a 20M-parameter proxy to transfer directly to the full model, even under the parameter-tied \emph{tie-word-embedding} architecture. A 1.4T-token Warmup--Stable--Decay curriculum is used, and we further show that \textbf{switching from AdamW to Muon during the decay phase} improves the 13-task reasoning average by 4.58\,\% while keeping every other hyper-parameter fixed, verifying that early AdamW stability can be paired with late Muon sharpening for better downstream performance. FP8-mixed-precision training balances accuracy and throughput. All checkpoints, recipes, and evaluation code are released under the Apache-2.0 license.\footnote{https://huggingface.co/XiaoduoAILab/Xmodel-2.5 and https://huggingface.co/XiaoduoAILab/Xmodel-2.5-history (training checkpoints).} Training code and evaluation harness: https://github.com/XiaoduoAILab/Xmodel-2.5.

Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM

TL;DR

This work addresses the need for data-efficient, edge-friendly reasoning in small language models by introducing Xmodel-2.5, a 1.3B decoder-only model built with maximal-update parameterisation (P). It combines a three-phase Warmup--Stable--Decay curriculum, FP8 mixed-precision training, and a late-stage optimizer switch (AdamW to Muon) to boost multi-step reasoning while using far fewer training tokens (T) than larger counterparts. Despite being smaller and trained with drastically less data, Xmodel-2.5 achieves 52.49% average on 13 reasoning benchmarks, closing 71% of the gap to a 1--2B leader and only 4.47pp behind Qwen-3, demonstrating strong data-efficiency and effective long-context adaptation up to 16k tokens. The results indicate that carefully paced training and lightweight context expansion can yield robust, edge-deployable reasoning agents, and the work provides reproducible recipes and resources for the community.

Abstract

Large language models deliver strong reasoning and tool-use skills, yet their computational demands make them impractical for edge or cost-sensitive deployments. We present \textbf{Xmodel-2.5}, a 1.3-billion-parameter small language model designed as a \emph{drop-in agent core}. Training with maximal-update parameterization (P) allows hyper-parameters tuned on a 20M-parameter proxy to transfer directly to the full model, even under the parameter-tied \emph{tie-word-embedding} architecture. A 1.4T-token Warmup--Stable--Decay curriculum is used, and we further show that \textbf{switching from AdamW to Muon during the decay phase} improves the 13-task reasoning average by 4.58\,\% while keeping every other hyper-parameter fixed, verifying that early AdamW stability can be paired with late Muon sharpening for better downstream performance. FP8-mixed-precision training balances accuracy and throughput. All checkpoints, recipes, and evaluation code are released under the Apache-2.0 license.\footnote{https://huggingface.co/XiaoduoAILab/Xmodel-2.5 and https://huggingface.co/XiaoduoAILab/Xmodel-2.5-history (training checkpoints).} Training code and evaluation harness: https://github.com/XiaoduoAILab/Xmodel-2.5.

Paper Structure

This paper contains 22 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Xmodel-2.5
  3. Background & Related Work
  4. Small Language Models for Reasoning
  5. Hyper-Parameter Transfer with Maximal-Update Parameterisation
  6. Efficient Training of Lightweight Models
  7. Methodology
  8. Model Architecture
  9. Three-Phase WSD Pre-Training
  10. Optimizer ablation: AdamW $\rightarrow$ Muon.
  11. FP8 Hybrid Format & Kernel Implementation
  12. Experimental Settings
  13. Baselines
  14. Tasks and Metrics
  15. Commonsense
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Data composition in the (a) Stable and (b) Decay phases of WSD LR scheduling. The Stable phase emphasizes broad pre-training data diversity, while the Decay phase focuses on high-quality instructional and SFT data to refine model capabilities.
  • Figure 2: Loss curve for Xmodel-2.5 1.3B.