Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Motif 2 12.7B technical report

Junghwan Lim, Sungmin Lee, Dongseok Kim, Taehyun Kim, Eunhwan Park, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, Wai Ting Cheung, Dahye Choi, Jaeheui Her, Jaeyeon Huh, Hanbin Jung, Changjin Kang, Beomgyu Kim, Minjae Kim, Taewhan Kim, Youngrok Kim, Hyukjin Kweon, Haesol Lee, Kungyu Lee, Dongpin Oh, Yeongjae Park, Bokki Ryu, Dongjoo Weon

TL;DR

Motif-2-12.7B introduces an open-weight foundation model that achieves strong instruction-following and reasoning with a 12.7B parameter footprint by integrating Grouped Differential Attention and a suite of systems-level optimizations. The approach couples width-preserving hypercloning with GDA, depth expansion, curriculum-aware pre-training on $5.5\times10^{12}$ tokens, and a three-stage supervised fine-tuning pipeline, complemented by high-throughput training via Parallel Muon and fused PolyNorm kernels. Across extensive benchmarks, Motif-2-12.7B-Base and -Instruct demonstrate competitive performance against larger open-weight models, highlighting data efficiency and architectural cleverness as viable routes to high capability under resource constraints. The work emphasizes openness and reproducibility, and outlines a future reinforced reasoning variant to further enhance multi-step problem solving while preserving fluency.

Abstract

We introduce Motif-2-12.7B, a new open-weight foundation model that pushes the efficiency frontier of large language models by combining architectural innovation with system-level optimization. Designed for scalable language understanding and robust instruction generalization under constrained compute budgets, Motif-2-12.7B builds upon Motif-2.6B with the integration of Grouped Differential Attention (GDA), which improves representational efficiency by disentangling signal and noise-control attention pathways. The model is pre-trained on 5.5 trillion tokens spanning diverse linguistic, mathematical, scientific, and programming domains using a curriculum-driven data scheduler that gradually changes the data composition ratio. The training system leverages the MuonClip optimizer alongside custom high-performance kernels, including fused PolyNorm activations and the Parallel Muon algorithm, yielding significant throughput and memory efficiency gains in large-scale distributed environments. Post-training employs a three-stage supervised fine-tuning pipeline that successively enhances general instruction adherence, compositional understanding, and linguistic precision. Motif-2-12.7B demonstrates competitive performance across diverse benchmarks, showing that thoughtful architectural scaling and optimized training design can rival the capabilities of much larger models.

Motif 2 12.7B technical report

TL;DR

Motif-2-12.7B introduces an open-weight foundation model that achieves strong instruction-following and reasoning with a 12.7B parameter footprint by integrating Grouped Differential Attention and a suite of systems-level optimizations. The approach couples width-preserving hypercloning with GDA, depth expansion, curriculum-aware pre-training on tokens, and a three-stage supervised fine-tuning pipeline, complemented by high-throughput training via Parallel Muon and fused PolyNorm kernels. Across extensive benchmarks, Motif-2-12.7B-Base and -Instruct demonstrate competitive performance against larger open-weight models, highlighting data efficiency and architectural cleverness as viable routes to high capability under resource constraints. The work emphasizes openness and reproducibility, and outlines a future reinforced reasoning variant to further enhance multi-step problem solving while preserving fluency.

Abstract

We introduce Motif-2-12.7B, a new open-weight foundation model that pushes the efficiency frontier of large language models by combining architectural innovation with system-level optimization. Designed for scalable language understanding and robust instruction generalization under constrained compute budgets, Motif-2-12.7B builds upon Motif-2.6B with the integration of Grouped Differential Attention (GDA), which improves representational efficiency by disentangling signal and noise-control attention pathways. The model is pre-trained on 5.5 trillion tokens spanning diverse linguistic, mathematical, scientific, and programming domains using a curriculum-driven data scheduler that gradually changes the data composition ratio. The training system leverages the MuonClip optimizer alongside custom high-performance kernels, including fused PolyNorm activations and the Parallel Muon algorithm, yielding significant throughput and memory efficiency gains in large-scale distributed environments. Post-training employs a three-stage supervised fine-tuning pipeline that successively enhances general instruction adherence, compositional understanding, and linguistic precision. Motif-2-12.7B demonstrates competitive performance across diverse benchmarks, showing that thoughtful architectural scaling and optimized training design can rival the capabilities of much larger models.

Paper Structure

This paper contains 28 sections, 2 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Architecture
  3. Pre-training
  4. Pre-training Data
  5. Pre-training Stage and Recipe
  6. Dataset Mixture Scheduling
  7. Large-Batch Training
  8. Reasoning Annealing Stage
  9. Pre-training Evaluation
  10. Evaluation Setup
  11. Evaluation Results
  12. Training System Optimization
  13. Infrastructure and Precision
  14. Kernel Fusion
  15. Parallel Muon
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Illustration of the All-to-All gather and scatter process in Parallel Muon. Each rank exchanges its sharded gradients with all other ranks during the gather phase and redistributes the computed results in the scatter phase.
  • Figure 2: Illustration of pipelined execution of all-to-all gather, computation, and scatter phases. A total of eight gradients are distributed across two ranks and partitioned into chunks of size two. Computation and communication are scheduled in a pipelined manner, allowing overlap between them and improving overall efficiency.