Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

xGen-small Technical Report

Erik Nijkamp, Bo Pang, Egor Pakhomov, Akash Gokul, Jin Qu, Silvio Savarese, Yingbo Zhou, Caiming Xiong

TL;DR

xGen-small presents a pair of compact decoder transformers (4B and 9B) engineered for true long-context processing up to 128K tokens. The approach combines domain-balanced, frequency-aware data curation, distributional sharpening during pre-training, and a two-stage length-extension workflow with Rotary Position Embeddings, followed by a three-phase post-training pipeline (SFT, DPO, GRPO) to achieve strong reasoning, math, and coding capabilities at reduced deployment costs. Empirical results show competitive or superior performance to similarly sized open models across standard benchmarks (MMLU, GSM8K, HumanEval) and long-context assessments (RULER), with notable gains in math and coding after post-training. These findings suggest that compact models, when backed by carefully engineered data pipelines and alignment procedures, can deliver state-of-the-art results for enterprise settings where latency, cost, and privacy are critical.

Abstract

We introduce xGen-small, a family of 4B and 9B Transformer decoder models optimized for long-context applications. Our vertically integrated pipeline unites domain-balanced, frequency-aware data curation; multi-stage pre-training with quality annealing and length extension to 128k tokens; and targeted post-training via supervised fine-tuning, preference learning, and online reinforcement learning. xGen-small delivers strong performance across various tasks, especially in math and coding domains, while excelling at long context benchmarks.

xGen-small Technical Report

TL;DR

xGen-small presents a pair of compact decoder transformers (4B and 9B) engineered for true long-context processing up to 128K tokens. The approach combines domain-balanced, frequency-aware data curation, distributional sharpening during pre-training, and a two-stage length-extension workflow with Rotary Position Embeddings, followed by a three-phase post-training pipeline (SFT, DPO, GRPO) to achieve strong reasoning, math, and coding capabilities at reduced deployment costs. Empirical results show competitive or superior performance to similarly sized open models across standard benchmarks (MMLU, GSM8K, HumanEval) and long-context assessments (RULER), with notable gains in math and coding after post-training. These findings suggest that compact models, when backed by carefully engineered data pipelines and alignment procedures, can deliver state-of-the-art results for enterprise settings where latency, cost, and privacy are critical.

Abstract

We introduce xGen-small, a family of 4B and 9B Transformer decoder models optimized for long-context applications. Our vertically integrated pipeline unites domain-balanced, frequency-aware data curation; multi-stage pre-training with quality annealing and length extension to 128k tokens; and targeted post-training via supervised fine-tuning, preference learning, and online reinforcement learning. xGen-small delivers strong performance across various tasks, especially in math and coding domains, while excelling at long context benchmarks.
Paper Structure (22 sections, 2 figures, 6 tables)

This paper contains 22 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Data-Curation
  3. Acquisition
  4. Deduplication
  5. Quality Filtering
  6. Domain Enrichment
  7. Curriculum Schedule
  8. Pre-Training
  9. Model
  10. Framework
  11. Training
  12. Length Extension
  13. Post-Training
  14. Supervised Fine-Tuning
  15. Off-Policy Preference Training
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Summary of xGen model performance. We introduce the xGen-small series of language models which are available in two sizes -- xGen-small-4B and xGen-small-9B. These open-source models provide strong performance across a variety of tasks and scale up to 128K context length.
  • Figure 2: Detailed RULER scores for 4k-128k context length. We compare RULER scores for models in the 3B to 4B range (left) and the 7B to 9B range (right).