Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instella: Fully Open Language Models with Stellar Performance

Jiang Liu, Jialian Wu, Xiaodong Yu, Yusheng Su, Prakamya Mishra, Gowtham Ramesh, Sudhanshu Ranjan, Chaitanya Manem, Ximeng Sun, Ze Wang, Pratik Prabhanjan Brahma, Zicheng Liu, Emad Barsoum

TL;DR

Instella tackles the transparency gap in language modeling by delivering a fully open 3B LLM family trained with openly available data and pipelines. It employs a two-stage pre-training regime, an in-house synthetic mathematics dataset, weight ensembling, then supervised fine-tuning and direct preference optimization to align with human preferences. The suite includes Instella-Long for 128K context and Instella-Math for long-horizon reasoning via GRPO reinforcement learning, achieving state-of-the-art results among fully open models and competitive performance with open-weight rivals. By releasing models, training code, data recipes, and evaluation protocols, Instella enables reproducible benchmarking and broad community participation in open, scalable language-model research.

Abstract

Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks, yet the majority of high-performing models remain closed-source or partially open, limiting transparency and reproducibility. In this work, we introduce Instella, a family of fully open three billion parameter language models trained entirely on openly available data and codebase. Powered by AMD Instinct MI300X GPUs, Instella is developed through large-scale pre-training, general-purpose instruction tuning, and alignment with human preferences. Despite using substantially fewer pre-training tokens than many contemporaries, Instella achieves state-of-the-art results among fully open models and is competitive with leading open-weight models of comparable size. We further release two specialized variants: Instella-Long, capable of handling context lengths up to 128K tokens, and Instella-Math, a reasoning-focused model enhanced through supervised fine-tuning and reinforcement learning on mathematical tasks. Together, these contributions establish Instella as a transparent, performant, and versatile alternative for the community, advancing the goal of open and reproducible language modeling research.

Instella: Fully Open Language Models with Stellar Performance

TL;DR

Instella tackles the transparency gap in language modeling by delivering a fully open 3B LLM family trained with openly available data and pipelines. It employs a two-stage pre-training regime, an in-house synthetic mathematics dataset, weight ensembling, then supervised fine-tuning and direct preference optimization to align with human preferences. The suite includes Instella-Long for 128K context and Instella-Math for long-horizon reasoning via GRPO reinforcement learning, achieving state-of-the-art results among fully open models and competitive performance with open-weight rivals. By releasing models, training code, data recipes, and evaluation protocols, Instella enables reproducible benchmarking and broad community participation in open, scalable language-model research.

Abstract

Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks, yet the majority of high-performing models remain closed-source or partially open, limiting transparency and reproducibility. In this work, we introduce Instella, a family of fully open three billion parameter language models trained entirely on openly available data and codebase. Powered by AMD Instinct MI300X GPUs, Instella is developed through large-scale pre-training, general-purpose instruction tuning, and alignment with human preferences. Despite using substantially fewer pre-training tokens than many contemporaries, Instella achieves state-of-the-art results among fully open models and is competitive with leading open-weight models of comparable size. We further release two specialized variants: Instella-Long, capable of handling context lengths up to 128K tokens, and Instella-Math, a reasoning-focused model enhanced through supervised fine-tuning and reinforcement learning on mathematical tasks. Together, these contributions establish Instella as a transparent, performant, and versatile alternative for the community, advancing the goal of open and reproducible language modeling research.

Paper Structure

This paper contains 23 sections, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Open-Weight versus Fully-Open Large Language Models
  4. Long-context Language Models
  5. Large Reasoning Models
  6. Instella
  7. Model Architecture
  8. Training Setup
  9. Pre-training
  10. Post-training
  11. Instella-Long
  12. Continued Pre-training
  13. Post-training
  14. Implementation Details
  15. Instella-Math
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Average Score versus Pre-training Tokens for base (left) and instruction-tuned (right) models. Instella surpasses prior fully open models of comparable size and, despite being trained on substantially fewer pre-training tokens, achieves competitive performance with state-of-the-art open-weight models for both (left) base models (Table \ref{['tab:pretrain']}) and (right) instruction-tuned models (Table \ref{['tab:posttrain']}).
  • Figure 2: Instella-3B model training pipeline.
  • Figure 3: Instella-Long model training pipeline.
  • Figure 4: Instella-Math model training pipeline.