Distilling Token-Trained Models into Byte-Level Models

TL;DR

The paper tackles the high cost of building byte-level language models by introducing a practical two-stage distillation pipeline that transfers knowledge from token-trained LLMs to hierarchical byte-level architectures. Stage 1 performs progressive knowledge distillation with embedding alignment, joint distillation, and boundary learning to align byte representations with token semantics; Stage 2 applies Byte-Level SFT with dechunking to enable end-to-end byte-space generation. Across Llama, Qwen, and OLMo families, the method achieves strong retention of teacher performance using roughly 125B training bytes, with significant data efficiency gains over prior baselines. Post-training techniques like on-policy distillation further boost reasoning capabilities, and extensive ablations validate the necessity of embedding alignment, the preferred decoding strategy, and the chosen encoder. The work provides practical tools, empirical insights, and open-source resources to advance efficient byte-level modeling for scalable language systems with reduced data and compute requirements.

Abstract

Byte Language Models (BLMs) have emerged as a promising direction for scaling language models beyond tokenization. However, existing BLMs typically require training from scratch on trillions of bytes, making them prohibitively expensive. In this paper, we propose an efficient distillation recipe that converts existing token-trained LLMs into BLMs while retaining comparable capabilities. Our recipe follows a two-stage curriculum: (1) Progressive Knowledge Distillation, which aligns byte-level representations with the embeddings of the token-trained teacher model; and (2) Byte-Level Supervised Fine-Tuning, which enables end-to-end generation entirely in the byte space. We validate our approach across multiple model families, including Llama, Qwen, and OLMo, and demonstrate that the distilled BLMs retain most of the teacher models' performance using only approximately 125B bytes.

Figures (3)

• Figure 1: Overview of the Two-Stage Distillation Framework. Stage 1: Progressive Knowledge Distillation includes: (a) Embedding Alignment to map byte patches to the token space ($L_{align}$); (b) Joint Distillation to synchronize hidden states and logits ($L_{distill}$); (c) Boundary Learning to align tokenization via the One-Byte Lookahead Router ($L_{boundary}$). Stage 2: Full Byte-Level SFT enables end-to-end generation in the byte space via Dechunk and Decoder modules.
• Figure 2: Training Dynamics across Stage 1. We track student performance in Stage 1. While simple tasks (e.g., PIQA) converge rapidly during Encoder Alignment (Step 1), knowledge-intensive tasks like MMLU exhibit a distinct phase transition, requiring the distribution matching of Joint Distillation (Step 2) to unlock performance gains.
• Figure 3: Visualization of Learned Token Boundaries.