Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, Jun Yu

TL;DR

Low-Rank Clone (LRC) presents a unified, alignment-free distillation framework that jointly compresses teacher weights via trainable low-rank projections and preserves behavioral fidelity through Activation Clone, notably leveraging FFN activations. By generating the student’s weights directly from the teacher and aligning a wide set of activations, LRC achieves state-of-the-art or competitive performance for small language models with orders of magnitude fewer training tokens. Across open-source teacher configurations, LRC matches or surpasses baselines trained on trillions of tokens, while delivering up to 1,000x training efficiency. The method also demonstrates compatibility with other compression techniques and provides insights into FFN knowledge transfer and data-quality effects, underscoring practical benefits for democratizing access to high-performing SLMs.

Abstract

Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC), an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC matches or surpasses state-of-the-art models trained on trillions of tokens--while using only 20B tokens, achieving over 1,000x training efficiency. Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone

TL;DR

Low-Rank Clone (LRC) presents a unified, alignment-free distillation framework that jointly compresses teacher weights via trainable low-rank projections and preserves behavioral fidelity through Activation Clone, notably leveraging FFN activations. By generating the student’s weights directly from the teacher and aligning a wide set of activations, LRC achieves state-of-the-art or competitive performance for small language models with orders of magnitude fewer training tokens. Across open-source teacher configurations, LRC matches or surpasses baselines trained on trillions of tokens, while delivering up to 1,000x training efficiency. The method also demonstrates compatibility with other compression techniques and provides insights into FFN knowledge transfer and data-quality effects, underscoring practical benefits for democratizing access to high-performing SLMs.

Abstract

Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC), an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC matches or surpasses state-of-the-art models trained on trillions of tokens--while using only 20B tokens, achieving over 1,000x training efficiency. Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.
Paper Structure (58 sections, 1 theorem, 15 equations, 7 figures, 22 tables, 2 algorithms)

This paper contains 58 sections, 1 theorem, 15 equations, 7 figures, 22 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Distillation and Structured Pruning
  4. Small Language Models (SLMs)
  5. Low-Rank Clone
  6. Background and Notation
  7. Low-Rank Projection
  8. Attention and FFN Weight Projection
  9. Embedding and LM Head Projection
  10. Structural Compatibility and Deployment
  11. Activation Clone
  12. Alignment Free Design
  13. Remarks
  14. Experiments
  15. Experiment Setup
  16. ...and 43 more sections

Key Result

Lemma 1

Let $\bm{W}_{\mathrm{down},i}^\mathrm{S}$ denote the FFN down-projection weight in the student model at layer $i$, derived via the low-rank projection from the teacher's weight $\bm{W}_{\mathrm{down},i}^\mathrm{T}$ and projection matrix $\bm{W}_{\mathrm{down},i}^\mathrm{p}$, such that: If the intermediate FFN activations $\bm{h}_{\mathrm{up},i}$ and $\bm{h}_{\mathrm{gate},i}$ are perfectly cloned

Figures (7)

  • Figure 1: LRC results that achieve higher accuracy with 1,000$\times$ fewer training tokens, significantly boosting efficiency.
  • Figure 2: The overall procedure of LRC. To ensure clarity, attention and normalization modules are omitted. LRC involves two main steps: (1) Low-Rank Projection: applying low-rank projection matrices to compress the teacher's weights into a lower-dimensional space, which are then assigned to the student. (2) Activation Clone, executing standard forward passes in both models to collect intermediate activations, which are aligned using Mean Squared Error (MSE) loss.
  • Figure 3: Effect of LRC component ablations on LM loss convergence over training time.
  • Figure 4: The trend of MMLU scores with increasing training tokens.
  • Figure 5: The trend of ARC-C scores with increasing training tokens.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Lemma 1: Alignment-Free FFN Output Cloning