Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One Student Knows All Experts Know: From Sparse to Dense

Fuzhao Xue, Xiaoxin He, Xiaozhe Ren, Yuxuan Lou, Yang You

TL;DR

This work addresses the gap between sparse MoE models and deployable dense networks by introducing knowledge integration, a two-stage framework that gathers knowledge from a pre-trained MoE and distills it into a dense student (OneS). Four knowledge gathering methods—Summation, Averaging, Top-K Knowledge Gathering, and SVD Knowledge Gathering—initialize the dense model, which is then refined with soft knowledge distillation to preserve MoE benefits. Across ImageNet and NLP benchmarks, OneS achieves substantial MoE benefits with far fewer trainable parameters and significant inference speedups, demonstrating strong practicality. The approach offers a scalable, hardware-friendly path to leverage MoE power in dense architectures, with promising results on vision and language tasks and clear directions for future improvements.

Abstract

Human education system trains one student by multiple experts. Mixture-of-experts (MoE) is a powerful sparse architecture including multiple experts. However, sparse MoE model is easy to overfit, hard to deploy, and not hardware-friendly for practitioners. In this work, inspired by the human education model, we propose a novel task, knowledge integration, to obtain a dense student model (OneS) as knowledgeable as one sparse MoE. We investigate this task by proposing a general training framework including knowledge gathering and knowledge distillation. Specifically, to gather key knowledge from different pre-trained experts, we first investigate four different possible knowledge gathering methods, \ie summation, averaging, Top-K Knowledge Gathering (Top-KG), and Singular Value Decomposition Knowledge Gathering (SVD-KG) proposed in this paper. We then refine the dense student model by knowledge distillation to offset the noise from gathering. On ImageNet, our OneS preserves $61.7\%$ benefits from MoE and achieves $78.4\%$ top-1 accuracy ImageNet with only $15$M parameters. On four natural language processing datasets, OneS obtains $88.2\%$ MoE benefits and outperforms the best baseline by $51.7\%$ using the same architecture and training data. In addition, compared with the MoE counterpart, OneS can achieve $3.7 \times$ inference speedup due to less computation and hardware-friendly architecture.

One Student Knows All Experts Know: From Sparse to Dense

TL;DR

This work addresses the gap between sparse MoE models and deployable dense networks by introducing knowledge integration, a two-stage framework that gathers knowledge from a pre-trained MoE and distills it into a dense student (OneS). Four knowledge gathering methods—Summation, Averaging, Top-K Knowledge Gathering, and SVD Knowledge Gathering—initialize the dense model, which is then refined with soft knowledge distillation to preserve MoE benefits. Across ImageNet and NLP benchmarks, OneS achieves substantial MoE benefits with far fewer trainable parameters and significant inference speedups, demonstrating strong practicality. The approach offers a scalable, hardware-friendly path to leverage MoE power in dense architectures, with promising results on vision and language tasks and clear directions for future improvements.

Abstract

Human education system trains one student by multiple experts. Mixture-of-experts (MoE) is a powerful sparse architecture including multiple experts. However, sparse MoE model is easy to overfit, hard to deploy, and not hardware-friendly for practitioners. In this work, inspired by the human education model, we propose a novel task, knowledge integration, to obtain a dense student model (OneS) as knowledgeable as one sparse MoE. We investigate this task by proposing a general training framework including knowledge gathering and knowledge distillation. Specifically, to gather key knowledge from different pre-trained experts, we first investigate four different possible knowledge gathering methods, \ie summation, averaging, Top-K Knowledge Gathering (Top-KG), and Singular Value Decomposition Knowledge Gathering (SVD-KG) proposed in this paper. We then refine the dense student model by knowledge distillation to offset the noise from gathering. On ImageNet, our OneS preserves benefits from MoE and achieves top-1 accuracy ImageNet with only M parameters. On four natural language processing datasets, OneS obtains MoE benefits and outperforms the best baseline by using the same architecture and training data. In addition, compared with the MoE counterpart, OneS can achieve inference speedup due to less computation and hardware-friendly architecture.
Paper Structure (31 sections, 25 equations, 4 figures, 8 tables)

This paper contains 31 sections, 25 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Mixture-of-Experts
  4. Problem Formulation
  5. Approach
  6. Knowledge Gathering from MoE
  7. Summation and Averaging
  8. Top-K Knowledge Gathering
  9. SVD Knowledge Gathering
  10. Knowledge Distillation
  11. Optimization
  12. Experiments
  13. Computer Vision
  14. Computer Vision
  15. Results on ImageNet
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Human education model matches MoE and dense model.
  • Figure 2: An overview of our general training framework proposed. The overall training framework is knowledge integration, and it includes two stages, knowledge gathering and knowledge distillation. In knowledge gathering, we investigate four different methods to merge the knowledge from MoE, including summation, averaging and Top-KG and SVD-KG.
  • Figure 3: Top-1 Accuracy of ablation on ImageNet to investigate the contribution of more global training epochs.
  • Figure 4: Top-1 Accuracy of ablation on ImageNet to ablate the SVD ratio $\lambda$.