ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

TL;DR

ToMoE addresses the challenge of deploying large language models on resource-constrained devices by converting dense decoders into sparse Mixture-of-Experts without updating base weights. It achieves this through differentiable dynamic pruning that assigns a fixed budget of active parameters, using top-K routing for MHA and top-1 routing for MLP, guided by a hypernetwork-generated set of expert embeddings. The approach jointly optimizes routers and expert configurations with regularizations that encourage full parameter utilization and balanced load, yielding consistent improvements over structural pruning and prior MoE-construction methods across multiple model families without fine-tuning. Practically, ToMoE reveals latent expert capacity within dense LLMs and offers a scalable, cost-efficient pathway for deploying and serving large models on diverse hardware.

Abstract

Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.

Figures (10)

• Figure 1: (a): The original LLM uses all parameters to process all the input text. (b): The static pruning for LLMs permanently removes model parameters, and the rest of the parameters are used to process all the input text. Our approach (c): LLMs with dynamic pruning use different sub-networks (illustrated by different colors) to process different tokens. We incorporate MoE to achieve a fixed expected budget in inference.
• Figure 2: ToMoE uses top-1 routing for MLP layers, and static and dynamic pruning along the head dimension for MHA layers.
• Figure 3: The training dynamics give different ratios $p$ of active parameters on the Qwen-2.5 7B model.
• Figure 4: Experts token allocation of ToMoE for the LLaMA-3 8B model collected on the WikiText dataset.
• Figure 5: Model width after ToMoE for the Qwen-2.5 7B model when the number of active parameters equals 50%.