Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

Haoyuan Wu, Haisheng Zheng, Zhuolun He, Bei Yu

TL;DR

The paper tackles capacity limits in instruction tuning of large language models by introducing Parameter-Efficient Sparsity Crafting (PESC), a method that upcycles dense models into sparse Mixture-of-Experts (MoE) blocks through the insertion of adapters. By initializing MoE experts from the original dense FFN and constraining training to lightweight adapters, while applying PEFT techniques like QLoRA to update remaining weights, PESC achieves scalable capacity expansion with minimal parameter overhead. The Camelidae family, trained with PESC, demonstrates strong general-task performance across coding, math, and reasoning benchmarks, outperforming several open-source sparse models and dense baselines and approaching GPT-3.5 in some settings. The work highlights the practical potential of adapter-based MoE design for efficient, high-capacity instruction tuning, while acknowledging trade-offs in parameter count and GPU memory versus full sparse upcycling.

Abstract

Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing (NLP) tasks. Instruction tuning, a successful paradigm, enhances the ability of LLMs to follow natural language instructions and exhibit robust generalization across general tasks. However, these models often encounter performance limitations across multiple tasks due to constrained model capacity. Expanding this capacity during the instruction tuning phase poses significant challenges. To address this issue, we introduce parameter-efficient sparsity crafting (PESC), which crafts dense models into sparse models using the mixture-of-experts (MoE) architecture. PESC integrates adapters into the MoE layers of sparse models, differentiating experts without altering the individual weights within these layers. This method significantly reduces computational costs and GPU memory requirements, facilitating model capacity expansion through a minimal parameter increase when guaranteeing the quality of approximation in function space compared to original sparse upcycling. Our empirical evaluation demonstrates the effectiveness of the PESC method. Using PESC during instruction tuning, our best sparse model outperforms other sparse and dense models and exhibits superior general capabilities compared to GPT-3.5. Our code is available at https://github.com/wuhy68/Parameter-Efficient-MoE.

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

TL;DR

The paper tackles capacity limits in instruction tuning of large language models by introducing Parameter-Efficient Sparsity Crafting (PESC), a method that upcycles dense models into sparse Mixture-of-Experts (MoE) blocks through the insertion of adapters. By initializing MoE experts from the original dense FFN and constraining training to lightweight adapters, while applying PEFT techniques like QLoRA to update remaining weights, PESC achieves scalable capacity expansion with minimal parameter overhead. The Camelidae family, trained with PESC, demonstrates strong general-task performance across coding, math, and reasoning benchmarks, outperforming several open-source sparse models and dense baselines and approaching GPT-3.5 in some settings. The work highlights the practical potential of adapter-based MoE design for efficient, high-capacity instruction tuning, while acknowledging trade-offs in parameter count and GPU memory versus full sparse upcycling.

Abstract

Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing (NLP) tasks. Instruction tuning, a successful paradigm, enhances the ability of LLMs to follow natural language instructions and exhibit robust generalization across general tasks. However, these models often encounter performance limitations across multiple tasks due to constrained model capacity. Expanding this capacity during the instruction tuning phase poses significant challenges. To address this issue, we introduce parameter-efficient sparsity crafting (PESC), which crafts dense models into sparse models using the mixture-of-experts (MoE) architecture. PESC integrates adapters into the MoE layers of sparse models, differentiating experts without altering the individual weights within these layers. This method significantly reduces computational costs and GPU memory requirements, facilitating model capacity expansion through a minimal parameter increase when guaranteeing the quality of approximation in function space compared to original sparse upcycling. Our empirical evaluation demonstrates the effectiveness of the PESC method. Using PESC during instruction tuning, our best sparse model outperforms other sparse and dense models and exhibits superior general capabilities compared to GPT-3.5. Our code is available at https://github.com/wuhy68/Parameter-Efficient-MoE.
Paper Structure (19 sections, 10 equations, 4 figures, 11 tables)

This paper contains 19 sections, 10 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminaries
  4. Parameter-Efficient Sparsity Crafting
  5. Model Design
  6. Experiments
  7. Settings
  8. Comparison with Chat LLMs
  9. Ablation Studies
  10. Routing Analysis
  11. Related Work
  12. Dense and Sparse Models
  13. Reuse of Trained Weights
  14. Parameter-Efficient Fine-Tuning
  15. Mixture of LoRA Experts
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Camelidae-8$\times$34B-pro achieves excellent performance across general tasks.
  • Figure 2: Overview of the parameter-efficient sparsity crafting with parameter-efficient experts.
  • Figure 3: Detailed design of the MoE layer for PESC utilizing parameter-efficient experts. All the FFN layers share the same weights.
  • Figure 4: Proportion of tokens assigned to each expert on different dataset subsets.