Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoFE: Mixture of Frozen Experts Architecture

Jean Seo, Jaeyoon Kim, Hyopil Shin

TL;DR

MoFE tackles the resource-intensive nature of large language models by fusing PEFT with MoE while freezing FFN blocks to fix the trainable parameter budget. The architecture combines a base model, frozen FFN-based experts, and a router that selects up to two experts per token, maintaining a fixed trainable size (e.g., $0.34\ ext{B}$) regardless of the number of experts. Empirically, MoFE delivers substantial training-time savings and outperforms other PEFT methods on several benchmarks, while enabling knowledge transfer from domain-specific experts; however, full fine-tuning can still surpass MoFE in some cases. The results support MoFE as a viable option for resource-constrained scenarios and multi-domain proficiency, with insights into optimal training strategies and domain-expert composition.

Abstract

We propose the Mixture of Frozen Experts (MoFE) architecture, which integrates Parameter-efficient Fine-tuning (PEFT) and the Mixture of Experts (MoE) architecture to enhance both training efficiency and model scalability. By freezing the Feed Forward Network (FFN) layers within the MoE framework, MoFE significantly reduces the number of trainable parameters, improving training efficiency while still allowing for effective knowledge transfer from the expert models. This facilitates the creation of models proficient in multiple domains. We conduct experiments to evaluate the trade-offs between performance and efficiency, compare MoFE with other PEFT methodologies, assess the impact of domain expertise in the constituent models, and determine the optimal training strategy. The results show that, although there may be some trade-offs in performance, the efficiency gains are substantial, making MoFE a reasonable solution for real-world, resource-constrained environments.

MoFE: Mixture of Frozen Experts Architecture

TL;DR

MoFE tackles the resource-intensive nature of large language models by fusing PEFT with MoE while freezing FFN blocks to fix the trainable parameter budget. The architecture combines a base model, frozen FFN-based experts, and a router that selects up to two experts per token, maintaining a fixed trainable size (e.g., ) regardless of the number of experts. Empirically, MoFE delivers substantial training-time savings and outperforms other PEFT methods on several benchmarks, while enabling knowledge transfer from domain-specific experts; however, full fine-tuning can still surpass MoFE in some cases. The results support MoFE as a viable option for resource-constrained scenarios and multi-domain proficiency, with insights into optimal training strategies and domain-expert composition.

Abstract

We propose the Mixture of Frozen Experts (MoFE) architecture, which integrates Parameter-efficient Fine-tuning (PEFT) and the Mixture of Experts (MoE) architecture to enhance both training efficiency and model scalability. By freezing the Feed Forward Network (FFN) layers within the MoE framework, MoFE significantly reduces the number of trainable parameters, improving training efficiency while still allowing for effective knowledge transfer from the expert models. This facilitates the creation of models proficient in multiple domains. We conduct experiments to evaluate the trade-offs between performance and efficiency, compare MoFE with other PEFT methodologies, assess the impact of domain expertise in the constituent models, and determine the optimal training strategy. The results show that, although there may be some trade-offs in performance, the efficiency gains are substantial, making MoFE a reasonable solution for real-world, resource-constrained environments.

Paper Structure

This paper contains 15 sections, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MoFE
  4. Architecture
  5. Main Components
  6. Empirical Analysis
  7. Experimental Setting
  8. What is the trade-off between efficiency and performance?
  9. How good is MoFE compared to other PEFT methods?
  10. What effect does the domain expertise of consisting models have?
  11. Expert Model
  12. Base Model
  13. What is the optimal training strategy?
  14. Conclusion
  15. Does the number of frozen FFN blocks affect performance?

Figures (1)

  • Figure 1: Mixture of Frozen Experts Architecture. In this example figure, the router uses 2 Feed Forward Network (FFN) blocks at each time step (m = 2), and there are 4 FFN blocks, or expert models, used (n = 4). In MoFE, the FFN blocks are frozen, so only the remaining parameters are updated. This makes the training process significantly more lightweight, regardless of the number of expert models integrated into the architecture.