Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning More Generalized Experts by Merging Experts in Mixture-of-Experts

Sejik Park

TL;DR

This work tackles the instability of learning shared features in Mixture-of-Experts (MoE) for multi-domain task incremental learning by introducing an expert-weight merging mechanism. By tracking expert usage and periodically averaging the weights of the two most frequently selected experts to form a generalized expert that replaces the least-used one, the method aims to enhance transfer learning and mitigate catastrophic forgetting within MTIL framed on MA with CLIP adapters. Experiments show modest but consistent gains in transfer and average accuracy across several MTIL tasks, while highlighting variability due to seed choice and the heuristic nature of merging. The approach offers a lightweight pathway to improve MoE plasticity and feature stitching, albeit with limitations that warrant further validation and broader MoE evaluations.

Abstract

We observe that incorporating a shared layer in a mixture-of-experts can lead to performance degradation. This leads us to hypothesize that learning shared features poses challenges in deep learning, potentially caused by the same feature being learned as various different features. To address this issue, we track each expert's usage frequency and merge the two most frequently selected experts. We then update the least frequently selected expert using the combination of experts. This approach, combined with the subsequent learning of the router's expert selection, allows the model to determine if the most frequently selected experts have learned the same feature differently. If they have, the combined expert can be further trained to learn a more general feature. Consequently, our algorithm enhances transfer learning and mitigates catastrophic forgetting when applied to multi-domain task incremental learning.

Learning More Generalized Experts by Merging Experts in Mixture-of-Experts

TL;DR

This work tackles the instability of learning shared features in Mixture-of-Experts (MoE) for multi-domain task incremental learning by introducing an expert-weight merging mechanism. By tracking expert usage and periodically averaging the weights of the two most frequently selected experts to form a generalized expert that replaces the least-used one, the method aims to enhance transfer learning and mitigate catastrophic forgetting within MTIL framed on MA with CLIP adapters. Experiments show modest but consistent gains in transfer and average accuracy across several MTIL tasks, while highlighting variability due to seed choice and the heuristic nature of merging. The approach offers a lightweight pathway to improve MoE plasticity and feature stitching, albeit with limitations that warrant further validation and broader MoE evaluations.

Abstract

We observe that incorporating a shared layer in a mixture-of-experts can lead to performance degradation. This leads us to hypothesize that learning shared features poses challenges in deep learning, potentially caused by the same feature being learned as various different features. To address this issue, we track each expert's usage frequency and merge the two most frequently selected experts. We then update the least frequently selected expert using the combination of experts. This approach, combined with the subsequent learning of the router's expert selection, allows the model to determine if the most frequently selected experts have learned the same feature differently. If they have, the combined expert can be further trained to learn a more general feature. Consequently, our algorithm enhances transfer learning and mitigates catastrophic forgetting when applied to multi-domain task incremental learning.
Paper Structure (7 sections, 3 figures, 3 tables, 1 algorithm)

This paper contains 7 sections, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Experiments
  5. Results
  6. Limitations
  7. Conclusion

Figures (3)

  • Figure 1: Overview of Principle The top Algorithm block shows how MoE training proceeds in relation to creating an expert with a new distribution with our suggested learning algorithm. The bottom left Avoid Catastrophic Forgetting block explains why having a more general distribution of the new expert can help in consistently creating the same features. And the bottom right Gain More Plasticity block shows why the updates in our algorithm can be beneficial for additional learning.
  • Figure 2: Overall Algorithm Explain how our algorithm operates on an MoE with routing done by the top-k selection method when the number of experts is set to three.
  • Figure 3: Number of Fixed Experts over Training This figure visualizes the number of fixed experts in each block of the encoder (horizontal axis) as the task progresses (vertical axis).