Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Models

Hongyu Wang, Jiayu Xu, Ruiping Wang, Yan Feng, Yitao Zhai, Peng Pei, Xunliang Cai, Xilin Chen

TL;DR

MoTE tackles the memory bottleneck of large multimodal Mixture-of-Experts by training a larger pool of low-precision ternary experts while keeping a frozen, high-precision shared FFN from the dense checkpoint. The method uses a three-stage training pipeline with quantization-aware training and a load-balancing loss to distribute routing across experts, yielding a memory footprint comparable to or smaller than full-precision baselines. Empirically, MoTE matches or exceeds full-precision up-cycling MoE-LLaVA at scales above 1.5B parameters and, under the same expert memory budget (e.g., $3.4$GB), achieves up to $4.3 ext{%}$ higher average end-task accuracy, with further gains when combined with post-training quantization. These results demonstrate the practicality of memory-efficient ternary MoEs for edge devices and scalable multimodal modeling.

Abstract

Large multimodal Mixture-of-Experts (MoEs) effectively scale the model size to boost performance while maintaining fixed active parameters. However, previous works primarily utilized full-precision experts during sparse up-cycling. Despite they show superior performance on end tasks, the large amount of experts introduces higher memory footprint, which poses significant challenges for the deployment on edge devices. In this work, we propose MoTE, a scalable and memory-efficient approach to train Mixture-of-Ternary-Experts models from dense checkpoint. Instead of training fewer high-precision experts, we propose to train more low-precision experts during up-cycling. Specifically, we use the pre-trained FFN as a shared expert and train ternary routed experts with parameters in {-1, 0, 1}. Extensive experiments show that our approach has promising scaling trend along model size. MoTE achieves comparable performance to full-precision baseline MoE-LLaVA while offering lower memory footprint. Furthermore, our approach is compatible with post-training quantization methods and the advantage further amplifies when memory-constraint goes lower. Given the same amount of expert memory footprint of 3.4GB and combined with post-training quantization, MoTE outperforms MoE-LLaVA by a gain of 4.3% average accuracy on end tasks, demonstrating its effectiveness and potential for memory-constrained devices.

MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Models

TL;DR

MoTE tackles the memory bottleneck of large multimodal Mixture-of-Experts by training a larger pool of low-precision ternary experts while keeping a frozen, high-precision shared FFN from the dense checkpoint. The method uses a three-stage training pipeline with quantization-aware training and a load-balancing loss to distribute routing across experts, yielding a memory footprint comparable to or smaller than full-precision baselines. Empirically, MoTE matches or exceeds full-precision up-cycling MoE-LLaVA at scales above 1.5B parameters and, under the same expert memory budget (e.g., GB), achieves up to higher average end-task accuracy, with further gains when combined with post-training quantization. These results demonstrate the practicality of memory-efficient ternary MoEs for edge devices and scalable multimodal modeling.

Abstract

Large multimodal Mixture-of-Experts (MoEs) effectively scale the model size to boost performance while maintaining fixed active parameters. However, previous works primarily utilized full-precision experts during sparse up-cycling. Despite they show superior performance on end tasks, the large amount of experts introduces higher memory footprint, which poses significant challenges for the deployment on edge devices. In this work, we propose MoTE, a scalable and memory-efficient approach to train Mixture-of-Ternary-Experts models from dense checkpoint. Instead of training fewer high-precision experts, we propose to train more low-precision experts during up-cycling. Specifically, we use the pre-trained FFN as a shared expert and train ternary routed experts with parameters in {-1, 0, 1}. Extensive experiments show that our approach has promising scaling trend along model size. MoTE achieves comparable performance to full-precision baseline MoE-LLaVA while offering lower memory footprint. Furthermore, our approach is compatible with post-training quantization methods and the advantage further amplifies when memory-constraint goes lower. Given the same amount of expert memory footprint of 3.4GB and combined with post-training quantization, MoTE outperforms MoE-LLaVA by a gain of 4.3% average accuracy on end tasks, demonstrating its effectiveness and potential for memory-constrained devices.

Paper Structure

This paper contains 33 sections, 7 equations, 9 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mixture of Experts.
  4. Model Quantization.
  5. MoTE: Mixture-of-Ternary-Experts
  6. Architecture
  7. Training recipe
  8. Training objectives
  9. Language modeling loss.
  10. Load balancing loss.
  11. Experiments
  12. Setup
  13. Model settings.
  14. Implementation details.
  15. Training data.
  16. ...and 18 more sections

Figures (9)

  • Figure 1: The overview of MoTE. We retain the pre-trained full-precision FFN as a shared expert and add a top-1 activated MoE layer with ternary experts. All experts and attention layers are initialized from the dense checkpoint.
  • Figure 2: Visualization of the routing distributions of all tokens, text tokens, image tokens across all experts on the en-test set of MMBench.
  • Figure 3: Visualization of the routing distributions of all tokens, text tokens, image tokens across all experts on various tasks.
  • Figure 4: Visualization of the modality-aware routing distributions for each expert on various tasks.
  • Figure 5: Visualization of the top-10 activated pathways for text and image modality on various tasks.
  • ...and 4 more figures