Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models

TL;DR

RoE addresses the computational burden of multi-modal LLMs by enabling dynamic, example-dependent routing within models that are already trained, effectively turning them into mixtures of experts without rearchitecting from scratch. The method adds adapter-based skip connections, a structure sparsity regularization, and routing tokens to align training and inference, and optimizes these via a three-stage training scheme. Across three MLLMs and ten VL benchmarks, RoE achieves meaningful speedups (e.g., ~24% on certain models) with minimal or no loss in accuracy, outperforming prior sparse MoE approaches on several tasks. The approach provides a practical, cost-efficient path to deploying high-capacity MLLMs in real-time settings while maintaining strong performance on vision-language reasoning tasks.

Abstract

Recently, mixture of experts (MoE) has become a popular paradigm for achieving the trade-off between modal capacity and efficiency of multi-modal large language models (MLLMs). Different from previous efforts, we are dedicated to exploring the dynamic expert path in an already exist MLLM and show that a standard MLLM can be also a mixture of experts. To approach this target, we propose a novel dynamic expert scheme for MLLMs, termed Routing Experts (RoE), which can achieve example-dependent optimal path routing without obvious structure tweaks. Meanwhile, a new regularization of structure sparsity is also introduced to enforce MLLMs to learn more short-cut inference, ensuring the efficiency. In addition, we also realize the first attempt of aligning the training and inference schemes of MLLMs in terms of network routing. To validate RoE, we apply it to a set of latest MLLMs, including LLaVA-1.5, LLaVA-HR and VILA, and conduct extensive experiments on a bunch of VL benchmarks. The experiment results not only show the great advantages of our RoE in improving MLLMs' efficiency, but also yield obvious advantages than MoE-LLaVA in both performance and speed, e.g., an average performance gain of 3.3% on 5 benchmarks while being faster.

Figures (3)

• Figure 1: The visualization of the $l1$-distances between the input and output features of each layer of LLaVA-7B LLaVA. A lower $l1$-distance indicates that this layer has less impact on the feature update of this example, which also suggests that it is not that important during inference. For two examples, the contributions of different layers are also different.
• Figure 2: Illustration of the proposed Routing Experts (RoE). Existing MoE models (a) often build a new sparse structure with multiple FFNs as experts, and each pathway takes the same computation for all examples. Our RoE (b) aims to explore the expert pathways within the model itself via adapter-based skip connections, realizing dynamic computation for different examples. (c) Routing tokens are used to decide layer-wise path selection, i.e., the adapter-based skip connection or the default Transformer layer. It also serves to align the training and testing of MLLMs.
• Figure 3: Examples of our RoE on LLaVA. Example (a) and (b) show the comparison between RoE-LLaVA and LLaVA and MoE-LLaVA. Our RoE-LLaVA can answer the questions as accurately and in detail as LLaVA, while being faster, i.e., more tokens per second (tokens/s$\uparrow$). Example (c) shows the predictions of RoE-LLaVA for the same image and different questions. RoE can adjust the choice and depth of expert pathways according to the question, i.e., the bar charts (the yellow ones denote the skipped layers). Example (d) shows the predictions for the same question but different images. RoE can also route different optimal expert pathways according to different visual content.