MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts

TL;DR

MoDE tackles multi-task adaptation for large language models by identifying redundancy in down-projection matrices of LoRA-based MoE and addressing it with a shared down-projection matrix and fine-grained rank-one adapters. The core idea is to decompose the LoRA update into dyadic components and route inputs through a token-level, per-column mixture of rank-one adapters, enabling expressive yet parameter-efficient task specialization. Empirical results on the Supernatural Instructions benchmark show MoDE outperforming LoRA, MoLORA, and MoLORA-SD under comparable parameter budgets, with robust ablations illustrating favorable trade-offs between the number of experts and the adapter rank. The work demonstrates practical, scalable parameter-efficient multi-task LLM adaptation and points to future directions in routing strategies, generalization to unseen tasks, and deployment on larger models.

Abstract

Parameter-efficient fine-tuning techniques like Low-Rank Adaptation (LoRA) have revolutionized the adaptation of large language models (LLMs) to diverse tasks. Recent efforts have explored mixtures of LoRA modules for multi-task settings. However, our analysis reveals redundancy in the down-projection matrices of these architectures. This observation motivates our proposed method, Mixture of Dyadic Experts (MoDE), which introduces a novel design for efficient multi-task adaptation. This is done by sharing the down-projection matrix across tasks and employing atomic rank-one adapters, coupled with routers that allow more sophisticated task-level specialization. Our design allows for more fine-grained mixing, thereby increasing the model's ability to jointly handle multiple tasks. We evaluate MoDE on the Supernatural Instructions (SNI) benchmark consisting of a diverse set of 700+ tasks and demonstrate that it outperforms state-of-the-art multi-task parameter-efficient fine-tuning (PEFT) methods, without introducing additional parameters. Our findings contribute to a deeper understanding of parameter efficiency in multi-task LLM adaptation and provide a practical solution for deploying high-performing, lightweight models.

Figures (6)

• Figure 1: Mixture of Dyadic Experts with 3 experts and a rank of 4, with each slice corresponding to a rank dimension. Our architecture allows independent routing at each rank. When number of mixtures is 1, our architecture is equivalent to traditional LoRA.
• Figure 2: Illustration of a basic LoRA module.
• Figure 3: Scatter plots showing the three most prominent principal components of all constituent vectors in the LoRA projection matrices for 15 independently trained single task model with shared initialization. Plots q_6_down and q_12_down (q_6_up and q_12_up) illustrate the down (up) projections of query matrices at layers 6 and 12, respectively. The clear clustering of down-projection vectors suggests that the down projection matrices are task-agnostic, motivating the design of the MoDE architecture.
• Figure 4: Illustration of (a) traditional LoRA Mixture-of-Experts, (b) traditional LoRA Mixture-of-Experts with shared down-projection matrix.
• Figure 5: Performance comparison among various model configurations on 15 tasks with a fixed parameter budget.