DR-LoRA: Dynamic Rank LoRA for Mixture-of-Experts Adaptation

TL;DR

A Dynamic Rank LoRA framework named DR-LoRA is proposed, which dynamically grows expert LoRA ranks during fine-tuning based on task-specific demands and employs an Expert Saliency Scoring mechanism that integrates expert routing frequency and LoRA rank importance to quantify each expert's demand for additional capacity.

Abstract

Mixture-of-Experts (MoE) has become a prominent paradigm for scaling Large Language Models (LLMs). Parameter-efficient fine-tuning (PEFT), such as LoRA, is widely adopted to adapt pretrained MoE LLMs to downstream tasks. However, existing approaches assign identical LoRA ranks to all experts, overlooking the intrinsic functional specialization within MoE LLMs. This uniform allocation leads to resource mismatch, task-relevant experts are under-provisioned while less relevant ones receive redundant parameters. We propose a Dynamic Rank LoRA framework named DR-LoRA, which dynamically grows expert LoRA ranks during fine-tuning based on task-specific demands. DR-LoRA employs an Expert Saliency Scoring mechanism that integrates expert routing frequency and LoRA rank importance to quantify each expert's demand for additional capacity. Experts with higher saliency scores are prioritized for rank expansion, enabling the automatic formation of a heterogeneous rank distribution tailored to the target task. Experiments on multiple benchmarks demonstrate that DR-LoRA consistently outperforms standard LoRA and static allocation strategies under the same parameter budget, achieving superior task performance with more efficient parameter utilization.

Figures (7)

• Figure 1: The overview of the DR-LoRA framework. Pre-trained expert weights are frozen, while each expert is equipped with a trainable LoRA module. These modules start with a small initial rank ($r_{\text{init}}$) and can dynamically grow ($\Delta r$) during training. Expert Saliency Scoring guides rank growth by integrating two real-time signals: (1) Expert Routing Frequency ($f_i$), tracked from the router's decisions to measure task relevance, and (2) LoRA Rank Importance ($g_i$), derived from the gradient signals of the trainable LoRA matrices (A and B) to measure learning intensity.
• Figure 2: Average accuracy on task-aligned benchmarks (GSM8k, HumanEval, IFEval) during training. DR-LoRA establishes early superiority and maintains the advantage throughout training.
• Figure 3: Performance degradation when masking expert subgroups. On math tasks (GSM8k), masking large experts causes 4$\times$ greater degradation than masking small experts, confirming task-aligned capacity allocation. On general knowledge (MMLU), both groups contribute similarly.
• Figure 4: Expert activation heatmaps on different tasks. On GSM8k, DR-LoRA and LoRA exhibit distinctly different activation patterns. On MMLU, both methods activate largely overlapping expert sets.
• Figure 5: Evolution of expert LoRA ranks during DR-LoRA training on OLMoE. Each heatmap shows the average rank per expert (averaged over up_proj and down_proj) at different training stages, with high-rank experts (darker red) concentrated in task-relevant positions.