MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

TL;DR

MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism, which significantly reduces latency in multi-tenant settings compared to previous LoRA-based methods.

Abstract

Low-rank adaptation (LoRA) and its mixture-of-experts (MOE) variants are highly effective parameter-efficient fine-tuning (PEFT) methods. However, they introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules in the Transformer layer. To address this issue, we propose Mixture of Low-Rank Adaptation (MiLoRA), a novel and efficient LoRA variant. MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism. This mechanism calculates expert routing results once before generating the first new token and reuses these results for subsequent tokens, reducing latency. Extensive experiments and analysis on commonsense reasoning tasks, math reasoning tasks, and widely used LLM evaluation benchmarks demonstrate that MiLoRA consistently outperforms strong PEFT baselines with comparable tunable parameter budgets. Additionally, MiLoRA significantly reduces latency in multi-tenant settings compared to previous LoRA-based methods.

Figures (5)

• Figure 1: Schematic illustration of our MiLoRA method. Left: The architecture of a Transformer layer as in LlaMA-2 Touvron2023Llama2O. There are seven linear modules and seven positions to add LoRA modules. Right: Upon receiving an input prompt, the LoRA router before each Transformer layer will take the input prompt's hidden states as input features and go through a pooler, an activation function, and the MOE router network to determine which LoRA module is activated (or used) (e.g., LoRA U in the figure). This routing decision is repeatedly used when generating subsequent tokens.
• Figure 2: Distribution of LoRA experts across Transformer layers.
• Figure 3: Performances under different proportion of activated experts.
• Figure 4: Performances under different coefficient $\lambda_{lb}$.
• Figure 5: Performances under different numbers of tunable parameters.