LoRAFusion: Efficient LoRA Fine-Tuning for LLMs

TL;DR

LoRAFusion tackles two main inefficiencies in LoRA-based fine-tuning of LLMs: memory-bound overhead from LoRA adapters and suboptimal throughput in multi-adapter, multi-job setups. It introduces FusedLoRA/FusedMultiLoRA kernels to shrink memory traffic by graph-splitting around the LoRA intermediate tensor, while deploying a MILP-guided, group-based scheduler to batch samples across adapters and jobs without violating execution dependencies. The results show end-to-end speedups up to 1.96× over Megatron-LM and 1.46× over mLoRA, plus kernel-level gains up to 1.39×, demonstrating significant improvements in throughput and load balance. LoRAFusion’s components are designed to be plug-and-play and generalizable to other LoRA variants and quantization, with open-source availability to enable broad adoption.

Abstract

Low-Rank Adaptation (LoRA) has become the leading Parameter-Efficient Fine-Tuning (PEFT) method for Large Language Models (LLMs), as it significantly reduces GPU memory usage while maintaining competitive fine-tuned model quality on downstream tasks. Despite these benefits, we identify two key inefficiencies in existing LoRA fine-tuning systems. First, they incur substantial runtime overhead due to redundant memory accesses on large activation tensors. Second, they miss the opportunity to concurrently fine-tune multiple independent LoRA adapters that share the same base model on the same set of GPUs. This leads to missed performance gains such as reduced pipeline bubbles, better communication overlap, and improved GPU load balance. To address these issues, we introduce LoRAFusion, an efficient LoRA fine-tuning system for LLMs. At the kernel level, we propose a graph-splitting method that fuses memory-bound operations. This design eliminates unnecessary memory accesses and preserves the performance of compute-bound GEMMs without incurring the cost of recomputation or synchronization. At the scheduling level, LoRAFusion introduces an adaptive batching algorithm for multi-job fine-tuning. It first splits LoRA adapters into groups to intentionally stagger batch execution across jobs, and then solves a bin-packing problem within each group to generate balanced, dependency-aware microbatches. LoRAFusion achieves up to $1.96\times$ ($1.47\times$ on average) end-to-end speedup compared to Megatron-LM, and up to $1.46\times$ ($1.29\times$ on average) improvement over mLoRA, the state-of-the-art multi-LoRA fine-tuning system. Our fused kernel achieves up to $1.39\times$ ($1.27\times$ on average) kernel performance improvement and can directly serve as a plug-and-play replacement in existing LoRA systems. We open-source LoRAFusion at https://github.com/CentML/lorafusion.

Figures (22)

• Figure 1: Comparison between traditional full model fine-tuning and LoRA fine-tuning.
• Figure 2: Comparison of (a) traditional batch padding, (b) dataset pre-packing, and (c) batch on-the-fly packing methods for LoRA fine-tuning of LLMs.
• Figure 3: Throughput comparison of the frozen linear layer ($n$=$k$=4096) vs. the corresponding LoRA linear layer with different numbers of tokens and ranks.
• Figure 4: Runtime breakdown of a LoRA linear module with $n$=$k$=4096, $r$=16, and $tokens$=8192. @ is matrix multiplication, d indicates a gradient, and .T represents a transpose.
• Figure 5: Ideal throughputof LLaMa-3.1-70B on 4 H100 GPUs vs. global batch sizes.