mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs

Abstract

Transformer-based, pre-trained large language models (LLMs) have demonstrated outstanding performance across diverse domains, particularly in the emerging {\em pretrain-then-finetune} paradigm. Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is commonly used to adapt a base LLM to multiple downstream tasks. Further, LLM platforms enable developers to fine-tune multiple models and develop various domain-specific applications simultaneously. However, existing model parallelism schemes suffer from high communication overhead and inefficient GPU utilization when training multiple LoRA tasks across GPUs and machines. In this paper, we present mLoRA, a parallelism-efficient fine-tuning system designed for training multiple LoRA across GPUs and machines. mLoRA introduces a novel LoRA-aware pipeline parallelism scheme that efficiently pipelines independent LoRA adapters and their distinct fine-tuning stages across GPUs and machines, along with a new LoRA-efficient operator to enhance GPU utilization during pipelined LoRA training. Our extensive evaluation shows that mLoRA can significantly reduce average fine-tuning task completion time, e.g., by 30\%, compared to state-of-the-art methods like FSDP. More importantly, mLoRA enables simultaneous fine-tuning of larger models, e.g., two Llama-2-13B models on four NVIDIA RTX A6000 48GB GPUs, which is not feasible for FSDP due to high memory requirements. Hence, mLoRA not only increases fine-tuning efficiency but also makes it more accessible on cost-effective GPUs. mLoRA has been deployed in AntGroup's production environment.

Figures (16)

• Figure 1: Sharing pre-trained model weights for fine-tuning multiple LoRA adapters with reduced overhead.
• Figure 2: Overview of mLoRA.
• Figure 3: The workflow of LoRAPP.
• Figure 4: (a) GPipe. The training data of the fine-tuning task are divided into four micro-batches within a mini-batch. Here, $F_i$ represents the forward propagation of the $i$th micro-batch, while $B_i$ represents its backward propagation. GPipe requires all micro-batches of the same mini-batch to be completed before proceeding to the next mini-batch. (b) (c) LoRAPP without mini-batch splitting. $Fi$ represents the forward propagation of the $i$th LoRA adapter, while $Bi$ is its backward propagation. (d) LoRAPP with mini-batch splitting. $Fi_j$ represents the forward propagation of the $j$th mini-batch, into which the macro-batch of training data for the $i$th LoRA adapter is divided, while $Bi_j$ represents its backward propagation. (e) LoRAPP with BatchLoRA.
• Figure 5: Overlapping communication in LoRAPP. $Fi$ represents the forward propagation of the $i$th LoRA adapter, while $Bi$ is its backward propagation.