Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Run LoRA Run: Faster and Lighter LoRA Implementations

Daria Cherniuk, Aleksandr Mikhalev, Ivan Oseledets

TL;DR

LoRA reduces trainable parameters by using low-rank adapters, but standard implementations often follow suboptimal computation chains. RunLoRA introduces a FLOPs-driven framework that selects the best forward/backward passes for LoRA layers under a parameter-reduction constraint, implemented as a PyTorch-compatible wrapper with QLoRA support. It achieves up to 28% speedups and memory savings across Llama, RoBERTa, OPT and Llama2-7b, validating its practical impact for efficient large-model fine-tuning. The work lays groundwork for future exploration of optimal graphs for approximate adapters and broader hardware-aware optimizations.

Abstract

LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This technique is used both for fine-tuning and full training of large language models. This paper presents the RunLoRA framework for efficient implementations of LoRA that significantly improves the speed of neural network training and fine-tuning using low-rank adapters. The proposed implementation optimizes the computation of LoRA operations based on dimensions of corresponding linear layer, layer input dimensions and lora rank by choosing best forward and backward computation graph based on FLOPs and time estimations, resulting in faster training without sacrificing accuracy. The experimental results show up to 28\% speedup on language modeling networks.

Run LoRA Run: Faster and Lighter LoRA Implementations

TL;DR

LoRA reduces trainable parameters by using low-rank adapters, but standard implementations often follow suboptimal computation chains. RunLoRA introduces a FLOPs-driven framework that selects the best forward/backward passes for LoRA layers under a parameter-reduction constraint, implemented as a PyTorch-compatible wrapper with QLoRA support. It achieves up to 28% speedups and memory savings across Llama, RoBERTa, OPT and Llama2-7b, validating its practical impact for efficient large-model fine-tuning. The work lays groundwork for future exploration of optimal graphs for approximate adapters and broader hardware-aware optimizations.

Abstract

LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This technique is used both for fine-tuning and full training of large language models. This paper presents the RunLoRA framework for efficient implementations of LoRA that significantly improves the speed of neural network training and fine-tuning using low-rank adapters. The proposed implementation optimizes the computation of LoRA operations based on dimensions of corresponding linear layer, layer input dimensions and lora rank by choosing best forward and backward computation graph based on FLOPs and time estimations, resulting in faster training without sacrificing accuracy. The experimental results show up to 28\% speedup on language modeling networks.
Paper Structure (9 sections, 9 equations, 3 figures, 11 tables, 5 algorithms)

This paper contains 9 sections, 9 equations, 3 figures, 11 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Problem setting and Methodology
  3. Numerical experiments
  4. Llama
  5. RoBerta
  6. OPT
  7. Llama2-7b
  8. Conclusion and Future Work
  9. Related Work

Figures (3)

  • Figure 1: Maximum speedups for forward-backward pass through network achieved on different families of models and with different data types. Note that OPT implementation used in these experiments does not support float32 training. b - batch size, r - lora rank, s - sequence length.
  • Figure 2: Areas of best forward/backward pass choice. Region under the dashed line satisfies condition \ref{['eq:param_red']}.
  • Figure 3: Areas of best forward/backward pass choice for the LlamaMLP linear layer. Input dimention equals 4096, output dimention equals 11008. Rank is 128.