FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic

TL;DR

FALQON is a novel framework that eliminates the quantization overhead from separate LoRA computational paths by directly merging LoRA adapters into an FP8-quantized backbone during fine-tuning, and its end-to-end FP8 workflow removes the need for post-training quantization, facilitating efficient deployment.

Abstract

Low-bit floating-point (FP) formats, such as FP8, provide significant acceleration and memory savings in model training thanks to native hardware support on modern GPUs and NPUs. However, we analyze that FP8 quantization offers speedup primarily for large-dimensional matrix multiplications, while inherent quantization overheads diminish speedup when applied to low-rank adaptation (LoRA), which uses small-dimensional matrices for efficient fine-tuning of large language models (LLMs). To address this limitation, we propose FALQON, a novel framework that eliminates the quantization overhead from separate LoRA computational paths by directly merging LoRA adapters into an FP8-quantized backbone during fine-tuning. Furthermore, we reformulate the forward and backward computations for merged adapters to significantly reduce quantization overhead, and introduce a row-wise proxy update mechanism that efficiently integrates substantial updates into the quantized backbone. Experimental evaluations demonstrate that FALQON achieves approximately a 3$\times$ training speedup over existing quantized LoRA methods with a similar level of accuracy, providing a practical solution for efficient large-scale model fine-tuning. Moreover, FALQON's end-to-end FP8 workflow removes the need for post-training quantization, facilitating efficient deployment. Code is available at https://github.com/iamkanghyunchoi/falqon.

Figures (9)

• Figure 1: Preliminary experiments on quantization overhead for LoRA ($r=64$) using linear layers from LLaMA-7B (self-attention, up-projection, down-projection). (a) FP8 reduces computation time. However, the quantization overhead dominates the computation and results in nearly fourfold higher latency. (b) FP8 shows consistently lower throughput than FP16 across ranks. These results demonstrate that FP8 quantization overhead cancels the expected acceleration in LoRA fine-tuning.
• Figure 2: Comparison between conventional LoRA fine-tuning and the proposed FALQON framework. Left: LoRA fine-tuning introduces redundant quantization for small-dimensional matrices (${\widetilde{A}}$, ${\widetilde{B}}$, and $O_A$), where the quantization overhead outweighs the computational benefit of FP8 arithmetic. Right: FALQON eliminates these overheads by directly melding LoRA into the quantized backbone, allowing the forward pass to be executed without separate small-tensor operations.
• Figure 3: Overall computational cost comparison of baselines and FALQON (lower is better).
• Figure 4: Breakdown analysis of LoRA fine-tuning. Compared to quantized LoRA baselines, which suffer from quantization overheads, FALQON minimizes redundant operations for superior efficiency.
• Figure 5: Comparison of training speed on various GPUs.