OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models
Kerim Büyükakyüz
TL;DR
Fine-tuning large language models is computationally intensive; LoRA provides parameter-efficient adaptation but can suffer from slow convergence. OLoRA introduces orthonormal initialization via QR decomposition to constrain low-rank updates, yielding faster convergence while maintaining LoRA's efficiency. Across multiple models and commonsense benchmarks, OLoRA demonstrates faster training and often superior final performance, albeit with a one-time QR overhead that is amortized over training. This approach offers a practical path to faster, more stable fine-tuning of large models with broader accessibility for downstream applications.
Abstract
The advent of large language models (LLMs) has revolutionized natural language processing, enabling unprecedented capabilities in understanding and generating human-like text. However, the computational cost and convergence times associated with fine-tuning these models remain significant challenges. Low-Rank Adaptation (LoRA) has emerged as a promising method to mitigate these issues by introducing efficient fine-tuning techniques with a reduced number of trainable parameters. In this paper, we present OLoRA, an enhancement to the LoRA method that leverages orthonormal matrix initialization through QR decomposition. OLoRA significantly accelerates the convergence of LLM training while preserving the efficiency benefits of LoRA, such as the number of trainable parameters and GPU memory footprint. Our empirical evaluations demonstrate that OLoRA not only converges faster but also exhibits improved performance compared to standard LoRA across a variety of language modeling tasks. This advancement opens new avenues for more efficient and accessible fine-tuning of LLMs, potentially enabling broader adoption and innovation in natural language applications.