A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation

TL;DR

The paper presents a unified, taxonomy-driven study of LoRA variants, introducing LoRAFactory as a modular codebase to implement 50+ variants under a common interface. It provides a theoretical framework based on low-rank update dynamics and conducts large-scale empirical evaluations across NLG, NLU, and image classification, revealing that learning-rate choice is a dominant factor and that properly tuned LoRA configurations often match or outperform many variants. The work clarifies the relationships among rank-adjustment, optimization-dynamics, initialization, and MoE integration approaches, and it demonstrates the practical value of standardized evaluation for fair comparisons. Overall, the results highlight LoRA’s robustness and emphasize the need for extensive hyperparameter search to achieve fair, meaningful benchmarking in PEFT research.

Abstract

Low-Rank Adaptation (LoRA) is a fundamental parameter-efficient fine-tuning method that balances efficiency and performance in large-scale neural networks. However, the proliferation of LoRA variants has led to fragmentation in methodology, theory, code, and evaluation. To this end, this work presents the first unified study of LoRA variants, offering a systematic taxonomy, unified theoretical review, structured codebase, and standardized empirical assessment. First, we categorize LoRA variants along four principal axes: rank, optimization dynamics, initialization, and integration with Mixture-of-Experts. Then, we review their relationships and evolution within a common theoretical framework focused on low-rank update dynamics. Further, we introduce LoRAFactory, a modular codebase that implements variants through a unified interface, supporting plug-and-play experimentation and fine-grained analysis. Last, using this codebase, we conduct a large-scale evaluation across natural language generation, natural language understanding, and image classification tasks, systematically exploring key hyperparameters. Our results uncover several findings, notably: LoRA and its variants exhibit pronounced sensitivity to the choices of learning rate compared to other hyperparameters; moreover, with proper hyperparameter configurations, LoRA consistently matches or surpasses the performance of most of its variants.

Figures (10)

• Figure 1: Hierarchical taxonomy of LoRA variants based on four core principle operational axes.
• Figure 2: Illustration of rank adjustment based LoRA variants.
• Figure 3: Illustration of optimization process adjustment based LoRA variants.
• Figure 4: Illustration of initialization adjustment based LoRA variants.
• Figure 5: Illustration of mixture of experts Integration based LoRA variants.