FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation

TL;DR

FlexLoRA tackles the cost of adapting large pre-trained models by introducing entropy-guided flexible low-rank adaptation for parameter-efficient fine-tuning. It leverages a matrix-level spectral entropy score $I(\mathbf{\Lambda})$ to guide bidirectional rank allocation under a global budget, employing a zero-impact initialization for newly added directions and an SVD-like update $\Delta \mathbf{W} = \mathbf{P} \mathbf{\Lambda} \mathbf{Q}$ with orthogonality regularization. The approach yields state-of-the-art or competitive results across NLP (GLUE, CoLA, RTE) and vision (VTAB) benchmarks at the same parameter budgets, outperforming LoRA and AdaLoRA. These findings underscore the importance of principled, matrix-aware capacity reallocation for effective, scalable PEFT with interpretable layer-wise adaptation patterns.

Abstract

Large pre-trained models achieve remarkable success across diverse domains, yet fully fine-tuning incurs prohibitive computational and memory costs. Parameter-efficient fine-tuning (PEFT) has thus become a mainstream paradigm. Among them, Low-Rank Adaptation (LoRA) introduces trainable low-rank matrices and shows strong performance, nevertheless, its fixed-rank design limits flexibility. Dynamic rank allocation methods mitigate this issue by pruning redundant directions; however, they often rely on heuristic, element-level metrics that globally sort rank directions without matrix-wise distinction, and they lack mechanisms to expand capacity in layers requiring additional adaptation. To overcome these limitations, we propose FlexLoRA, an entropy-guided flexible low-rank adaptation framework that (i) evaluates matrix importance via spectral energy entropy, (ii) supports rank pruning and expansion under a global budget, and (iii) employs zero-impact initialization for newly added singular directions to ensure stability. By addressing granularity, flexibility, and stability limitations, FlexLoRA provides a more principled solution for PEFT. Extensive experiments show that FlexLoRA consistently outperforms state-of-the-art baselines across benchmarks. Codes are available at https://github.com/Chongjie-Si/Subspace-Tuning.

Figures (3)

• Figure 1: Framework of FlexLoRA. For each weight matrix $\mathbf{W}_k$, FlexLoRA represents the update in an SVD-like form $\Delta \mathbf{W} = \mathbf{P}_k \mathbf{\Lambda}_k \mathbf{Q}_k$, where $\mathbf{\Lambda}_k$ is a diagonal matrix. It then computes a spectral entropy–based importance score for each $\Delta \mathbf{W}$. All scores are globally ranked under a given rank budget: matrices with lower scores prune the least significant direction in $\mathbf{\Lambda}_k$, while those with higher scores receive additional ranks. The newly allocated ranks are initialized with a zero-impact scheme to preserve the original input while enabling subsequent learning.
• Figure 2: Visualization of rank allocation during FlexLoRA training on the CoLA task. We selected the modules with the most significant changes in rank and sorted them by layer depth. Modules are sorted by layer depth from top (shallow) to bottom (deep). K, V, Q denote key, value, and query projections; A is attention output; I is intermediate dense; O is output dense. Darker colors indicate more capacity, lighter colors indicate stronger pruning.
• Figure 3: Singular value distributions of ten randomly sampled LoRA modules trained on the CoLA task.