Null-LoRA: Low-Rank Adaptation on Null Space

TL;DR

Null-LoRA tackles the inefficiency of traditional low-rank fine-tuning by leveraging the null space of pretrained weights. It freezes halves of the low-rank factors, cross-pairs them, and enforces updates to lie in the null space, thereby increasing effective rank while keeping trainable parameters low. The method also adapts the per-layer rank based on each layer's nullity and balances update norms, achieving state-of-the-art results on image-text retrieval and VQA with fewer tunable parameters and no latency overhead. Overall, Null-LoRA demonstrates robust parameter efficiency for cross-modal tasks by decoupling fine-tuning updates from pre-trained directions.

Abstract

Parameter-efficient fine-tuning methods have gained considerable popularity for adapting large-scale models to downstream tasks, particularly LoRA and its variants. Existing methods perform low-rank adaptation over the full parameter space. However, fine-tuning within a subspace can achieve comparable effectiveness. Inspired by the observation that pre-trained models possess non-trivial null spaces, we propose Null-space based Low-Rank Adaptation (Null-LoRA). Null-LoRA effectively reduces redundancy and enhances effective rank by freezing portions of the low-rank matrices. To further improve parameter efficiency, Null-LoRA constrains the entire incremental update within the null space, maximizing the utilization of incremental updates to adapt to new task paradigms. Null-LoRA surpasses the state of the art with fewer parameters in extensive experiments across image-text retrieval and visual question answering tasks.

Figures (1)

• Figure 1: An overview of our proposed Null-LoRA. Note that $Null(\cdot)$ donates the right null space of a matrix.