Collaborative Low-Rank Adaptation for Pre-Trained Vision Transformers

TL;DR

CLoRA tackles the challenge of efficiently fine-tuning pre-trained Vision Transformers by increasing the effective capacity of low-rank adaptation without inflating trainable parameters. It achieves this with two mechanisms: base-space sharing, which constructs all LRMs from a common set of base down/up-projection spaces, and SADE, which regularizes the learned projections to promote diverse representations. The resulting method consistently improves accuracy across VTAB-1K, FGVC, and larger ViT backbones while reducing GFLOPs and parameter counts, and it extends effectively to point-cloud transformers for 3D tasks. This approach enables scalable, storage-efficient fine-tuning suitable for data-scarce domains and large-model deployment.

Abstract

Low-rank adaptation (LoRA) has achieved remarkable success in fine-tuning pre-trained vision transformers for various downstream tasks. Existing studies mainly focus on exploring more parameter-efficient strategies or more effective representation learning schemes. However, these methods either sacrifice fine-tuning performance or introduce excessive trainable parameters, failing to strike a balance between learning performance and parameter efficiency. To address this problem, we propose a novel tuning method named collaborative low-rank adaptation (CLoRA) in this paper. CLoRA consists of base-space sharing and sample-agnostic diversity enhancement (SADE) components. To maintain parameter efficiency while expanding the learning capacity of low-rank modules (LRMs), base-space sharing allows all LRMs to share a set of down/up-projection spaces. In CLoRA, the low-rank matrices obtained from the shared spaces collaboratively construct each LRM. Since the representations extracted by these matrices may contain redundant information, SADE is employed to regularize the similarities among them to encourage diverse representations in the training process. We conduct extensive experiments on widely used image and point cloud datasets to evaluate the performance of CLoRA. Experimental results demonstrate that CLoRA strikes a better balance between learning performance and parameter efficiency, while requiring the fewest GFLOPs for point cloud analysis, compared with the state-of-the-art methods.

Figures (3)

• Figure 1: Low-rank structures of (a) LoRA, (b) ARC, and (c) CLoRA.
• Figure 2: Illustration of CLoRA. In LoRA, LRMs are applied to adapt the input representations of MHA and FFN blocks, yielding $\{ \Delta {{\bf{W}}_i} = {{\bf{A}}_i}{{\bf{B}}_i}\} _{i = 1}^m$, where $\{ {{\bf{A}}_i}{{\bf{B}}_i}\} _{i = 1}^m$ are independent of each other. By contrast, each LRM in CLoRA effectively and efficiently utilizes the information contained in $\{ {{\bf{A}}_i}{{\bf{B}}_i}\} _{i = 1}^m$ through the base-space sharing and SADE components.
• Figure 3: Performance verification of the two components in CLoRA.