Group Orthogonal Low-Rank Adaptation for RGB-T Tracking

TL;DR

This work tackles parameter-efficient fine-tuning for RGB-T tracking by addressing rank-space redundancy in LoRA. It introduces Group Orthogonal Low-Rank Adaptation (GOLA), which partitions LoRA ranks via SVD, freezes crucial ranks to preserve pretrained priors, clusters redundant ranks into groups, and enforces inter-group orthogonality to promote complementary, non-overlapping feature learning across modalities. The approach yields state-of-the-art results on multiple RGB-T benchmarks while maintaining efficient inference, demonstrating that structured rank management and orthogonality constraints can significantly boost cross-modal tracking performance. GOLA offers a principled path toward more expressive, parameter-efficient multimodal tracking with practical implications for real-time deployment.

Abstract

Parameter-efficient fine-tuning has emerged as a promising paradigm in RGB-T tracking, enabling downstream task adaptation by freezing pretrained parameters and fine-tuning only a small set of parameters. This set forms a rank space made up of multiple individual ranks, whose expressiveness directly shapes the model's adaptability. However, quantitative analysis reveals low-rank adaptation exhibits significant redundancy in the rank space, with many ranks contributing almost no practical information. This hinders the model's ability to learn more diverse knowledge to address the various challenges in RGB-T tracking. To address this issue, we propose the Group Orthogonal Low-Rank Adaptation (GOLA) framework for RGB-T tracking, which effectively leverages the rank space through structured parameter learning. Specifically, we adopt a rank decomposition partitioning strategy utilizing singular value decomposition to quantify rank importance, freeze crucial ranks to preserve the pretrained priors, and cluster the redundant ranks into groups to prepare for subsequent orthogonal constraints. We further design an inter-group orthogonal constraint strategy. This constraint enforces orthogonality between rank groups, compelling them to learn complementary features that target diverse challenges, thereby alleviating information redundancy. Experimental results demonstrate that GOLA effectively reduces parameter redundancy and enhances feature representation capabilities, significantly outperforming state-of-the-art methods across four benchmark datasets and validating its effectiveness in RGB-T tracking tasks.

Figures (8)

• Figure 1: Comparison of rank importance score distribution between LoRA and our proposed GOLA. The rank space of LoRA exhibits significant redundancy, while our proposed GOLA effectively mitigates this issue.
• Figure 2: (a) Our proposed Group Orthogonal Low-Rank Adaptation (GOLA) framework. We decompose pretrained ranks into crucial ranks and redundant rank groups, freezing the crucial ranks to retain generalization. Our inter-group orthogonal constraint boosts the expressiveness of the redundant rank groups, fully utilizing the redundant rank space. (b) The overall architecture of our tracking framework. GOLA is applied to each linear layer of the backbone for fine-tuning.
• Figure 3: Comparison between GOLA-B with different trackers across various attributes in the LasHeR testing set.
• Figure 4: Impact of the number of crucial ranks and groups.
• Figure 5: Qualitative comparison of GOLA-B against 4 state-of-the-art trackers on 4 video sequences.