Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA

TL;DR

This work tackles continual learning in vision-language models by reinterpreting a single LoRA as a decomposable Rank-1 Expert Pool and dynamically composing sparse, task-specific updates via a CLS-guided router. It introduces an Activation-Guided Orthogonal (AGO) loss to minimize inter-task interference by focusing orthogonality on historically important experts, enabling efficient, domain-aware learning with merged weights that incur no inference latency. Across 11 multi-domain tasks and the X-TAIL benchmark, the method achieves state-of-the-art results, reduces trainable parameters by up to 96.7%, and even surpasses the zero-shot CLIP upper bound in Transfer, all without external data or task-ID discriminators. The practical impact is a lightweight, scalable continual-learning framework for VLMs that preserves generalization while dramatically reducing computational burden.

Abstract

Continual learning (CL) in vision-language models (VLMs) faces significant challenges in improving task adaptation and avoiding catastrophic forgetting. Existing methods usually have heavy inference burden or rely on external knowledge, while Low-Rank Adaptation (LoRA) has shown potential in reducing these issues by enabling parameter-efficient tuning. However, considering directly using LoRA to alleviate the catastrophic forgetting problem is non-trivial, we introduce a novel framework that restructures a single LoRA module as a decomposable Rank-1 Expert Pool. Our method learns to dynamically compose a sparse, task-specific update by selecting from this expert pool, guided by the semantics of the [CLS] token. In addition, we propose an Activation-Guided Orthogonal (AGO) loss that orthogonalizes critical parts of LoRA weights across tasks. This sparse composition and orthogonalization enable fewer parameter updates, resulting in domain-aware learning while minimizing inter-task interference and maintaining downstream task performance. Extensive experiments across multiple settings demonstrate state-of-the-art results in all metrics, surpassing zero-shot upper bounds in generalization. Notably, it reduces trainable parameters by 96.7% compared to the baseline method, eliminating reliance on external datasets or task-ID discriminators. The merged LoRAs retain less weights and incur no inference latency, making our method computationally lightweight.

Figures (10)

• Figure 1: Advantages of our method over previous works: (a) ZSCL and GIFT introduce large external datasets or synthetic images from generative models for model regularization. (b) MoE-a and RAIL introduce additional components and utilize prior knowledge during testing. Our method uses NO external data knowledge and introduces NO additional burden. Meanwhile, it has fewer training parameters and lower GPU cost, with Transfer, a metric to measure the generalization ability of VLMs, exceeding the upper bound of the original CLIP zero-shot performance.
• Figure 2: The overall framework of our proposed method. LoRA is configured for all linear layers of the CLIP text and image encoder transformers, with the original parameters frozen. Each LoRA module is treated as a Rank-1 Expert Pool. For each input, the [CLS] token is extracted to guide a router that composes a sparse update by selecting critical experts from this pool. During training, an Activation-Guided Orthogonal loss is calculated between the current LoRA and those from past tasks. After training, the composed LoRA weights are merged back into the original model for zero-overhead inference.
• Figure 3: Comparison of the Average metric between setting no rank to zero and setting either one rank (a) or two adjacent ranks (b) to zero during the merging process. The rank is sorted in ascending order of Frobenius-norm: The contributions of each rank increase from left to right in sequence. The figure indicates that a higher contribution does not necessarily lead to performance improvement and shows ranks with low contribution may also play a crucial role in the task.
• Figure 4: Visualization of the activation frequencies of LoRA under different tasks. The darker the color, the higher frequency. "$\blacktriangle$" means disregarded experts. This figure shows frequency of the first LoRA layer in image encoder, indicating that the composed part of each task is exactly the experts with relatively higher activation.
• Figure 5: (a) Parameter collision rate comparison: Our method has fewer parameter collisions than vanilla orthogonal LoRA and non-orthogonal LoRA. (b) $\lambda$ hyperparameter exploration: Our method is robust to $\lambda$, outperforming vanilla orthogonal LoRA in all metrics.