Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models

Yujin Jo, Taesup Kim

TL;DR

The paper tackles continual learning for vision-language foundation models without memory buffers or parameter growth. It introduces NuSA-CL, which identifies an intrinsic null space of the current weights via SVD and learns a low-rank update confined to that null space, merging the update into the backbone with W_t = W_{t-1} + \Delta W_t and \Delta W_t = U_n M V_n^\top. This persistent, memory-free constraint preserves zero-shot capabilities while enabling sequential adaptation, achieving a favorable efficiency-performance trade-off and strong data-efficiency across MTIL and CIFAR100 benchmarks. Theoretical analysis provides interference bounds and forgetting control, and empirical results demonstrate robust long-sequence scalability, making NuSA-CL practical for real-world, resource-constrained deployment of zero-shot VLMs. Overall, NuSA-CL offers a scalable, fixed-budget solution for continual learning in multimodal models, balancing stability and plasticity through intrinsic null-space updates that accumulate knowledge without overwriting core capabilities.

Abstract

Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalization, enabling deployment in a wide range of real-world tasks without additional task-specific training. However, in real deployment scenarios with evolving environments or emerging classes, these models inevitably face distributional shifts and novel tasks. In such contexts, static zero-shot capabilities are insufficient, and there is a growing need for continual learning methods that allow models to adapt over time while avoiding catastrophic forgetting. We introduce NuSA-CL (Null Space Adaptation for Continual Learning), a lightweight memory-free continual learning framework designed to address this challenge. NuSA-CL employs low-rank adaptation and constrains task-specific weight updates to lie within an approximate null space of the model's current parameters. This strategy minimizes interference with previously acquired knowledge, effectively preserving the zero-shot capabilities of the original model. Unlike methods relying on replay buffers or costly distillation, NuSA-CL imposes minimal computational and memory overhead, making it practical for deployment in resource-constrained, real-world continual learning environments. Experiments show that our framework not only effectively preserves zero-shot transfer capabilities but also achieves highly competitive performance on continual learning benchmarks. These results position NuSA-CL as a practical and scalable solution for continually evolving zero-shot VLMs in real-world applications.

Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models

TL;DR

The paper tackles continual learning for vision-language foundation models without memory buffers or parameter growth. It introduces NuSA-CL, which identifies an intrinsic null space of the current weights via SVD and learns a low-rank update confined to that null space, merging the update into the backbone with W_t = W_{t-1} + \Delta W_t and \Delta W_t = U_n M V_n^\top. This persistent, memory-free constraint preserves zero-shot capabilities while enabling sequential adaptation, achieving a favorable efficiency-performance trade-off and strong data-efficiency across MTIL and CIFAR100 benchmarks. Theoretical analysis provides interference bounds and forgetting control, and empirical results demonstrate robust long-sequence scalability, making NuSA-CL practical for real-world, resource-constrained deployment of zero-shot VLMs. Overall, NuSA-CL offers a scalable, fixed-budget solution for continual learning in multimodal models, balancing stability and plasticity through intrinsic null-space updates that accumulate knowledge without overwriting core capabilities.

Abstract

Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalization, enabling deployment in a wide range of real-world tasks without additional task-specific training. However, in real deployment scenarios with evolving environments or emerging classes, these models inevitably face distributional shifts and novel tasks. In such contexts, static zero-shot capabilities are insufficient, and there is a growing need for continual learning methods that allow models to adapt over time while avoiding catastrophic forgetting. We introduce NuSA-CL (Null Space Adaptation for Continual Learning), a lightweight memory-free continual learning framework designed to address this challenge. NuSA-CL employs low-rank adaptation and constrains task-specific weight updates to lie within an approximate null space of the model's current parameters. This strategy minimizes interference with previously acquired knowledge, effectively preserving the zero-shot capabilities of the original model. Unlike methods relying on replay buffers or costly distillation, NuSA-CL imposes minimal computational and memory overhead, making it practical for deployment in resource-constrained, real-world continual learning environments. Experiments show that our framework not only effectively preserves zero-shot transfer capabilities but also achieves highly competitive performance on continual learning benchmarks. These results position NuSA-CL as a practical and scalable solution for continually evolving zero-shot VLMs in real-world applications.
Paper Structure (39 sections, 2 theorems, 6 equations, 3 figures, 10 tables)

This paper contains 39 sections, 2 theorems, 6 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Continual Learning with Parameter-Efficient Fine-Tuning
  4. Orthogonal Projection and Null Space Approaches
  5. SVD-Guided Adaptation in Foundation Models
  6. Method: Null Space Adaptation for Continual Learning
  7. Identifying The Intrinsic Null Space
  8. Constrained Adaptation within the Null Space
  9. Continual Adaptation via Update Merging
  10. Theoretical Motivation
  11. Interference Bound for a Single Update
  12. Forgetting Control in Continual Learning
  13. Experiments
  14. Experimental Setup
  15. Benchmarks.
  16. ...and 24 more sections

Key Result

Lemma 1

Let $W = U\Sigma V^\top$ be the SVD of a weight matrix, and let $\Delta W = U_n M V_n^\top$ be an update restricted to its intrinsic null space. The interference in parameter space, measured by the Frobenius inner product, is bounded by: where $\sigma_{\text{max}}^{\text{null}} := \sigma_{k+1}$ is the largest singular value within the null space.

Figures (3)

  • Figure 1: The NuSA-CL framework. Starting with the weights from the previous task $W_{t-1}$, we first perform SVD to identify the intrinsic null space. A new low-rank update $\Delta W_t$ is then learned under a persistent constraint that confines it to this space. Finally, the update is merged to produce the new weights $W_t \leftarrow W_{t-1} + \Delta W$, completing the cycle.
  • Figure 2: Our method actively accumulates knowledge by utilizing the model's intrinsic capacity, while other methods overwrite it. The plots track the evolution of the effective rank (solid lines, left axis) and the null ratio (dashed lines, right axis) for the text and vision encoders. NuSA-CL is the only method to show a steady increase in effective rank, demonstrating that it fills the underutilized null space to learn new tasks. This provides direct evidence for its ability to mitigate catastrophic forgetting.
  • Figure 3: Ablations on subspace and rank.(Left) Subspace selection. Across ranks, Tail (null-like) consistently yields the lowest forgetting than Top and Random). (Right) Update rank. On the Tail subspace, overall CL performance peaks around $r_{\max}{=}128$ (cf. Transfer/Avg./Last), indicating that a moderate rank balances retention (stability) and on-task adaptation (plasticity).

Theorems & Definitions (3)

  • Lemma 1: Bounded Interference via Null Space Constraint
  • proof
  • Theorem 2: Cumulative Interference Bound