InTAct: Interval-based Task Activation Consolidation for Continual Learning

TL;DR

The paper tackles representation drift in continual learning, especially under domain shifts, by introducing InTAct, a method that preserves past task functionality through activation-level regularization rather than parameter constraints or data replay. It builds activation interval representations in the form of cumulative hypercubes per layer and enforces a set of regularizers (IntDrift, Var, Align, Feat) to stabilize prior activations while allowing adaptation elsewhere. This approach directly targets the network’s functional evolution, enabling integration with state-of-the-art prompt-based continual learning frameworks to reduce drift and improve Average Accuracy by up to 8 percentage points on benchmarks like DomainNet and ImageNet-R without changing architecture or using past data. The method is architecture-agnostic, lightweight, and data-free with strong empirical gains across CIL and DIL settings, offering a principled balance between stability and plasticity and advancing robust, scalable continual learning in real-world, domain-diverse scenarios.

Abstract

Continual learning aims to enable neural networks to acquire new knowledge without forgetting previously learned information. While recent prompt-based methods perform strongly in class-incremental settings, they remain vulnerable under domain shifts, where the input distribution changes but the label space remains fixed. This exposes a persistent problem known as representation drift. Shared representations evolve in ways that overwrite previously useful features and cause forgetting even when prompts isolate task-specific parameters. To address this issue, we introduce InTAct, a method that preserves functional behavior in shared layers without freezing parameters or storing past data. InTAct captures the characteristic activation ranges associated with previously learned tasks and constrains updates to ensure the network remains consistent within these regions, while still allowing for flexible adaptation elsewhere. In doing so, InTAct stabilizes the functional role of important neurons rather than directly restricting parameter values. The approach is architecture-agnostic and integrates seamlessly into existing prompt-based continual learning frameworks. By regulating representation changes where past knowledge is encoded, InTAct achieves a principled balance between stability and plasticity. Across diverse domain-incremental benchmarks, including DomainNet and ImageNet-R, InTAct consistently reduces representation drift and improves performance, increasing Average Accuracy by up to 8 percentage points over state-of-the-art baselines.

Figures (7)

• Figure 1: During training on task $(N{+}1)$, InTAct preserves the functional region established by task $N$ (pink) while allowing the model to learn new knowledge in a distinct region (orange). After training, these regions merge into an expanded protected region (yellow), defining where the layer’s transformation should remain stable in future updates. The dashed curves illustrate transformations from earlier tasks, highlighting that their behaviors remain preserved within their respective regions, while adaptation is freely allowed outside them.
• Figure 2: Integration of InTAct with prompt-based methods. InTAct stabilizes features within the pretrained extractor and constrains activation changes in the classifier, mitigating representation drift across tasks.
• Figure 3: The model learns a Gaussian function in three sequential tasks. At each stage, InTAct constrains activations within previously established hypercubes, allowing new segments to be learned without overwriting prior knowledge.
• Figure 4: Illustration of how the InTAct loss components (Eq. 23) regulate representation updates between consecutive tasks. $\mathcal{L}_{\text{IntDrift}}$ and $\mathcal{L}_{\text{Feat}}$ preserve previously learned in-layer transformations within protected activation hypercubes (pink), maintaining consistency of past feature space while allowing adaptation in new regions. $\mathcal{L}_{\text{Var}}$ constrains the expansion of activation hypercubes so that new regions (orange) remain compact and balanced across tasks, preventing uncontrolled growth. $\mathcal{L}_{\text{Align}}$ enforces smooth transitions by aligning consecutive hypercubes and avoiding fragmentation of functional regions. Purple outlines denote the InTAct-regularized case, where the representation remains stable and well-aligned across tasks.
• Figure 5: Impact of the $\lambda_{\text{IntDrift}}$ hyperparameter on AA for the CODA-Prompt method. Results are averaged over 2 random seeds.