Domain Generalizable Continual Learning

TL;DR

This work introduces domain generalizable continual learning (DGCL), a setting where sequential tasks come from distinct domains yet must generalize across all encountered domains. To tackle DGCL, it proposes adaptive Domain Transformation (DoT), a plug-in based on pre-trained models that disentangles semantic- and domain-relevant information and adaptively transforms representations via an attention mechanism to align outputs across domains. DoT models both class- and domain-distribution information (e.g., Gaussian class means/covariances and domain prototypes) and optimizes with dual contrastive losses, while the DoT module can be discarded and reinitialized for new tasks to maintain efficiency. Empirically, DoT substantially improves state-of-the-art CL baselines under DGCL across Office-Home, DigitsDG, CORe50, and DomainNet, achieving strong unseen-domain generalization with lightweight parameter overhead, and demonstrating the method’s practical potential for robust, domain-aware continual learning in real-world settings.

Abstract

To adapt effectively to dynamic real-world environments, intelligent systems must continually acquire new skills while generalizing them to diverse, unseen scenarios. Here, we introduce a novel and realistic setting named domain generalizable continual learning (DGCL): a model learns sequential tasks with each involving a single domain, aiming to perform well across all encountered tasks and domains. This setting poses unique challenges in acquiring, retaining, and leveraging both semantic- and domain-relevant information for robust generalization. Although state-of-the-art continual learning (CL) methods have employed pre-trained models (PTMs) to enhance task-specific generalization, they typically assume identical training and testing domains for each task and therefore perform poorly in DGCL. To this end, we propose adaptive Domain Transformation (DoT), an innovative PTMs-based approach tailored to DGCL. Inspired by the distributed-plus-hub theory of the human brain, DoT disentangles semantic- and domain-relevant information in representation learning, and adaptively transforms task representations across various domains for output alignment, ensuring balanced and generalized predictions. DoT serves as a plug-in strategy that greatly facilitates state-of-the-art CL baselines under both full parameter tuning and parameter-efficient tuning paradigms in DGCL, validated by extensive experiments. Also, DoT is shown to accumulate domain-generalizable knowledge from DGCL, and ensure resource efficiency with a lightweight implementation.

Figures (7)

• Figure 1: Motivation of DGCL and DoT. a, The human brain naturally supports DGCL through distributed-plus-hub of memory attributes to reconstruct task-dependent representations kudithipudi2022biologicalpatterson2007you. b, Inspired by the human brain, we propose to address DGCL by coordinating semantic- and domain-relevant information in task-dependent representations. Best viewed in color.
• Figure 2: Comparison of DGCL with representative CL settings, such as TIL, CIL, DIL, and VIL. DGCL is the only setting with training and test sets of each task belonging to different domains.
• Figure 3: Empirical investigation of DGCL with state-of-the-art CL baselines. a-d, We report the average accuracy of all encountered tasks and domains under CIL and DGCL settings (i.e., average all-domain accuracy $A_{\text{all}}$). e-h, We further report the average accuracy of tasks with seen domains (i.e., average in-domain accuracy $A_{\text{in}}$) and tasks with unseen domains (i.e., average out-domain accuracy $A_{\text{out}}$). Please refer to Sec. \ref{['sec.exp_setup']} for implementation details. Methods in light gray focus on instructing $f_\theta (\cdot)$, while those covered in light blue focus on rectifying $h_\psi (\cdot)$ with or without instructing $f_\theta (\cdot)$. DP, DualPrompt. SP, S-Prompt. CODA, CODA-Prompt.
• Figure 4: Layer-wise feature visualization using t-SNE. We extract the layer-wise features $\boldsymbol{r}^{(l)}$ of a ViT-B/16 backbone on the test set of Split DigitsDG, which has been trained using SLCA for all incremental tasks. a-d, Points are colored with their class labels. e-h, Points are colored with their domain identities. Best viewed in color.
• Figure 5: Illustration of the proposed framework for DGCL. In step 1, the entire model is trained with incremental tasks, where the feature distributions $\mathcal{H}_c$ and $\mathcal{P}_d$ are preserved. In step 2, the DoT module is trained to generate pseudo feature based on sampled features. In step 3, the output layer $h_\psi$ is aligned with both sampled features and pseudo features produced by the DoT module.