Source-Free Cross-Domain Continual Learning

TL;DR

This work tackles the problem of source-free cross-domain continual learning (SFCDCL), where labeled samples from the source domain are unavailable during continual adaptation. It introduces REFEREE, a dual-branch framework that couples a source-pretrained backbone with a frozen vision-language model (CLIP) to promote domain-invariant and domain-specific representations without source data. Key components include frequency-aware prompting to suppress high-frequency noise, an uncertainty-weighting scheme to down-weight unreliable pseudo-labels, and a gradient-free kernel discriminant analysis (KLDA) based on random Fourier features to mitigate double forgetting. Extensive experiments on 21 cross-domain tasks across VisDA, Office-31, Office-Home, and DomainNet demonstrate that REFEREE achieves significant gains over methods with access to source-domain samples, highlighting its practical impact for privacy-preserving continual learning. The approach offers a scalable, black-box-compatible solution that leverages foundation models to enable robust cross-domain adaptation without compromising data privacy.

Abstract

Although existing cross-domain continual learning approaches successfully address many streaming tasks having domain shifts, they call for a fully labeled source domain hindering their feasibility in the privacy constrained environments. This paper goes one step ahead with the problem of source-free cross-domain continual learning where the use of source-domain samples are completely prohibited. We propose the idea of rehearsal-free frequency-aware dynamic prompt collaborations (REFEREE) to cope with the absence of labeled source-domain samples in realm of cross-domain continual learning. REFEREE is built upon a synergy between a source-pre-trained model and a large-scale vision-language model, thus overcoming the problem of sub-optimal generalizations when relying only on a source pre-trained model. The domain shift problem between the source domain and the target domain is handled by a frequency-aware prompting technique encouraging low-frequency components while suppressing high-frequency components. This strategy generates frequency-aware augmented samples, robust against noisy pseudo labels. The noisy pseudo-label problem is further addressed with the uncertainty-aware weighting strategy where the mean and covariance matrix are weighted by prediction uncertainties, thus mitigating the adverse effects of the noisy pseudo label. Besides, the issue of catastrophic forgetting (CF) is overcome by kernel linear discriminant analysis (KLDA) where the backbone network is frozen while the classification is performed using the linear discriminant analysis approach guided by the random kernel method. Our rigorous numerical studies confirm the advantage of our approach where it beats prior arts having access to source domain samples with significant margins.

Figures (5)

• Figure 1: Source-Free Cross-Domain Continual Learning Problems: a model is presented with sequences of labeled learning tasks in the source domain and unlabeled learning tasks in the target domain where there exists the domain shift problem between the source domain and the target domain. A model is tasked to generalize well to the unlabeled learning task of the target domain without using any labeled learning tasks of the source domain for domain adaptations.
• Figure 2: REFEREE architecture. A dual-branch framework couples a frozen source-pretrained encoder (ViT) with a frozen vision--language model (CLIP). For each target image, both branches produce scores that are fused $\hat{p}=\alpha\,p+\beta\,p_{\text{VLM}}$ to obtain pseudo-labels and entropies. A frequency-aware prompting module (DWT) suppresses high-frequency components and augments low-frequency content. The fused confidence defines uncertainty weights $\omega\!\in\![0,1]$ that weight the class means $\mu$ and shared covariance $\Sigma$ for KLDA over random Fourier features; these statistics update the target classifier across tasks while the backbones remain frozen. No source-domain samples are used during target adaptation.
• Figure 3: Average accuracy (%) comparison across different baselines for four datasets: VisDA, Office-31, Office-Home, and DomainNet
• Figure 4: Sensitivity to transform dimension $D$. Performance saturates from $D\!\approx\!500$ onward.
• Figure 5: Sensitivity to kernel width $\sigma$ in KLDA. Accuracy is flat across four orders of magnitude.