Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Progressive Conservative Adaptation for Evolving Target Domains

Gangming Zhao, Chaoqi Chen, Wenhao He, Chengwei Pan, Chaowei Fang, Jinpeng Li, Xilin Chen, Yizhou Yu

TL;DR

This work addresses evolving domain adaptation (EDA), where target distributions shift sequentially and historic target data are unavailable. It introduces PCAda, a meta-learning framework that combines Progressive Adaptive Prototypes for rapid classifier-head adaptation and Conservative Sparse Attention to limit interference with past feature knowledge. The inner loop updates rely on adaptive prototypes and pseudo-labels, while the outer loop enforces stable, domain-aware feature updates through a sparse attention mechanism, all guided by an evolving-domain objective that involves $d(\cdot,\cdot)$ and the target evolution rate $\alpha$. Empirical results on Rotated MNIST, Caltran, and Portraits demonstrate state-of-the-art performance and reduced forgetting across evolving targets, indicating strong practical potential for real-world continual domain shifts.

Abstract

Conventional domain adaptation typically transfers knowledge from a source domain to a stationary target domain. However, in many real-world cases, target data usually emerge sequentially and have continuously evolving distributions. Restoring and adapting to such target data results in escalating computational and resource consumption over time. Hence, it is vital to devise algorithms to address the evolving domain adaptation (EDA) problem, \emph{i.e.,} adapting models to evolving target domains without access to historic target domains. To achieve this goal, we propose a simple yet effective approach, termed progressive conservative adaptation (PCAda). To manage new target data that diverges from previous distributions, we fine-tune the classifier head based on the progressively updated class prototypes. Moreover, as adjusting to the most recent target domain can interfere with the features learned from previous target domains, we develop a conservative sparse attention mechanism. This mechanism restricts feature adaptation within essential dimensions, thus easing the inference related to historical knowledge. The proposed PCAda is implemented with a meta-learning framework, which achieves the fast adaptation of the classifier with the help of the progressively updated class prototypes in the inner loop and learns a generalized feature without severely interfering with the historic knowledge via the conservative sparse attention in the outer loop. Experiments on Rotated MNIST, Caltran, and Portraits datasets demonstrate the effectiveness of our method.

Progressive Conservative Adaptation for Evolving Target Domains

TL;DR

This work addresses evolving domain adaptation (EDA), where target distributions shift sequentially and historic target data are unavailable. It introduces PCAda, a meta-learning framework that combines Progressive Adaptive Prototypes for rapid classifier-head adaptation and Conservative Sparse Attention to limit interference with past feature knowledge. The inner loop updates rely on adaptive prototypes and pseudo-labels, while the outer loop enforces stable, domain-aware feature updates through a sparse attention mechanism, all guided by an evolving-domain objective that involves and the target evolution rate . Empirical results on Rotated MNIST, Caltran, and Portraits demonstrate state-of-the-art performance and reduced forgetting across evolving targets, indicating strong practical potential for real-world continual domain shifts.

Abstract

Conventional domain adaptation typically transfers knowledge from a source domain to a stationary target domain. However, in many real-world cases, target data usually emerge sequentially and have continuously evolving distributions. Restoring and adapting to such target data results in escalating computational and resource consumption over time. Hence, it is vital to devise algorithms to address the evolving domain adaptation (EDA) problem, \emph{i.e.,} adapting models to evolving target domains without access to historic target domains. To achieve this goal, we propose a simple yet effective approach, termed progressive conservative adaptation (PCAda). To manage new target data that diverges from previous distributions, we fine-tune the classifier head based on the progressively updated class prototypes. Moreover, as adjusting to the most recent target domain can interfere with the features learned from previous target domains, we develop a conservative sparse attention mechanism. This mechanism restricts feature adaptation within essential dimensions, thus easing the inference related to historical knowledge. The proposed PCAda is implemented with a meta-learning framework, which achieves the fast adaptation of the classifier with the help of the progressively updated class prototypes in the inner loop and learns a generalized feature without severely interfering with the historic knowledge via the conservative sparse attention in the outer loop. Experiments on Rotated MNIST, Caltran, and Portraits datasets demonstrate the effectiveness of our method.
Paper Structure (14 sections, 18 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 14 sections, 18 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Evolving Domain Adaptation in Meta-learning
  5. Progressive Adaptive Prototype in Inner Loop
  6. Conservative Sparse Attention in Outer Loop
  7. Experiment
  8. Datasets
  9. Experimental Settings
  10. Analysis and Results
  11. Baselines
  12. Results on Evolving Domain
  13. Ablation Study
  14. Conclusion

Figures (5)

  • Figure 1: The Evolving Domain Adaptation (EDA) paradigm. During meta-training, a small number of labeled source data and a portion of evolving unlabeled target data are accessible; During meta-testing, new target data evolve and the source is inaccessible. We expect the model to perform well on the online target data that comes continuously.
  • Figure 2: Overview of the PCAda. In the inner loop, the prototype-based replay updates the classification head; In the outer loop, the sparse attention mechanism helps the feature extractor adapt to the evolving domain. In meta-testing, we fix the encoder and update the classification head, decoder, prototype and feature extractor in the changing target domain.
  • Figure 3: Hyperparameter Sensitivity.
  • Figure 4: UMAP on Caltran. As the color changes, the target domain continues to evolve. The model is trained only in the source domain and JAN cannot capture the evolution of the target domain. EAML does not learn very well about the special properties of data in evolution, such as periodicity. From left to right, it shows the results from source, jan, eaml, and ours. As shown that our method is capable of continually learning evolving knowledge.
  • Figure 5: The Visualization of SAM.