Domain-Aware Continual Zero-Shot Learning

TL;DR

A Domain-Invariant Network (DIN) is proposed to learn factorized features for shifting domains and improved textual representation for unseen classes and significantly outperforms existing baselines by over 5% in harmonic accuracy and over 1% in backward transfer and achieves a new SoTA.

Abstract

Modern visual systems have a wide range of potential applications in vision tasks for natural science research, such as aiding in species discovery, monitoring animals in the wild, and so on. However, real-world vision tasks may experience changes in environmental conditions, leading to shifts in how captured images are presented. To address this issue, we introduce Domain-Aware Continual Zero-Shot Learning (DACZSL), a task to recognize images of unseen categories in continuously changing domains. Accordingly, we propose a Domain-Invariant Network (DIN) to learn factorized features for shifting domains and improved textual representation for unseen classes. DIN continually learns a global shared network for domain-invariant and task-invariant features, and per-task private networks for task-specific features. Furthermore, we enhance the dual network with class-wise learnable prompts to improve class-level text representation, thereby improving zero-shot prediction of future unseen classes. To evaluate DACZSL, we introduce two benchmarks, DomainNet-CZSL and iWildCam-CZSL. Our results show that DIN significantly outperforms existing baselines by over 5% in harmonic accuracy and over 1% in backward transfer and achieves a new SoTA.

Figures (5)

• Figure 1: DACZSL Setting. Wild images undergo domain shifts due to various factors such as location, time, camera positions, and so on. In DACZSL, at each time $t$, images of new classes are revealed to train the model, and the model is then evaluated on new domains containing both seen classes, i.e., classes revealed up to time $t$, and unseen classes, i.e., classes to be revealed after time $t$.
• Figure 2: DIN involves two key steps. First, we concatenate class-wise learnable prompts to each class token embedding. These prompts are learned by minimizing the contrastive loss between the text feature $t(\texttt{CLASS})$ and the global visual feature $z_G$ from the global network $G$. Next, we employ adversarial training to refine the global feature $z_G$, and implement disentanglement techniques for the local feature $z_t$. This approach ensures that the global network encodes domain- and task-invariant information, while the local network processes task-specific information.
• Figure 3: Results on the iWildCam-CZSL dataset demonstrate that our method, DIN, achieves significantly higher harmonic mean accuracy compared to other methods.
• Figure 4: t-SNE visualization of latent features from the global network after training on DomainNet-CZSL task 1, highlighting 5 classes.
• Figure 7: Comparison with adaptive continual learning methods of noise-reduced DomainNet. + Tf means we use CLIP radford2021learning pre-trained text Transformer. We report the average over all possible target domains.