Incremental Open-set Domain Adaptation

TL;DR

This work proposes IOSDA-Net, a two-stage learning pipeline, which replicates prior domains from random noise using a generative framework and creates a pseudo source domain that is adapted to the present target domain.

Abstract

Catastrophic forgetting makes neural network models unstable when learning visual domains consecutively. The neural network model drifts to catastrophic forgetting-induced low performance of previously learnt domains when training with new domains. We illuminate this current neural network model weakness and develop a forgetting-resistant incremental learning strategy. Here, we propose a new unsupervised incremental open-set domain adaptation (IOSDA) issue for image classification. Open-set domain adaptation adds complexity to the incremental domain adaptation issue since each target domain has more classes than the Source domain. In IOSDA, the model learns training with domain streams phase by phase in incremented time. Inference uses test data from all target domains without revealing their identities. We proposed IOSDA-Net, a two-stage learning pipeline, to solve the problem. The first module replicates prior domains from random noise using a generative framework and creates a pseudo source domain. In the second step, this pseudo source is adapted to the present target domain. We test our model on Office-Home, DomainNet, and UPRN-RSDA, a newly curated optical remote sensing dataset.

Figures (10)

• Figure 1: Illustrating the flow of domain data across different time-stamps, IOSDA performs lifelong open-set domain adaptive classification tasks. At each time-stamp, IOSDA encounters a single previously unseen domain and retains model parameters from previous time-stamps. IOSDA classifies closed-set object classes based on their known classes and identifies unseen classes as open-set. During inference, data can originate from any previously encountered domains up to the current time-stamp.
• Figure 2: The diagram above shows the entire process flow. One domain $D_1(x,y)$ is available at the initial timestamp. The $MDCGAN_0$ is trained with the original samples of $D_1(x,y)$ to preserve this domain. Next timestamp, a new target domain appears, but the original source domain $D_1(x,y)$ is not available. Instead, $MDCGAN_0$ generates source samples and labels ($G_1/L_1$). At that moment, $MEOSDA_0$ does domain adaptation training using the pseudo source ($G_1/L_1$) as Source domain and original target domain samples ($D_2(x)$). After $MEOSDA_0$ training, target domain samples ($D_2(x)$) were pseudo-labeled. A new $MDCGAN_1$ was trained with generated source and Target domain pseudo labels before this timestamp disappeared. At the next time stamp ($Timestamp_2$), $MDCGAN_1$ generates sources $G_1/L_1$, $G_2/L_2$, which depict as multiple sources ($D_1(x)$, $D_2(x)$ respectively) , and the newly arrived target domain ($D_3(x)$) goes through $MEOSDA_1$ to perform multi-source domain adaptation and then trains a new $MDCGAN_2$ with the generated sources and the target domain with pseudo labels. No raw samples of prior domains are needed; just the $MDCGAN$ learned in the immediate previous timestamp is saved.
• Figure 3: Diagram of Multi-Domain and Class-guided Generative Adversarial Network. Novel MDCGAN has a generator, discriminator, and three output branches (label classifier, domain classifier, and domain discriminator). The discriminator uses the generator's output and the original samples' class and domain labels, while the generator uses random noise. For training, domain discriminator (real/fake) utilises adversarial loss and label, and domain classifier uses cross entropy loss.
• Figure 4: The MEOSDA diagram illustrates the architecture at timestamp 4, featuring $Domain_4$ as the unsupervised target domain and $Domain_1$, $Domain_2$, $Domain_3$ as source domains. In our incremental setup, we employ feature replay to replicate original data for pseudo sources. MEOSDA utilizes a shared feature extractor across all domains, followed by separate classifier branches for each source domain. Each classifier outputs $k+1$ classes, where the first $k$ represent closed classes and the $k+1$th represents open set classes.
• Figure 5: Office home