Dual-stream Feature Augmentation for Domain Generalization

TL;DR

This work proposes a Dual-stream Feature Augmentation (DFA) method by constructing some hard features from two perspectives that are integrated into a unified learnable feature disentangle model to improve the transferability and robustness of the model.

Abstract

Domain generalization (DG) task aims to learn a robust model from source domains that could handle the out-of-distribution (OOD) issue. In order to improve the generalization ability of the model in unseen domains, increasing the diversity of training samples is an effective solution. However, existing augmentation approaches always have some limitations. On the one hand, the augmentation manner in most DG methods is not enough as the model may not see the perturbed features in approximate the worst case due to the randomness, thus the transferability in features could not be fully explored. On the other hand, the causality in discriminative features is not involved in these methods, which harms the generalization ability of model due to the spurious correlations. To address these issues, we propose a Dual-stream Feature Augmentation~(DFA) method by constructing some hard features from two perspectives. Firstly, to improve the transferability, we construct some targeted features with domain related augmentation manner. Through the guidance of uncertainty, some hard cross-domain fictitious features are generated to simulate domain shift. Secondly, to take the causality into consideration, the spurious correlated non-causal information is disentangled by an adversarial mask, then the more discriminative features can be extracted through these hard causal related information. Different from previous fixed synthesizing strategy, the two augmentations are integrated into a unified learnable feature disentangle model. Based on these hard features, contrastive learning is employed to keep the semantic consistency and improve the robustness of the model. Extensive experiments on several datasets demonstrated that our approach could achieve state-of-the-art performance for domain generalization. Our code is available at: https://github.com/alusi123/DFA.

Figures (20)

• Figure 1: $X_{1}$ and $X_{2}$ represent two different domains, and $Y$ represents the label space shared by both source domains. (a) The dashed areas represent the mathematical statistical relationship between each domain and labels. Obviously, the areas not only include the shared part, but also contain the specific parts. (b) The dashed area represents the domain invariant information across multiple source domains. However, the spurious correlation information still exists in it. (c) Our motivation is to learn domain invariant features that have causal relationships with the labels.
• Figure 2: Diagram of our feature augmentation. For domain related augmentation, the domain-specific information with the most abundant style attributes is selected to construct hard features. For causal related augmentation, the most correlated non-causal information within the most similar class is selected to construct hard features.
• Figure 3: The framework of DFA. We first generate domain-invariant features and domain-specific features by dual-path feature disentangle module, and employ adversarial mask module to disentangle spurious correlated non-causal information from domain-invariant features. We combine superior features with domain-specific information and non-causal inferior information by special strategy respectively to achieve dual-stream feature augmentation. At last, Contrastive Learning (CL) is adopted to the augmented features and domain-invariant features. The dashed lines denote that the gradient is detached.
• Figure 4: Visualization of attention maps of the last convolutional layer for our baseline and DFA. We use ResNet18 as the backbone and train on the PACS dataset, with Art Painting serving as the target domain.
• Figure 7: The confusion matrix of baseline and DFA. Each color represents a target domain, ordered from left to right as follows: Art, Cartoon, Photo, Sketch. The top row is the baseline, and the bottom row is our DFA.