Patch-aware Batch Normalization for Improving Cross-domain Robustness

TL;DR

A novel method called patch-aware batch normalization (PBN), which first split feature maps of a batch into non-overlapping patches along the spatial dimension, and then independently normalize each patch to jointly optimize the shared BN parameter at each iteration.

Abstract

Despite the significant success of deep learning in computer vision tasks, cross-domain tasks still present a challenge in which the model's performance will degrade when the training set and the test set follow different distributions. Most existing methods employ adversarial learning or instance normalization for achieving data augmentation to solve this task. In contrast, considering that the batch normalization (BN) layer may not be robust for unseen domains and there exist the differences between local patches of an image, we propose a novel method called patch-aware batch normalization (PBN). To be specific, we first split feature maps of a batch into non-overlapping patches along the spatial dimension, and then independently normalize each patch to jointly optimize the shared BN parameter at each iteration. By exploiting the differences between local patches of an image, our proposed PBN can effectively enhance the robustness of the model's parameters. Besides, considering the statistics from each patch may be inaccurate due to their smaller size compared to the global feature maps, we incorporate the globally accumulated statistics with the statistics from each batch to obtain the final statistics for normalizing each patch. Since the proposed PBN can replace the typical BN, it can be integrated into most existing state-of-the-art methods. Extensive experiments and analysis demonstrate the effectiveness of our PBN in multiple computer vision tasks, including classification, object detection, instance retrieval, and semantic segmentation.

Figures (6)

• Figure 1: The illustration of the differences between local patches within an image. We randomly divide each image into four non-overlapping patches and calculate the statistics (mean and standard deviation) of all pixels within each patch. As shown, there are discrepancies between the patches in terms of both statistics and visualization. Motivated by this observation, we propose a novel method called patch-aware batch normalization (PBN) that leverages these differences to improve cross-domain robustness.
• Figure 2: Comparison between the typical batch normalization (BN) and our patch-aware batch normalization (PBN). These figures show the normalization operation for the $i$-th channel. In (b), we assume that feature maps of the $i$-th channel are randomly divided into 4 patches.
• Figure 3: The experimental results of our PBN and recently SOTA methods in the "CIFAR-10 $\to$ CIFAR-10-C" task. Note that "ERM" is the baseline, i.e., the raw Wide ResNet DBLP:conf/bmvc/ZagoruykoK16 with the cross-entropy loss.
• Figure 4: The analysis of the randomness along the channel dimension. In this figure, "P124-S2" denotes that we randomly select 2 digits from $\{1,2,4\}$ as the number of patches.
• Figure 5: The analysis of the hyper-parameter $\lambda$ in Eq. \ref{['eq07']} in (a). (b) is the result of using our method at different positions of the backbone. Both (a) and (b) are conducted on PACS.