Improving Out-of-Domain Robustness with Targeted Augmentation in Frequency and Pixel Spaces

TL;DR

The paper tackles out-of-domain robustness under domain adaptation, where labeled source data and unlabeled target data come from different distributions. It introduces Frequency-Pixel Connect, a dataset-agnostic augmentation that jointly perturbs frequency-space amplitudes and pixel content to simulate domain shifts while preserving label-relevant semantics, followed by a two-stage training regime: contrastive pretraining on a mixed unlabeled distribution and LP-FT fine-tuning on labeled data. Across four real-world benchmarks spanning vision, medical, audio, and astronomy (iWildCam, Camelyon17, BirdCalls, Galaxy10), the method achieves superior OOD performance compared to generic and dataset-specific augmentations, and analyses of connectivity corroborate its effectiveness in aligning cross-domain semantics while perturbing spurious domain cues. While the approach reduces reliance on domain-specific priors, it requires tuning of mixing ratios; future directions include adaptive parameter learning and integration with self-supervised objectives to further boost OOD generalization.

Abstract

Out-of-domain (OOD) robustness under domain adaptation settings, where labeled source data and unlabeled target data come from different distributions, is a key challenge in real-world applications. A common approach to improving OOD robustness is through data augmentations. However, in real-world scenarios, models trained with generic augmentations can only improve marginally when generalized under distribution shifts toward unlabeled target domains. While dataset-specific targeted augmentations can address this issue, they typically require expert knowledge and extensive prior data analysis to identify the nature of the datasets and domain shift. To address these challenges, we propose Frequency-Pixel Connect, a domain-adaptation framework that enhances OOD robustness by introducing a targeted augmentation in both the frequency space and pixel space. Specifically, we mix the amplitude spectrum and pixel content of a source image and a target image to generate augmented samples that introduce domain diversity while preserving the semantic structure of the source image. Unlike previous targeted augmentation methods that are both dataset-specific and limited to the pixel space, Frequency-Pixel Connect is dataset-agnostic, enabling broader and more flexible applicability beyond natural image datasets. We further analyze the effectiveness of Frequency-Pixel Connect by evaluating the performance of our method connecting same-class cross-domain samples while separating different-class examples. We demonstrate that Frequency-Pixel Connect significantly improves cross-domain connectivity and outperforms previous generic methods on four diverse real-world benchmarks across vision, medical, audio, and astronomical domains, and it also outperforms other dataset-specific targeted augmentation methods.

Figures (5)

• Figure 1: Feature decomposition and augmentation examples.Top: For each dataset (iWildCam, Camelyon17, BirdCalls, Galaxy10), we show a source image, its corresponding augmented image generated by our method. Bottom: Based on the decomposition framework shen2022connect, we annotate representative features in the datasets across $x_{\text{obj}}$, $x_{d\text{:robust}}$, $x_{d\text{:spu}}$, $x_{\text{noise}}$. Our method effectively randomizes $x_{d\text{:spu}}$, varies $x_{d\text{:robust}}$ while preserving or $x_{\text{obj}}$.
• Figure 2: The figure shows the amplitude-only and phase-only reconstructions of the two input images, and the process where the amplitudes of Image 1 and Image 2 are first mixed, and the mixed amplitude is combined with the phase of Image 2 to generate an augmentation of Image 2.
• Figure 3: Overview of our Frequency-Pixel Mixing strategy. Given two images, we first do amplitude interpolation in the frequency space and pixel-space blending with mixing ratio $\lambda_1$. The final augmented image is obtained by combining the two intermediate results with a mixing ratio $\lambda_2$.
• Figure 4: We plot the in-domain (ID) performance versus out-of-domain (OOD) performance for all methods across four datasets. Our method consistently outperforms all baselines in OOD generalization. 'CL' refers to 'Connect Later'.
• Figure 5: Effect of Mixing Ratios on ID and OOD Performance.