Is Large-Scale Pretraining the Secret to Good Domain Generalization?

TL;DR

The paper questions whether improvements in multi-source Domain Generalization (DG) stem primarily from stronger pretraining rather than genuine learning of new features. It introduces the Alignment Hypothesis, positing that final DG performance is high if and only if pretraining yields strong alignment between image and class-text embeddings, formalized as $AlignmentScore(I,T) = \frac{\langle f_I(I), f_T(T) \rangle}{\|f_I(I)\| \cdot \|f_T(T)\|}$. Using five DomainBed datasets and an OpenCLIP-ViT-B/16 backbone, the authors create In-Pretraining (IP) and Out-of-Pretraining (OOP) evaluation splits to analyze DG methods under high- and low-alignment conditions. They find that DG methods excel on IP data but struggle on OOP data, with CLIPood performing best on IP and showing limited gains on OOP, while methods like MIRO+MPA provide notable OOP improvements, indicating a reliance on pretraining alignment. The work advocates rethinking DG benchmarking by incorporating IP/OOP splits to better measure generalization beyond pretraining alignment and motivates developing methods that learn transferable features beyond the backbone’s alignment.”

Abstract

Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results. However, it remains unclear if DG finetuning methods are becoming better over time, or if improved benchmark performance is simply an artifact of stronger pre-training. Prior studies have shown that perceptual similarity to pre-training data correlates with zero-shot performance, but we find the effect limited in the DG setting. Instead, we posit that having perceptually similar data in pretraining is not enough; and that it is how well these data were learned that determines performance. This leads us to introduce the Alignment Hypothesis, which states that the final DG performance will be high if and only if alignment of image and class label text embeddings is high. Our experiments confirm the Alignment Hypothesis is true, and we use it as an analysis tool of existing DG methods evaluated on DomainBed datasets by splitting evaluation data into In-pretraining (IP) and Out-of-pretraining (OOP). We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP. Put together, our findings highlight the need for DG methods which can generalize beyond pretraining alignment.

Figures (33)

• Figure 1: An overview of desired and actual behaviour of DG methods.1) DG methods are initialized with foundation models like CLIP. Pre-trained embeddings are relatively well aligned with ground truth labels on both source and target data for most samples (In-Pretraining, IP), but some samples are not well aligned (Out-of-pretraining, OOP). 2) An ideal DG method would strengthen alignment for both OOP and IP data with ground truth labels. 3) Our analysis shows that DG methods only result in strong alignment for IP data, leaving OOP data misaligned (Figure \ref{['fig:alignment_correlation']}).
• Figure 2: Comparing the Predictive Power of the Alignment and Image Similarity Hypotheses for Domain Generalization (DG).a.) Image Similarity Hypothesis: The cosine similarity between a test image and its closest match from the pre-training set (Perceptual Similarity Score) shows relatively weak predictive power for accuracy, implying that visual resemblance alone is not fully indicative of downstream performance. b.) Alignment Hypothesis: In contrast, the cosine similarity between image and ground truth text-label embedding after pre-training (Alignment Score) is highly predictive of model accuracy after fine-tuning on five DG datasets, with Alignment Score distributions shown in the colored histograms. This suggests that image-text pairs well-aligned during pre-training result in better performance on target tasks.
• Figure 3: Representative DomainBed dataset samples and their labels at various AlignmentScore values. At very low AlignmentScores, most labels (red boxes) are incorrect. At very high AlignmentScores, text present in the image corresponds to the label.
• Figure 4: Nearest neighbors of target images in pre-training LAION data.
• Figure 5: MMD between pre-training and target, source and target, and target and target for PACS. Target is more similar to pre-training than source. Despite this, Alignment is a better predictor of DG performance than perceptual similarity.