SemAlign: Language Guided Semi-supervised Domain Generalization

TL;DR

This work tackles semi-supervised domain generalization (SSDG) by moving beyond sole reliance on pseudo-labeling accuracy and toward maximizing data utilization. It introduces SemAlign, which aligns intermediate features with the semantically rich space of a Vision-Language Model (via CLIP-based class prototypes) to promote domain-invariant representations, complemented by entropy-based and adaptive negative learning losses, plus targeted Fourier and texture-based augmentations. Key contributions include a residual feature refinement mechanism, multi-term feature and output space losses, and data-augmented training that leverages all unlabeled data, validated on four benchmarks with state-of-the-art results and evidence of improved Effective Data Utilization ($EDU$). The approach offers a practical, label-efficient pathway to robust domain generalization in diverse visual domains, with future work aimed at relaxing the need for meaningful class labels.

Abstract

Semi-supervised Domain Generalization (SSDG) addresses the challenge of generalizing to unseen target domains with limited labeled data. Existing SSDG methods highlight the importance of achieving high pseudo-labeling (PL) accuracy and preventing model overfitting as the main challenges in SSDG. In this light, we show that the SSDG literature's excessive focus on PL accuracy, without consideration for maximum data utilization during training, limits potential performance improvements. We propose a novel approach to the SSDG problem by aligning the intermediate features of our model with the semantically rich and generalized feature space of a Vision Language Model (VLM) in a way that promotes domain-invariance. The above approach is enhanced with effective image-level augmentation and output-level regularization strategies to improve data utilization and minimize overfitting. Extensive experimentation across four benchmarks against existing SSDG baselines suggests that our method achieves SOTA results both qualitatively and quantitatively. The code will be made publicly available.

Figures (6)

• Figure 1: A conceptual diagram illustrating the proposed domain-invariant semantic alignment approach, where features are aligned with semantically rich representations from the text encoder of a VLM to achieve semi-supervised domain generalization.
• Figure 2: Left: PL accuracy after thresholding, Right: The correct pseudo-labels retained after thresholding as a proportion of the entire dataset (EDU) of the baselines FixMatch fixmatch, FBC-SA FBCSA, DGWM dgwm, StyleMatch stylematch, and ours. Our method achieves significant gains in proportion of samples retained after thresholding while maintaining comparable PL accuracy to SOTA SSL-based-SSDG methods. Here A, C, P, and S denote Art-painting, Cartoon, Photos, and Sketch domains, respectively.
• Figure 3: For class labels of the classification problem we generate domain invariant representations using CLIP CLIP text encoder and further refine them using the RFR module to create the class prototype matrix ($\mathbf{K}^*$). The orthogonality of these class prototypes is encouraged using $\mathcal{L}_\mathrm{orth}$. For a weakly augmented view, we align the feature extractor output with the corresponding class prototype guided by the PL generated by the classifier. Alignment is constrained using $\mathcal{L}_\mathrm{SA}$ and $\mathcal{L}_\mathrm{con}$. To reduce overfitting we use a stochastic classifier. At the classifier level, we minimize competition from competitive classes using $\mathcal{L}_\mathrm{EML}$ and allow learning from rejected samples at the pseudo-labeling using $\mathcal{L}_\mathrm{ANL}$. $\mathcal{L}_\mathrm{s}$ and $\mathcal{L}_\mathrm{u}$ are same as in FixMatch fixmatch.
• Figure 4: Our augmentation methods. (a) Raw image, (b) Phase-only image reconstruction, (c) Amplitude swapping, (d) Texture reduction. We apply these augmentations randomly along-side FixMatch fixmatch augmentations as the strong augmentation.
• Figure 5: (a) t-SNE visualization of the feature space for PACS dataset (10 labels per class). The average log Fisher Discriminant Ratio (FDR) ($\pm$1 standard deviation) was calculated over 5 random seeds, over all 4 domains. (b) Similarity matrices for the class prototype embeddings of the PACS dataset, before and after adding the RFR module. RFR reduces the similarity between class prototypes while preserving the semantics.