Source-Free Domain Adaptation by Optimizing Batch-Wise Cosine Similarity

TL;DR

This paper tackles Source-Free Domain Adaptation (SFDA) for image classification, where only a pretrained source model and unlabeled target data are available. It introduces neighborhood signature, a robust representation of target samples' local structure, and a single loss term that aligns predictions with class encodings across a batch using cosine-similarity, while integrating intra-class diversity, confidence inertia, and class-imbalance scaling. The class encoding for a sample is chosen from either its neighborhood signature or its own prediction based on lower entropy, and the final loss combines multiple cues through a mask to mitigate noisy neighbors. Empirically, the approach achieves state-of-the-art performance on VisDA and competitive results on PACS and Office-31, with ablations confirming the value of each component and analyses showing robustness to key hyperparameters.

Abstract

Source-Free Domain Adaptation (SFDA) is an emerging area of research that aims to adapt a model trained on a labeled source domain to an unlabeled target domain without accessing the source data. Most of the successful methods in this area rely on the concept of neighborhood consistency but are prone to errors due to misleading neighborhood information. In this paper, we explore this approach from the point of view of learning more informative clusters and mitigating the effect of noisy neighbors using a concept called neighborhood signature, and demonstrate that adaptation can be achieved using just a single loss term tailored to optimize the similarity and dissimilarity of predictions of samples in the target domain. In particular, our proposed method outperforms existing methods in the challenging VisDA dataset while also yielding competitive results on other benchmark datasets.

Figures (3)

• Figure 1: An overview of the proposed Source-Free Domain Adaptation (SFDA) approach. For a batch of data, the encoder maps the input images to the feature space, where we compute the neighbourhood signatures of the samples. Using these neighbourhood signatures, we estimate the class encoding of the samples, and then we optimize the cosine similarity between the output prediction of each sample and the class encoding of all other samples in the batch as per our proposed loss function.
• Figure 2: Accuracy vs the number of nearest neighbors (K) on PACS
• Figure 3: Accuracy vs the decay base for $\alpha$ on PACS