High-order Neighborhoods Know More: HyperGraph Learning Meets Source-free Unsupervised Domain Adaptation

TL;DR

This work tackles Source-free Unsupervised Domain Adaptation (SFDA) by introducing Hyper-SFDA, which leverages high-order neighborhood information through hypergraph learning on the unlabeled target domain and explicitly models domain shift with self-loops. An anchor-based hyperedge construction captures complex relationships among multiple target samples, while a self-loop mechanism weights samples by domain uncertainty via entropy-based measures. An adaptive relation-based objective, combined with a weight-averaged regularization term, guides training to pull within-cluster samples closer and push between-cluster samples apart with soft attention, improving robustness to domain differences. Extensive experiments on Office-31, Office-Home, VisDA, and PointDA-10 demonstrate state-of-the-art performance, with ablations confirming the additive benefits of high-order relations, self-loops, and adaptive losses. This approach highlights the practical impact of hypergraph-based structure modeling for SFDA under real-world privacy constraints.

Abstract

Source-free Unsupervised Domain Adaptation (SFDA) aims to classify target samples by only accessing a pre-trained source model and unlabelled target samples. Since no source data is available, transferring the knowledge from the source domain to the target domain is challenging. Existing methods normally exploit the pair-wise relation among target samples and attempt to discover their correlations by clustering these samples based on semantic features. The drawback of these methods includes: 1) the pair-wise relation is limited to exposing the underlying correlations of two more samples, hindering the exploration of the structural information embedded in the target domain; 2) the clustering process only relies on the semantic feature, while overlooking the critical effect of domain shift, i.e., the distribution differences between the source and target domains. To address these issues, we propose a new SFDA method that exploits the high-order neighborhood relation and explicitly takes the domain shift effect into account. Specifically, we formulate the SFDA as a Hypergraph learning problem and construct hyperedges to explore the local group and context information among multiple samples. Moreover, we integrate a self-loop strategy into the constructed hypergraph to elegantly introduce the domain uncertainty of each sample. By clustering these samples based on hyperedges, both the semantic feature and domain shift effects are considered. We then describe an adaptive relation-based objective to tune the model with soft attention levels for all samples. Extensive experiments are conducted on Office-31, Office-Home, VisDA, and PointDA-10 datasets. The results demonstrate the superiority of our method over state-of-the-art counterparts.

Figures (7)

• Figure 1: (Left) The pair-wise relation for sample $c$ only considers the affinity to sample $b$ in its neighborhood, but it fails to consider the high-order relation between sample $a$ and $c$, resulting in inaccurate predictions. (Middle) Comparison of the pair-wise relation based method AaD and our method on the accuracy of target samples' nearest neighbors having the correct predicted labels. A higher accuracy indicates similar samples are well-clustered, which thereby demonstrates using high-order relations enables better clustering. (Right) "Neighborhood misleading ratio" and "High-confidence" denote the mismatch between the predicted label and ground truth label of neighbors, and neighbors with high prediction confidence lln2023source. Without involving the domain shift in optimization, the misleading ratio fluctuates among different categories, indicating the domain shift is not generally solved. These figures are validated on the VisDA dataset peng2017visda.
• Figure 2: Overview of the proposed method Hyper-SFDA. (a) Initial results. (b) The hyperedges are constructed on the target domain to capture complicated higher-order neighborhood relations among multiple samples. (c) A self-loop strategy is proposed to consider the domain shift effect. (d) Clustering results by considering both hyperedges and self-loops. (e) After clustering, the model is trained using the proposed Adaptive Relation-based Objective, which pulls close samples in the same cluster and pushes away samples in different clusters with different attention levels. (f) Final results. See text for more details.
• Figure 3: (Left) Effect of different intervals in updating hypergraph. (Right) Effect of different hyperedge degrees.
• Figure 4: Effect of different numbers of nearest neighbors on Office-31 (Left) and Office-Home (Right).
• Figure 5: (Left) Effect of different scale factor $\gamma$. (Right) Effect of different balancing factor $\beta$.