Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale

TL;DR

This work tackles SFUDA, where neither target labels nor source data are available during adaptation. It introduces a three-step pipeline—model pre-adaptation to smooth predictions, Hypothesis Consolidation from Prediction Rationale (HCPR) using GradCAM-based rationales over multiple hypotheses, and FixMatch-based semi-supervised learning—to derive high-quality pseudo-labels. By ranking hypothesis rationales against class-wise centroids and selecting reliable hypotheses, the method converts SFUDA into a semi-supervised domain adaptation problem and achieves state-of-the-art results on Office-Home, DomainNet, and VisDA-C. The approach is compatible with existing SFUDA methods (e.g., SHOT, AaD) and offers favorable memory and runtime characteristics, with code available for integration and extension.

Abstract

Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data. The primary difficulty in this task is that the model's predictions may be inaccurate, and using these inaccurate predictions for model adaptation can lead to misleading results. To address this issue, this paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis. By consolidating these hypothesis rationales, we identify the most likely correct hypotheses, which we then use as a pseudo-labeled set to support a semi-supervised learning procedure for model adaptation. To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning. Extensive experimental results demonstrate that our approach achieves state-of-the-art performance in the SFUDA task and can be easily integrated into existing approaches to improve their performance. The codes are available at \url{https://github.com/GANPerf/HCPR}.

Figures (10)

• Figure 1: The visualizations illustrate the GradCAM selvaraju2017grad for predicting the image to a specific class. In the right-half section, it can be observed that even though the prediction is incorrect, the obtained rationale (region highlighted in the GradCAM) based on the correct label remains reasonable and resembles the rationale of the corresponding class depicted in the left-half section.
• Figure 2: In our method, we generate multiple prediction hypotheses based on the posterior probability of the current model. An image $I$ and its hypothetical label form a hypothesis, for example, ($I$, $y=\text{clock}$). For each hypothesis, GradCAM is calculated based on the hypothetical label, resulting in the corresponding rationale representation $a$. Subsequently, we calculate the centroid for the rationale representation of each class.
• Figure 3: These examples demonstrate the generation of reliable hypotheses. In Case 1, the rank ID of the second hypothesis derived from the image is lower than $\tau_1$, while all other hypotheses from the same image have ranks larger than $\tau_2$. Consequently, the second hypothesis of $I_1$ is selected as a reliable hypothesis. In Case 2, no hypothesis is selected because it has two hypotheses with rank IDs less than $\tau_2$, indicating a conflict between those hypotheses. Similarly, Case 3 is not selected because none of its hypotheses has rank IDs lower than $\tau_1$.
• Figure 4: Comparison of step 1 w/o FAR, w/o step 1, SHOT as step 1, and Ours on the trunk accuracy and average top-1 accuracy on the VisDA-C dataset.
• Figure 5: VisDA-C average accuracy (%) of the proposed method using different $\tau_1$ and $\tau_2$.