Is user feedback always informative? Retrieval Latent Defending for Semi-Supervised Domain Adaptation without Source Data

TL;DR

The paper tackles semi-supervised domain adaptation in deployed environments where user feedback provides a small labeled target set but is biased toward misclassified samples (Negatively Biased Feedback, NBF). It reveals that NBF can degrade adaptation when plugged into existing SemiSDA methods and introduces Retrieval Latent Defending (RLD), a plug-in strategy that balances the supervised signal by appending defending samples retrieved from a latent bank of pseudo-labeled data. RLD constructs a candidate bank by labeling unlabeled target data with top-probability pseudo-labels and selects defending samples per labeled instance to maintain balanced class discriminability, introducing the overall loss $\mathcal{L}_{total} = \mathcal{L}_{sup} + \mathcal{L}_{unsup} + \frac{1}{k \cdot B} \sum_{b=1}^{k \cdot B} \mathcal{H}(\hat{y}_{LD}^{b}, f_{\theta}(x_{LD}^{b}))$. Across natural image benchmarks (DomainNet-126, OfficeHome) and a real-world medical imaging task (MIMIC-CXR-V2), integrating RLD with multiple SemiSDA/SemiSL baselines yields consistent improvements, demonstrating a scalable, practical approach to robust adaptation under user feedback biases.

Abstract

This paper aims to adapt the source model to the target environment, leveraging small user feedback (i.e., labeled target data) readily available in real-world applications. We find that existing semi-supervised domain adaptation (SemiSDA) methods often suffer from poorly improved adaptation performance when directly utilizing such feedback data, as shown in Figure 1. We analyze this phenomenon via a novel concept called Negatively Biased Feedback (NBF), which stems from the observation that user feedback is more likely for data points where the model produces incorrect predictions. To leverage this feedback while avoiding the issue, we propose a scalable adapting approach, Retrieval Latent Defending. This approach helps existing SemiSDA methods to adapt the model with a balanced supervised signal by utilizing latent defending samples throughout the adaptation process. We demonstrate the problem caused by NBF and the efficacy of our approach across various benchmarks, including image classification, semantic segmentation, and a real-world medical imaging application. Our extensive experiments reveal that integrating our approach with multiple state-of-the-art SemiSDA methods leads to significant performance improvements.

Figures (11)

• Figure 1: (a) User feedback. Users can provide feedback while interacting with an ML product, where feedback is likely to be biased towards misclassified samples, which we define as Negatively Biased Feedback (NBF). (b) Adaptation results. We adapt the source model with small user feedback and large unlabeled target data using previous semi-supervised domain adaptation (SemiSDA) algorithms. Compared to random feedback, which is the classical SemiSDA setup where labeled data is a random subset of target data, model adaptation with NBF leads to subpar performance. This paper analyzes this problem and introduces a scalable solution.
• Figure 2: Adaptation with user feedback can be effective in alleviating performance degradation caused by domain shift. However, there are some challenges: HTML]D2DEC6(7,6)(i) user feedback may be a biased sampling of the true target distribution due to the nature of feedback, HTML]D2DEC6(9,6)(ii) the amount of the ground truths (GT) labels obtained through feedback is small, and HTML]D2DEC6(11,6)(iii) only unlabeled target data is typically available, not source data.
• Figure 3: Effect of negatively biased feedback. Our novel observations are that (a) user-provided feedback in practice has a biased distribution in each class cluster (the bottom center sub-figure) which is in contrast to random feedback, (b) Existing SemiSDA methods adapt the model by dominating the labeled data points (the right sub-figures) even though they are biasedly positioned, and (c) NBF prevents the model from having a decision boundary for true class clusters and leads to inferior adaptation performance (the bottom right sub-figure).
• Figure 4: Even though labeled data $(x_{lb}, y_{lb})$ is biasedly positioned, the model needs to be adapted with balanced class discriminability (i.e., decision boundary). (8,6)(i) However, previous SemiSDA methods have overlooked this fact and used the labeled data naively by applying a cross-entropy loss, leading to inadequate adaptation performance. (9,6)(ii) To alleviate this problem, we propose a scalable adapting approach, retrieval latent defending, which allows the model to adjust the balance of a mini-batch on each iteration by using latent defending samples $x_{LD}$ together with labeled data $x_{lb}$.
• Figure 5: NBF leads to higher performance than PBF. We compare different user-feedback configurations when the total number of feedback is 378 (top) and 630 (bottom). Positive and negative feedback refers to feedback from correct and incorrect model predictions, respectively. We run three random seed experiments and describe the average performance and standard deviation in the parenthesis.