Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Bias

TL;DR

This work tackles robust self-training under sample selection bias by replacing overconfident softmax confidences with a diversity-based confidence, $s_\mathcal{T}(\mathbf{x})$, computed from an ensemble of $M$ heads. The authors formulate a loss that combines a supervised term with a diversity penalty, and provide theoretical results showing convergence to a unique stationary point and a lower bound linking diversity to predictive performance, influenced by the quality of the learned representation. Empirically, $\mathcal{T}$-similarity improves calibration and boosts accuracy across multiple pseudo-labeling policies and data modalities under SSB, demonstrating robustness to distribution shift and bias, with extensive analyses on datasets, ablations, and sensitivity. The approach is lightweight to implement, integrates with existing SSL methods, and has practical implications for reliable semi-supervised learning in real-world biased labeling scenarios.

Abstract

Self-training is a well-known approach for semi-supervised learning. It consists of iteratively assigning pseudo-labels to unlabeled data for which the model is confident and treating them as labeled examples. For neural networks, softmax prediction probabilities are often used as a confidence measure, although they are known to be overconfident, even for wrong predictions. This phenomenon is particularly intensified in the presence of sample selection bias, i.e., when data labeling is subject to some constraint. To address this issue, we propose a novel confidence measure, called $\mathcal{T}$-similarity, built upon the prediction diversity of an ensemble of linear classifiers. We provide the theoretical analysis of our approach by studying stationary points and describing the relationship between the diversity of the individual members and their performance. We empirically demonstrate the benefit of our confidence measure for three different pseudo-labeling policies on classification datasets of various data modalities. The code is available at https://github.com/ambroiseodt/tsim.

Figures (13)

• Figure 1: Unlabeled data (circles) colored by the confidence value of being from the orange class (right point cloud), from blue to orange as it increases. Left: Given a model trained on few labeled examples (blueintro orangeintro), softmax may provide wrong confidence estimates for unlabeled data. Right: Our method averages confidence estimates of a diverse set of classifiers leading to a well-calibrated model robust to distribution shift.
• Figure 2: Architecture of the model.
• Figure 3: Visualization of sample selection bias on Mushrooms. First row: Distribution of the projection values on the first principal component (PC1). Second row: Visualization of the projection values on the PC1 and the PC2.
• Figure 4: Test accuracies of the different baselines on $5$ datasets. Full results are in Appendix \ref{['app:app_failure_self']}.
• Figure 5: Increasing the diversity leads to a better-calibrated classifier in both IID and SSB settings.

Theorems & Definitions (14)

• proof
• proof
• proof
• proof
• proof
• proof
• proof
• Remark E.1
• proof
• proof