Unsupervised Learning of Unbiased Visual Representations

TL;DR

This paper tackles the problem of learning unbiased visual representations without explicit bias annotations. It introduces U-EnD, a three-step unsupervised framework that first captures bias in a dedicated encoder, then derives bias pseudo-labels via clustering, and finally applies EnD-style debiasing with pseudo-labels to produce an unbiased classifier. A theoretical model quantifies model biasedness through mutual information and a bias tendency parameter, linking training dynamics to learning of easier biased patterns. Empirical results on synthetic and real-world datasets (e.g., Biased-MNIST, CelebA, IMDB Face, corrupted CIFAR-10) demonstrate state-of-the-art debiasing performance and, in some cases, parity or superiority over fully supervised baselines, highlighting the method’s potential for robust generalization when bias annotations are unavailable.

Abstract

Deep neural networks often struggle to learn robust representations in the presence of dataset biases, leading to suboptimal generalization on unbiased datasets. This limitation arises because the models heavily depend on peripheral and confounding factors, inadvertently acquired during training. Existing approaches to address this problem typically involve explicit supervision of bias attributes or reliance on prior knowledge about the biases. In this study, we address the challenging scenario where no explicit annotations of bias are available, and there's no prior knowledge about its nature. We present a fully unsupervised debiasing framework with three key steps: firstly, leveraging the inherent tendency to learn malignant biases to acquire a bias-capturing model; next, employing a pseudo-labeling process to obtain bias labels; and finally, applying cutting-edge supervised debiasing techniques to achieve an unbiased model. Additionally, we introduce a theoretical framework for evaluating model biasedness and conduct a detailed analysis of how biases impact neural network training. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of our method, showcasing state-of-the-art performance in various settings, occasionally surpassing fully supervised debiasing approaches.

Figures (9)

• Figure 1: Effect of the regularization term \ref{['eq:end-full-new']} with respect to $z_0$. The features extracted from samples belonging to the same target class (same arrow's vertex) are entangled through the $R^\parallel$ term (in light green) while features for the same bias class (same color in this case, with respect to $z_0$, dark green) are disentangled through the $R^\perp$ term (in pink).
• Figure 2: Overview of our unsupervised debiasing approach: first we train a bias-capturing encoder, then we determine bias pseudo-labels with a bias predictor. Finally, we employ the predicted labels for training a final debiased classifier. In this figure, we use Biased MNIST (bahng2019rebias) as an example, where the bias is given by a strong correlation between digit and color.
• Figure 3: Biased MNIST by bahng2019rebias. The bias is given by the correlation between digit and background color.
• Figure 4: Biased-MNIST accuracy on the training set (a), and on the unbiased test set (b) and (c). Results are reported in terms of mean and std across three different runs for every value of $\rho$. Given that the number of bias classes (colors) and target classes (digits) is the same, we can compute the bias accuracy by finding the permutation of predicted labels which maximizes the overlap with the ground truth bias labels.
• Figure 5: Normalized mutual information in case of perfect learner. As the number of classes $N_T$ increases, the curve smoothens. For low $N_T$ and low $\rho$ values, the anti-correlation phenomenon rises, and the mutual information increases.