Efficient Bias Mitigation Without Privileged Information

TL;DR

The paper tackles the problem of biased DNNs arising from spuriously correlated cues in the absence of group labels. It introduces Targeted Augmentations for Bias mitigation (TAB), a hyperparameter-free unsupervised pipeline that uses the full training history of a helper model to identify bias-conflicting samples via per-class loss-history clustering and to generate a group-balanced dataset for retraining. TAB achieves improved worst-group accuracy across synthetic and real-world vision tasks while maintaining competitive mean accuracy and without costly hyperparameter searches, enabling more practical deployment. The approach is simple to implement, does not require group annotations during training or validation, and reduces model-selection overhead, though it entails a double-training loop and clustering on loss histories, with opportunities for scalability and augmentation enhancements. Overall, TAB advances practical debiasing by delivering robust performance improvements in realistic resource-constrained settings.

Abstract

Deep neural networks trained via empirical risk minimisation often exhibit significant performance disparities across groups, particularly when group and task labels are spuriously correlated (e.g., "grassy background" and "cows"). Existing bias mitigation methods that aim to address this issue often either rely on group labels for training or validation, or require an extensive hyperparameter search. Such data and computational requirements hinder the practical deployment of these methods, especially when datasets are too large to be group-annotated, computational resources are limited, and models are trained through already complex pipelines. In this paper, we propose Targeted Augmentations for Bias Mitigation (TAB), a simple hyperparameter-free framework that leverages the entire training history of a helper model to identify spurious samples, and generate a group-balanced training set from which a robust model can be trained. We show that TAB improves worst-group performance without any group information or model selection, outperforming existing methods while maintaining overall accuracy.

Figures (6)

• Figure 1: Given a model $f_\theta$ and a training set $\mathcal{D}$, TAB learns a robust model as follows: (1) We train $f_\theta$ while keeping track of each sample's loss at every epoch. (2) We split samples of each class $l$ into two groups, a minority group $\mathcal{D}^{-}_l$ and a majority group $\mathcal{D}^{+}_l$, by clustering samples using their entire training histories. (3) We construct a group-balanced dataset by upsampling each minority group $\mathcal{D}_l^-$ so that $\mathcal{D}_l^-$ has the same size as the majority group $\mathcal{D}_l^+$. (4) We train $f_\theta$ from scratch using the augmented dataset.
• Figure 2: Accuracy throughout training on Waterbirds for (a) bias-conflicting validation samples, (b) bias-aligned validation samples, (c) all validation samples, and (d) bias-conflicting training samples. The shaded area marks when, as the loss converges (plot d), the validation WGA increases (green segment in plot a) while the bias-aligned accuracy (plot b) and mean accuracy (plot c) temporarily decrease (red segments).
• Figure 3: Test WGA for several hyperparameterisations of ERM, MaskTune asgari_masktune_bias_mitigation_neurips_22, LfF nam_lff_bias_mitigation_neurips_20, and JTT liu_jtt_bias_mitigation_icml_21 on Waterbirds. The crossed yellow bar shows the model with the highest validation accuracy across each method. This model has (1) a lower WGA than the model selected using validation WGA (starred bar), and (2) a similar or worse WGA than an equivalent ERM model (dashed line), implying the selected models are as biased as their ERM equivalents (i.e., no debiasing occurred). For clarity, on top of each bar we show each model's ranking when ordered based on their validation accuracies.
• Figure 5: (Left) Mean accuracy for TAB and ERM as the correlation strength varies in Even-Odd and cMNIST. (Right) Worst run wall-clock time (solid bars) and total model selection time as a function of TAB's worst run (crossed bars) for all methods in cMNIST.
• Figure H.1: Mean accuracy (left group of solid bars) and WGA (right group of dashed bars) of TAB with and without random transformations of upsampled examples in Even-Odd and cMNIST as we vary $p$ for both datasets. As baselines, we show the performance of an equivalent ERM model trained both with and without the same random transformations for its training set.