A Systematic Study of Bias Amplification

TL;DR

The paper investigates bias amplification in prediction by conducting a systematic, controlled study using a synthetic bias in an image-classification task. It introduces BiasAmp_A→T to compare dataset bias with model predictions and explores six research questions across data bias, model capacity, training size, calibration, training dynamics, and task-difficulty relations. The findings reveal that bias amplification correlates with data bias, capacity, and training data size, exhibits a v-shaped capacity curve, and varies with calibration and the relative ease of recognizing group vs class membership, offering practical guidance on hyperparameter tuning and data requirements to mitigate amplification. The work highlights the nuanced tradeoffs between accuracy and fairness and provides code to facilitate replication and mitigation efforts.

Abstract

Recent research suggests that predictions made by machine-learning models can amplify biases present in the training data. When a model amplifies bias, it makes certain predictions at a higher rate for some groups than expected based on training-data statistics. Mitigating such bias amplification requires a deep understanding of the mechanics in modern machine learning that give rise to that amplification. We perform the first systematic, controlled study into when and how bias amplification occurs. To enable this study, we design a simple image-classification problem in which we can tightly control (synthetic) biases. Our study of this problem reveals that the strength of bias amplification is correlated to measures such as model accuracy, model capacity, model overconfidence, and amount of training data. We also find that bias amplification can vary greatly during training. Finally, we find that bias amplification may depend on the difficulty of the classification task relative to the difficulty of recognizing group membership: bias amplification appears to occur primarily when it is easier to recognize group membership than class membership. Our results suggest best practices for training machine-learning models that we hope will help pave the way for the development of better mitigation strategies. Code can be found at https://github.com/facebookresearch/cv_bias_amplification.

Figures (11)

• Figure 1: Left: Two examples of inversions performed on Fashion MNIST images. Right: Two examples of inversions performed on CIFAR-100 images. For each pair, the original image is on the left and the inverted image is on the right.
• Figure 2: Bias amplification, $\textrm{BiasAmp}_{A \rightarrow T}$, as a function of the degree of bias, $\epsilon$, for (left) ResNet-18 models trained on the Fashion MNIST dataset and (right) ResNet-110 models trained on the CIFAR-100 dataset. Shaded regions indicate the 95% confidence intervals over 20 independent experiments.
• Figure 3: Bias amplification (top) and accuracy (bottom) on the CIFAR-100 dataset as a function of three measures of model capacity. Each line represents a different amount of bias ($\epsilon$) in the training set. Shaded regions indicate the $95\%$ confidence intervals across 20 models. Left: Results for varying model depths. Middle: Results for varying model widths. Right: Results for varying weight decays.
• Figure 4: Bias amplification (left) and accuracy (right) of ResNet-110 models on the CIFAR-100 dataset as a function of the proportion of the training set used for training the models. The number of epochs for each model is scaled depending on the amount of training data used. Shaded regions indicate the $95\%$ confidence intervals across 20 models.
• Figure 5: Expected calibration error (ECE) of ResNet-110 models on the CIFAR-100 dataset as a function of dataset bias, $\epsilon$. Shaded regions indicate the 95% confidence intervals across 20 models.