DPA: A one-stop metric to measure bias amplification in classification datasets

TL;DR

The paper addresses bias amplification in classification datasets by introducing Directional Predictability Amplification (DPA), a one-stop, predictability-based metric that is directional and applicable to both balanced and unbalanced data. DPA uses attacker-based predictability in two directions, $A \rightarrow T$ and $T \rightarrow A$, combining dataset bias and model bias into scores $DPA_{A \rightarrow T}$ and $DPA_{T \rightarrow A}$ that are bounded in $[-1,1]$ and computed as relative changes to account for dataset bias. It improves over prior metrics by providing directionality, robustness to attacker choice, and a relative-change interpretation, and it is demonstrated to be the most reliable across COMPAS, COCO, and ImSitu datasets. The work includes extensive experiments, including controlled bias injections, to show DPA’s ability to detect both positive and negative amplification and its value for model selection and bias intervention planning. A public library further enables practitioners to compare DPA with existing metrics conveniently.

Abstract

Most ML datasets today contain biases. When we train models on these datasets, they often not only learn these biases but can worsen them -- a phenomenon known as bias amplification. Several co-occurrence-based metrics have been proposed to measure bias amplification in classification datasets. They measure bias amplification between a protected attribute (e.g., gender) and a task (e.g., cooking). These metrics also support fine-grained bias analysis by identifying the direction in which a model amplifies biases. However, co-occurrence-based metrics have limitations -- some fail to measure bias amplification in balanced datasets, while others fail to measure negative bias amplification. To solve these issues, recent work proposed a predictability-based metric called leakage amplification (LA). However, LA cannot identify the direction in which a model amplifies biases. We propose Directional Predictability Amplification (DPA), a predictability-based metric that is (1) directional, (2) works with balanced and unbalanced datasets, and (3) correctly identifies positive and negative bias amplification. DPA eliminates the need to evaluate models on multiple metrics to verify these three aspects. DPA also improves over prior predictability-based metrics like LA: it is less sensitive to the choice of attacker function (a hyperparameter in predictability-based metrics), reports scores within a bounded range, and accounts for dataset bias by measuring relative changes in predictability. Our experiments on well-known datasets like COMPAS (a tabular dataset), COCO, and ImSitu (image datasets) show that DPA is the most reliable metric to measure bias amplification in classification problems. To compare DPA with existing bias amplification metrics, we released a one-stop library of major bias amplification metrics at https://github.com/kerner-lab/Bias-Amplification.

Figures (6)

• Figure 1: $A \rightarrow T$ and $T \rightarrow A$ bias amplification using our predictability-based metric $DPA$
• Figure 2: The graphs show trends between (a) $LA$ vs model bias ($\lambda^{M}$), and between (b) $DPA$ vs model bias ($\Psi^{M}$), at different values of dataset bias ($\lambda^{D}$ for $LA$ and $\Psi^{D}$ for $DPA$). For the same model bias, $DPA$ reported much higher bias amplification values (compared to $LA$) when the dataset bias is small.
• Figure 3: For different attacker functions, we show the bias amplification reported by $DPA$ with absolute change (blue line) and $DPA$ with relative change (red line) on the synthetic dataset.
• Figure 4: For different attacker functions, we show the bias amplification reported by $DPA$ with absolute change (blue line) and $DPA$ with relative change (red line) on the ImSitu dataset.
• Figure 5: VGG-16 model attributions for images from the balanced COCO sub-dataset. For all cases, as the person in the image is progressively masked, the VGG-16 often relies on unlabeled background objects to predict gender.